AI ile Web Scraping: Etik, Yasal, Pratik Rehber

Web scraping = public data → yapısal. AI ile HTML parsing artık hız problemi değil, etik ve yasal sorumluluk ana iş.

Etik ve Yasal

✅ Genelde OK

• Public sayfalar (login arkası değil)
• robots.txt uyumu
• Rate limit (saatlik X)
• ToS okuma
• Akademik araştırma
• Şahsi kullanım
• Veri attribution (kaynak göster)

🟡 Dikkat

• Kişisel veri (GDPR / KVKK)
• Telifli içerik (görsel, makale tam metin)
• Rakipten veri (haksız rekabet?)
• Cookie consent gerektiren

❌ Yasak / Risk

• Login arkası (CFAA — ABD)
• DDoS seviyesi istek
• ToS açıkça yasakladığı
• E-ticaret fiyat scraping (rakip)
• Spam / phishing kaynağı

Pratik Kural

Önce: API var mı?
Sonra: RSS / sitemap var mı?
Son çare: HTML scrape
Her zaman: rate limit + identify yourself

Tool Karşılaştırma

Tool	Tip	Use Case
requests + BeautifulSoup	Python klasik	Simple HTML
Scrapy	Python framework	Crawl at scale
Playwright	Browser automation	JS-heavy SPA
Selenium	Browser (eski)	Yedek
Firecrawl	AI-powered	HTML → markdown
Apify	Hosted	No-code
Browse AI	No-code	Visual config
ScrapingBee	Proxy + browser	Anti-bot bypass
Bright Data	Enterprise	Massive scale

ChatGPT ile Hızlı Scrape

"Aşağıdaki URL'den veri çıkar:

URL: [...]

İhtiyaç:
- Ürün adı
- Fiyat
- Stok durumu
- Görsel URL

Yapay sonuç JSON format.
"

ChatGPT (browse mode) sayfayı okur, çıktı verir. Sınırlı (rate limit), hızlı işler için.

Basit Python Scraping

"Trendyol product page scraper Python:

URL pattern: trendyol.com/.../p-{id}

Extract:
- Title
- Price (TL)
- Stock
- Image URLs
- Rating
- Review count
- Specs (dict)

Stack:
- requests + BeautifulSoup
- Lokal cache (file)
- Rate limit (1 req/sec)
- User agent rotation
- Error handling
- Retry with backoff
- JSON output

Ekle: robots.txt check first."

Playwright (JS-Heavy Sites)

from playwright.async_api import async_playwright

async def scrape_product(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)
        
        # Wait for dynamic content
        await page.wait_for_selector('.product-title')
        
        title = await page.text_content('.product-title')
        price = await page.text_content('.price')
        
        # JS execution
        data = await page.evaluate('''() => {
            return {
                title: document.querySelector('h1').innerText,
                price: window.productData?.price
            };
        }''')
        
        await browser.close()
        return data

"Aşağıdaki SPA için Playwright scraper:

Site: [...]
Veri: [...]

Edge cases:
- Loading spinner (wait until done)
- Lazy load (scroll)
- Pagination (next button click)
- Cookie consent (auto-accept)
- Geo-restriction (proxy)
- 2FA / Captcha (manual or skip)

Anti-detect:
- playwright-stealth
- Realistic user agent
- Mouse movements
- Random delays
- Cookie persistence
"

AI-Powered Scraping (Firecrawl)

import firecrawl

app = firecrawl.FirecrawlApp(api_key="...")

# Single page
result = app.scrape_url('https://example.com', {
    'formats': ['markdown', 'html'],
    'extract': {
        'schema': {
            'type': 'object',
            'properties': {
                'title': {'type': 'string'},
                'price': {'type': 'number'},
                'in_stock': {'type': 'boolean'}
            }
        }
    }
})

# Crawl entire site
result = app.crawl_url('https://example.com', {
    'limit': 100,
    'scrapeOptions': {'formats': ['markdown']}
})

Firecrawl HTML’i markdown’a + structured data’ya çevirir. LLM friendly.

Scrapy (Large Scale)

"Scrapy spider:

Site: news portal
Volume: 10K article/day
Schedule: 6 saat bir kere

Spider:
class NewsSpider(scrapy.Spider):
    name = 'news'
    custom_settings = {
        'ROBOTSTXT_OBEY': True,
        'DOWNLOAD_DELAY': 1,
        'CONCURRENT_REQUESTS_PER_DOMAIN': 4,
        'AUTOTHROTTLE_ENABLED': True,
    }
    
    def parse(self, response):
        for article in response.css('article'):
            yield {
                'title': article.css('h2::text').get(),
                'date': article.css('time::attr(datetime)').get(),
                'url': article.css('a::attr(href)').get()
            }
        
        next_page = response.css('a.next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

Pipeline:
- Dedupe (URL hash)
- Validate (required fields)
- Save (PostgreSQL)
- Notify (Slack)

Monitor:
- Job duration
- Success rate
- New articles count
"

AI ile Selector Generate

"Bu HTML için CSS selector ihtiyacım:

[HTML paste]

İhtiyaç:
- Ürün adı (h1 değil, dynamic class)
- Fiyat (currency format)
- Stok status
- Image carousel

Çıktı:
- Robust selector (data-* attr prefer)
- Fallback selector (eğer ilk fail)
- Test code (assert structure)
"

Schema Extraction

"Aşağıdaki HTML'den structured data:

[HTML]

Schema:
{
  "title": string,
  "author": string,
  "publishDate": ISO date,
  "tags": string[],
  "imageUrl": string,
  "wordCount": number
}

Çıktı: Python function HTML → dict
+ validation + error handling
"

Anti-Bot Strategies

Polite Scraping (önerilen)

✅
- User-Agent identify ("MyBot 1.0; +https://mysite.com/bot")
- robots.txt obey
- Rate limit (1 req/sec)
- Cache (don't re-scrape same URL)
- Off-peak hours
- Email contact in user-agent

Aggressive (etik dışı + risk)

🟡 Proxy rotation (residential IPs)
🟡 User agent rotation
🟡 Mouse movements (Puppeteer Extra Stealth)
🟡 Captcha solving (2Captcha)
❌ Browser fingerprint spoof
❌ Login auto with stolen creds

⚠️ Aggressive = yasal risk + işletme isim leke.

CAPTCHA

• reCAPTCHA v2 / v3
• hCaptcha
• Turnstile (Cloudflare)
• FunCaptcha
• Custom image challenges

Bypass:

• 2Captcha API ($1-3 / 1000 solved)
• AI vision (OpenAI Vision can solve simple)
• Headless browser fingerprint
• ⚠️ ToS ihlali genelde

Önerilen: site sahibi ile API anlaş, scrape değil.

Veri Saklama

"Scraped data storage:

Hacim: 1M product / month
Update: günde değişen ürün
Query: arama (full text)

Stack:
- PostgreSQL (master)
- Meilisearch / Elastic (search)
- S3 (raw HTML archive)
- Redis (cache hot products)

Schema:
- product (id, source_url, scraped_at, fields)
- product_history (price changes over time)
- product_image (separate table)

GDPR/KVKK:
- Kişisel veri scrape edilmedi (consent)
- Data retention policy
- Right to delete
"

Detay: Database Tasarım

Use Case: Fiyat Karşılaştırma

Detay: Alışveriş AI

"3 site fiyat karşılaştırma:
- Trendyol
- Hepsiburada
- Amazon TR

Her gün 2 kere, top 1000 ürün.

Output: CSV with diff %, lowest, trend.
"

Use Case: Job Listings

"LinkedIn job scraper (API yok):

⚠️ LinkedIn ToS bunu yasaklar. 
Alternative: LinkedIn Talent API (resmi)
Yedek: Glassdoor, Kariyer.net, Indeed
"

Use Case: News Aggregator

"News scraper:
- 20 Türk haber portalı
- RSS varsa öncelik
- Yoksa HTML scrape
- Dedupe (semantic similarity)
- AI ile özet
- Category classification
- Newsletter integration
"

Yaygın Hatalar

1. robots.txt ignore: Yasal risk
2. No rate limit: Banned IP
3. No retry: Geçici hata = veri kayıp
4. No deduplication: Aynı veri 10x
5. No validation: Garbage in, garbage out
6. Identity gizleme: Site sahibi reverse engineer
7. Stale selector: Site update = code break

Sonraki Adımlar

• Backend Geliştirme
• Database Tasarım
• Veri Analizi
• Alışveriş AI
• KVKK Uyumlu AI

Özet

Web scraping + AI = HTML problem çözüldü, etik problem yok olmadı. Firecrawl / Playwright / Scrapy + ChatGPT selector + structured extract. Anahtar: API önce, robots.txt, rate limit, transparency. Aggressive anti-bot bypass = yasal + ahlaki risk.