Mastering Scrapy-Playwright: Building High-Throughput Hybrid Pipelines for SPA Extraction

Scraping dynamic Single Page Applications (SPAs) often creates an engineering trade-off: vanilla Scrapy is fast but fails on client-side JavaScript hydration, while pure Playwright scripts accurately render pages but introduce significant memory overhead. Combining them via scrapy-playwright yields an enterprise hybrid crawler that minimizes resource use.

Targeted Route Aborting in settings.py

The key to scaling browser instances inside Scrapy is preventing Chromium from downloading non-essential assets like images, custom web fonts, and tracking scripts:

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_MAX_CONTEXTS = 8
PLAYWRIGHT_BROWSER_TYPE = "chromium"
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    "args": [
        "--no-sandbox",
        "--disable-dev-shm-usage",
        "--disable-gpu",
        "--blink-settings=imagesEnabled=false"
    ]
}

Conditional Browser Dispatch

Never route every request through Playwright. Use standard HTTP requests for server-side HTML or internal JSON endpoints, and reserve browser contexts strictly for complex hydration: