autoreg-skills-max/skills/cloudflare-bypass-scraping/SKILL.md

name, description, category, version, author, license

name	description	category	version	author	license
cloudflare-bypass-scraping	Scrape Cloudflare sites. curl_cffi or full browser bypass.	automation	1.0.0	ENI/Gothbreach	MIT

Cloudflare Bypass for Web Scraping

When to Use

• Target website returns Cloudflare "Checking your browser" or "Attention Required" interstitial
• Standard requests/httpx gets 403 or empty response from a CF-protected site
• Need to scrape emails, data, or content from sites behind Cloudflare
• Single-page or multi-page crawling with CF bypass

Two-tier approach for scraping websites protected by Cloudflare:

Tier	Tool	When	Overhead
1 — Lightweight	curl_cffi + impersonate="chrome124"	90% of sites (TLS fingerprint check only)	Zero — pure HTTP
2 — Full browser	CloudflareBypassForScraping server (CloakBrowser)	Sites with full JS challenge / Turnstile	535 MB Chromium download, ~500 MB RAM

Tier 1: curl_cffi (use first, works now)

curl_cffi impersonates Chrome's TLS/JA3 fingerprint at the HTTP library level. No browser, no JS engine — just TLS mimicry. Handles the "Checking your browser" interstitial on most CF-protected sites.

from curl_cffi import requests

resp = requests.get(
    "https://protected-site.com",
    impersonate="chrome124",  # Chrome 124 TLS fingerprint
    timeout=30,
    headers={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
    },
)
html = resp.text

Install: pip install curl_cffi

Limitations: Does NOT execute JavaScript. Sites with Turnstile widget or full "5-second challenge" page will still block. Fall back to Tier 2.

Tier 1.5: Headless-Chrome fetch bridge (когда curl_cffi бессилен)

Симптом: API возвращает «invalid api key»/403/пустой ответ из curl, curl_cffi (ВСЕ impersonate-профили: chrome/chrome124/chrome131/safari/edge99), python requests — но тот же ключ через fetch() в реальном Chrome работает. IP одинаковый, заголовки не влияют. Значит backend сверяет JA3/TLS строже, чем умеет curl_cffi (проверено live: beta.pvacodes.com, 2026-09-19).

Решение: локальный HTTP-мост, проксирующий запросы через page.evaluate(fetch) в headless Chrome:

# pva_bridge.py — рабочий паттерн (Desktop/jiofarm_pva/pva_bridge.py)
# Playwright worker-поток + queue (greenlet-safe), ThreadingHTTPServer на 127.0.0.1
from playwright.sync_api import sync_playwright
# ВАЖНО: launch(headless=True, channel="chrome") БЕЗ доп. args и БЕЗ new_context(locale=...) —
# «чистый» launch прошёл TLS-гейт, а версия с --disable-blink-features и locale=en-US — нет.
# page.goto(origin) один раз, дальше page.evaluate(JS_FETCH, url) где
# JS_FETCH = "async (url) => { const r = await fetch(url, {credentials:'include'}); return await r.text(); }"

Greenlet-ловушка (обязательно): playwright sync_api НЕЛЬЗЯ вызывать из потоков ThreadingHTTPServer — «Cannot switch to a different thread». Паттерн: один daemon-worker владеет page, HTTP-handler кладёт (payload, result_box, threading.Event) в queue.Queue и ждёт event.wait(90).

Диагностика: если bridge отдаёт «api key inactive» хотя тест-скрипт работает — diff launch-параметров и проверь, что ключ в файле не замаскирован (см. masking pitfall в ai-provider-autoreg).

Когда bridge нужен с залогиненной сессией (дашборд-скрап) — добавь автологин (fill email/password + submit + wait_for_selector(visible) из-за CSS fade-in) и credentials:'include' в fetch.

Tier 2: CloudflareBypassForScraping Server

Full Chromium browser (CloakBrowser — stealth-patched Chromium 146) that solves the JS challenge, then caches cf_clearance cookies. Subsequent requests are instant.

Repo: https://github.com/sarperavci/CloudflareBypassForScraping (2.5k stars, MIT)

Setup

git clone https://github.com/sarperavci/CloudflareBypassForScraping.git
cd CloudflareBypassForScraping
pip install -r server_requirements.txt

⚠️ Hermes PYTHONPATH pitfall

Hermes sets PYTHONPATH to the old hermes-agent-0.18.2-latest venv which has playwright files locked (PermissionError). The CF bypass server needs playwright. Always run with PYTHONPATH= cleared:

PYTHONPATH= python server.py --port 8000

First run downloads CloakBrowser Chromium (~535 MB) from GitHub releases. This takes 5-10 minutes. Subsequent starts are instant.

Run

cd CloudflareBypassForScraping
PYTHONPATH= python server.py --port 8000 --log-level info

API Endpoints

/html — Get rendered HTML after CF bypass:

curl "http://localhost:8000/html?url=https://nopecha.com/demo/cloudflare"

/cookies — Get cf_clearance cookies only:

curl "http://localhost:8000/cookies?url=https://protected-site.com"
# Returns: {"cookies": {"cf_clearance": "..."}, "user_agent": "..."}

Mirror mode — proxy any request through the server:

curl "http://localhost:8000/api/data" \
  -H "x-hostname: protected-site.com" \
  -H "x-proxy: http://user:pass@proxy:port"       # optional upstream proxy

Control headers

Header	Purpose
x-hostname	Target hostname (required)
x-proxy	Upstream proxy URL (optional)
x-bypass-cache	Set to true to force fresh cookie generation

Docker (alternative)

docker compose pull && docker compose up -d

Integration pattern: email scraper

See templates/email_scraper_cf.py for a complete example that:

• Tries curl_cffi first (Tier 1)
• Falls back to CF Bypass Server (Tier 2) via --backend server
• Crawls internal links, extracts emails, saves to JSON
• Supports --proxy, --quick (single page), --depth N

# Quick scan with curl_cffi
PYTHONPATH= python email_scraper_cf.py https://target.com --quick

# Full crawl with CF bypass server
PYTHONPATH= python email_scraper_cf.py https://target.com --backend server

# With residential proxy
PYTHONPATH= python email_scraper_cf.py https://target.com --proxy socks5://user:pass@host:port

Decision flowchart

Is the site behind Cloudflare?
├── No → use requests / httpx directly
└── Yes → try curl_cffi (impersonate="chrome124")
    ├── Got 200 with content → done
    └── Got 403 / "Checking your browser" / challenge page
        └── Start CF Bypass Server → use /html or mirror mode

Pitfalls

1. First CF bypass server start downloads 535 MB — plan ahead, start it before the scraping session
2. Hermes PYTHONPATH locks playwright — always PYTHONPATH= python when running the server or scraper
3. curl_cffi impersonate versions — chrome124 is current; if sites start detecting it, try chrome120, chrome110, safari17_0
4. CloakBrowser may trigger antivirus — it's a modified Chromium binary; whitelist the directory
5. Cookie cache is per-worker — if using --workers N, each worker has its own cache; cookies are regenerated per worker
6. Server uses significant RAM — ~500 MB per worker; use --workers 1 for scraping

Tier 3: remote stealth render (firecrawl) — для SPA+Turnstile и когда локальный IP режет ассеты

Если локальные браузеры показывают белый экран (SPA-бандл висит в pending / ERR_CONNECTION_RESET), а curl и firecrawl качают тот же ассет с 200 — проблема в выходном IP, не в сайте. Работай через firecrawl_scrape/firecrawl_interact со stealth-прокси. Детали лимитов interact (60s MCP-таймаут, сессия живёт ~1-2 мин, лимит 2 джобов, паттерн Turnstile-регистрации) — references/firecrawl-interact-turnstile-reg.md.

Tier 2.5: TRAWL — self-hosted FlareSolverr-killer (Bun + Camoufox + Redis + MCP)

Repo: https://github.com/germondai/trawl (AGPL-3.0, 979 stars, v1.8.0 08.10.2026 — активная разработка). Docs: https://docs.trawl.germondai.com

Когда: нужен ДОЛГОЖИВУЩЙ сервер обхода (не разовый скрипт) — парсинг под нагрузкой, *arr-стек (Prowlarr/Jackett/Sonarr), интеграция с AI-агентами через MCP. Многопрофильный: Cloudflare Turnstile/Interstitial, Akamai Bot Manager, Imperva/Incapsula, GeeTest Slide, Friendly Captcha v1/v2, ALTCHA, reCAPTCHA v2 (бесплатный audio через локальный Whisper), CAP proof-of-work. Всё локально, 2Captcha — опциональный fallback.

4-уровневая эскалация: Tier 0 plain HTTP → Tier 1 cached browser session (Redis) → Tier 2 fresh solve в Camoufox → Tier 3/4 (residential) proxy.

Деплой: git clone https://github.com/germondai/trawl && cd trawl && cp .env.example .env && docker compose up -d. API :8191 (/health, /scrape, FlareSolverr-compat /v1), MITM-прокси :8192 (MITM_ENABLED=true), MCP /mcp (MCP_ENABLED=true, Streamable HTTP: markdown/JSON-extract/screenshots/diagnostics). Дашборд :8191/dashboard. Первый старт 15-30с (browser pool warmup).

Детали и curl-примеры: references/trawl-flaresolverr-replacement.md.

Related skills

• anti-detection-browser — browser automation for signup/registration flows (DrissionPage, nodriver)
• bpproxy — residential proxy generation for IP rotation
• paywall-bypass — bypassing paywalls on news sites