vlad/autoreg-skills-max
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
autoreg-skills-max/skills/cloudflare-bypass-scraping/SKILL.md

10 KiB
Raw Permalink Blame History

name, description, category, version, author, license
name description category version author license
cloudflare-bypass-scraping Scrape Cloudflare sites. curl_cffi or full browser bypass. automation 1.0.0 ENI/Gothbreach MIT

Cloudflare Bypass for Web Scraping

When to Use

  • Target website returns Cloudflare "Checking your browser" or "Attention Required" interstitial
  • Standard requests/httpx gets 403 or empty response from a CF-protected site
  • Need to scrape emails, data, or content from sites behind Cloudflare
  • Single-page or multi-page crawling with CF bypass

Two-tier approach for scraping websites protected by Cloudflare:

Tier Tool When Overhead
1 — Lightweight curl_cffi + impersonate="chrome124" 90% of sites (TLS fingerprint check only) Zero — pure HTTP
2 — Full browser CloudflareBypassForScraping server (CloakBrowser) Sites with full JS challenge / Turnstile 535 MB Chromium download, ~500 MB RAM

Tier 1: curl_cffi (use first, works now)

curl_cffi impersonates Chrome's TLS/JA3 fingerprint at the HTTP library level. No browser, no JS engine — just TLS mimicry. Handles the "Checking your browser" interstitial on most CF-protected sites.

from curl_cffi import requests

resp = requests.get(
    "https://protected-site.com",
    impersonate="chrome124",  # Chrome 124 TLS fingerprint
    timeout=30,
    headers={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
    },
)
html = resp.text

Install: pip install curl_cffi

Limitations: Does NOT execute JavaScript. Sites with Turnstile widget or full "5-second challenge" page will still block. Fall back to Tier 2.

Tier 1.5: Headless-Chrome fetch bridge (когда curl_cffi бессилен)

Симптом: API возвращает «invalid api key»/403/пустой ответ из curl, curl_cffi (ВСЕ impersonate-профили: chrome/chrome124/chrome131/safari/edge99), python requests — но тот же ключ через fetch() в реальном Chrome работает. IP одинаковый, заголовки не влияют. Значит backend сверяет JA3/TLS строже, чем умеет curl_cffi (проверено live: beta.pvacodes.com, 2026-09-19).

Решение: локальный HTTP-мост, проксирующий запросы через page.evaluate(fetch) в headless Chrome:

# pva_bridge.py — рабочий паттерн (Desktop/jiofarm_pva/pva_bridge.py)
# Playwright worker-поток + queue (greenlet-safe), ThreadingHTTPServer на 127.0.0.1
from playwright.sync_api import sync_playwright
# ВАЖНО: launch(headless=True, channel="chrome") БЕЗ доп. args и БЕЗ new_context(locale=...) —
# «чистый» launch прошёл TLS-гейт, а версия с --disable-blink-features и locale=en-US — нет.
# page.goto(origin) один раз, дальше page.evaluate(JS_FETCH, url) где
# JS_FETCH = "async (url) => { const r = await fetch(url, {credentials:'include'}); return await r.text(); }"

Greenlet-ловушка (обязательно): playwright sync_api НЕЛЬЗЯ вызывать из потоков ThreadingHTTPServer — «Cannot switch to a different thread». Паттерн: один daemon-worker владеет page, HTTP-handler кладёт (payload, result_box, threading.Event) в queue.Queue и ждёт event.wait(90).

Диагностика: если bridge отдаёт «api key inactive» хотя тест-скрипт работает — diff launch-параметров и проверь, что ключ в файле не замаскирован (см. masking pitfall в ai-provider-autoreg).

Когда bridge нужен с залогиненной сессией (дашборд-скрап) — добавь автологин (fill email/password + submit + wait_for_selector(visible) из-за CSS fade-in) и credentials:'include' в fetch.

Tier 2: CloudflareBypassForScraping Server

Full Chromium browser (CloakBrowser — stealth-patched Chromium 146) that solves the JS challenge, then caches cf_clearance cookies. Subsequent requests are instant.

Repo: https://github.com/sarperavci/CloudflareBypassForScraping (2.5k stars, MIT)

Setup

git clone https://github.com/sarperavci/CloudflareBypassForScraping.git
cd CloudflareBypassForScraping
pip install -r server_requirements.txt

⚠️ Hermes PYTHONPATH pitfall

Hermes sets PYTHONPATH to the old hermes-agent-0.18.2-latest venv which has playwright files locked (PermissionError). The CF bypass server needs playwright. Always run with PYTHONPATH= cleared:

PYTHONPATH= python server.py --port 8000

First run downloads CloakBrowser Chromium (~535 MB) from GitHub releases. This takes 5-10 minutes. Subsequent starts are instant.

Run

cd CloudflareBypassForScraping
PYTHONPATH= python server.py --port 8000 --log-level info

API Endpoints

/html — Get rendered HTML after CF bypass:

curl "http://localhost:8000/html?url=https://nopecha.com/demo/cloudflare"

/cookies — Get cf_clearance cookies only:

curl "http://localhost:8000/cookies?url=https://protected-site.com"
# Returns: {"cookies": {"cf_clearance": "..."}, "user_agent": "..."}

Mirror mode — proxy any request through the server:

curl "http://localhost:8000/api/data" \
  -H "x-hostname: protected-site.com" \
  -H "x-proxy: http://user:pass@proxy:port"       # optional upstream proxy

Control headers

Header Purpose
x-hostname Target hostname (required)
x-proxy Upstream proxy URL (optional)
x-bypass-cache Set to true to force fresh cookie generation

Docker (alternative)

docker compose pull && docker compose up -d

Integration pattern: email scraper

See templates/email_scraper_cf.py for a complete example that:

  • Tries curl_cffi first (Tier 1)
  • Falls back to CF Bypass Server (Tier 2) via --backend server
  • Crawls internal links, extracts emails, saves to JSON
  • Supports --proxy, --quick (single page), --depth N
# Quick scan with curl_cffi
PYTHONPATH= python email_scraper_cf.py https://target.com --quick

# Full crawl with CF bypass server
PYTHONPATH= python email_scraper_cf.py https://target.com --backend server

# With residential proxy
PYTHONPATH= python email_scraper_cf.py https://target.com --proxy socks5://user:pass@host:port

Decision flowchart

Is the site behind Cloudflare?
├── No → use requests / httpx directly
└── Yes → try curl_cffi (impersonate="chrome124")
    ├── Got 200 with content → done
    └── Got 403 / "Checking your browser" / challenge page
        └── Start CF Bypass Server → use /html or mirror mode

Pitfalls

  1. First CF bypass server start downloads 535 MB — plan ahead, start it before the scraping session
  2. Hermes PYTHONPATH locks playwright — always PYTHONPATH= python when running the server or scraper
  3. curl_cffi impersonate versions — chrome124 is current; if sites start detecting it, try chrome120, chrome110, safari17_0
  4. CloakBrowser may trigger antivirus — it's a modified Chromium binary; whitelist the directory
  5. Cookie cache is per-worker — if using --workers N, each worker has its own cache; cookies are regenerated per worker
  6. Server uses significant RAM — ~500 MB per worker; use --workers 1 for scraping

Tier 3: remote stealth render (firecrawl) — для SPA+Turnstile и когда локальный IP режет ассеты

Если локальные браузеры показывают белый экран (SPA-бандл висит в pending / ERR_CONNECTION_RESET), а curl и firecrawl качают тот же ассет с 200 — проблема в выходном IP, не в сайте. Работай через firecrawl_scrape/firecrawl_interact со stealth-прокси. Детали лимитов interact (60s MCP-таймаут, сессия живёт ~1-2 мин, лимит 2 джобов, паттерн Turnstile-регистрации) — references/firecrawl-interact-turnstile-reg.md.

Tier 2.5: TRAWL — self-hosted FlareSolverr-killer (Bun + Camoufox + Redis + MCP)

Repo: https://github.com/germondai/trawl (AGPL-3.0, 979 stars, v1.8.0 08.10.2026 — активная разработка). Docs: https://docs.trawl.germondai.com

Когда: нужен ДОЛГОЖИВУЩЙ сервер обхода (не разовый скрипт) — парсинг под нагрузкой, *arr-стек (Prowlarr/Jackett/Sonarr), интеграция с AI-агентами через MCP. Многопрофильный: Cloudflare Turnstile/Interstitial, Akamai Bot Manager, Imperva/Incapsula, GeeTest Slide, Friendly Captcha v1/v2, ALTCHA, reCAPTCHA v2 (бесплатный audio через локальный Whisper), CAP proof-of-work. Всё локально, 2Captcha — опциональный fallback.

4-уровневая эскалация: Tier 0 plain HTTP → Tier 1 cached browser session (Redis) → Tier 2 fresh solve в Camoufox → Tier 3/4 (residential) proxy.

Деплой: git clone https://github.com/germondai/trawl && cd trawl && cp .env.example .env && docker compose up -d. API :8191 (/health, /scrape, FlareSolverr-compat /v1), MITM-прокси :8192 (MITM_ENABLED=true), MCP /mcp (MCP_ENABLED=true, Streamable HTTP: markdown/JSON-extract/screenshots/diagnostics). Дашборд :8191/dashboard. Первый старт 15-30с (browser pool warmup).

Детали и curl-примеры: references/trawl-flaresolverr-replacement.md.

Related skills

  • anti-detection-browser — browser automation for signup/registration flows (DrissionPage, nodriver)
  • bpproxy — residential proxy generation for IP rotation
  • paywall-bypass — bypassing paywalls on news sites