196 lines
10 KiB
Markdown
196 lines
10 KiB
Markdown
---
|
||
name: cloudflare-bypass-scraping
|
||
description: Scrape Cloudflare sites. curl_cffi or full browser bypass.
|
||
category: automation
|
||
version: 1.0.0
|
||
author: ENI/Gothbreach
|
||
license: MIT
|
||
---
|
||
|
||
# Cloudflare Bypass for Web Scraping
|
||
|
||
## When to Use
|
||
|
||
- Target website returns Cloudflare "Checking your browser" or "Attention Required" interstitial
|
||
- Standard `requests`/`httpx` gets 403 or empty response from a CF-protected site
|
||
- Need to scrape emails, data, or content from sites behind Cloudflare
|
||
- Single-page or multi-page crawling with CF bypass
|
||
|
||
Two-tier approach for scraping websites protected by Cloudflare:
|
||
|
||
| Tier | Tool | When | Overhead |
|
||
|------|------|------|----------|
|
||
| 1 — Lightweight | `curl_cffi` + `impersonate="chrome124"` | 90% of sites (TLS fingerprint check only) | Zero — pure HTTP |
|
||
| 2 — Full browser | CloudflareBypassForScraping server (CloakBrowser) | Sites with full JS challenge / Turnstile | 535 MB Chromium download, ~500 MB RAM |
|
||
|
||
## Tier 1: curl_cffi (use first, works now)
|
||
|
||
`curl_cffi` impersonates Chrome's TLS/JA3 fingerprint at the HTTP library level. No browser, no JS engine — just TLS mimicry. Handles the "Checking your browser" interstitial on most CF-protected sites.
|
||
|
||
```python
|
||
from curl_cffi import requests
|
||
|
||
resp = requests.get(
|
||
"https://protected-site.com",
|
||
impersonate="chrome124", # Chrome 124 TLS fingerprint
|
||
timeout=30,
|
||
headers={
|
||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
|
||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||
"Accept-Language": "en-US,en;q=0.9",
|
||
},
|
||
)
|
||
html = resp.text
|
||
```
|
||
|
||
**Install**: `pip install curl_cffi`
|
||
|
||
**Limitations**: Does NOT execute JavaScript. Sites with Turnstile widget or full "5-second challenge" page will still block. Fall back to Tier 2.
|
||
|
||
## Tier 1.5: Headless-Chrome fetch bridge (когда curl_cffi бессилен)
|
||
|
||
**Симптом:** API возвращает «invalid api key»/403/пустой ответ из curl, curl_cffi (ВСЕ impersonate-профили: chrome/chrome124/chrome131/safari/edge99), python requests — но тот же ключ через `fetch()` в реальном Chrome работает. IP одинаковый, заголовки не влияют. Значит backend сверяет JA3/TLS строже, чем умеет curl_cffi (проверено live: beta.pvacodes.com, 2026-09-19).
|
||
|
||
**Решение:** локальный HTTP-мост, проксирующий запросы через `page.evaluate(fetch)` в headless Chrome:
|
||
|
||
```python
|
||
# pva_bridge.py — рабочий паттерн (Desktop/jiofarm_pva/pva_bridge.py)
|
||
# Playwright worker-поток + queue (greenlet-safe), ThreadingHTTPServer на 127.0.0.1
|
||
from playwright.sync_api import sync_playwright
|
||
# ВАЖНО: launch(headless=True, channel="chrome") БЕЗ доп. args и БЕЗ new_context(locale=...) —
|
||
# «чистый» launch прошёл TLS-гейт, а версия с --disable-blink-features и locale=en-US — нет.
|
||
# page.goto(origin) один раз, дальше page.evaluate(JS_FETCH, url) где
|
||
# JS_FETCH = "async (url) => { const r = await fetch(url, {credentials:'include'}); return await r.text(); }"
|
||
```
|
||
|
||
**Greenlet-ловушка (обязательно):** playwright sync_api НЕЛЬЗЯ вызывать из потоков ThreadingHTTPServer — «Cannot switch to a different thread». Паттерн: один daemon-worker владеет page, HTTP-handler кладёт `(payload, result_box, threading.Event)` в `queue.Queue` и ждёт `event.wait(90)`.
|
||
|
||
**Диагностика:** если bridge отдаёт «api key inactive» хотя тест-скрипт работает — diff launch-параметров и проверь, что ключ в файле не замаскирован (см. masking pitfall в `ai-provider-autoreg`).
|
||
|
||
Когда bridge нужен с залогиненной сессией (дашборд-скрап) — добавь автологин (fill email/password + submit + `wait_for_selector(visible)` из-за CSS fade-in) и `credentials:'include'` в fetch.
|
||
|
||
## Tier 2: CloudflareBypassForScraping Server
|
||
|
||
Full Chromium browser (CloakBrowser — stealth-patched Chromium 146) that solves the JS challenge, then caches `cf_clearance` cookies. Subsequent requests are instant.
|
||
|
||
**Repo**: `https://github.com/sarperavci/CloudflareBypassForScraping` (2.5k stars, MIT)
|
||
|
||
### Setup
|
||
|
||
```bash
|
||
git clone https://github.com/sarperavci/CloudflareBypassForScraping.git
|
||
cd CloudflareBypassForScraping
|
||
pip install -r server_requirements.txt
|
||
```
|
||
|
||
### ⚠️ Hermes PYTHONPATH pitfall
|
||
|
||
Hermes sets `PYTHONPATH` to the old `hermes-agent-0.18.2-latest` venv which has playwright files locked (PermissionError). The CF bypass server needs playwright. **Always run with `PYTHONPATH=` cleared:**
|
||
|
||
```bash
|
||
PYTHONPATH= python server.py --port 8000
|
||
```
|
||
|
||
First run downloads CloakBrowser Chromium (~535 MB) from GitHub releases. This takes 5-10 minutes. Subsequent starts are instant.
|
||
|
||
### Run
|
||
|
||
```bash
|
||
cd CloudflareBypassForScraping
|
||
PYTHONPATH= python server.py --port 8000 --log-level info
|
||
```
|
||
|
||
### API Endpoints
|
||
|
||
**`/html` — Get rendered HTML after CF bypass:**
|
||
```bash
|
||
curl "http://localhost:8000/html?url=https://nopecha.com/demo/cloudflare"
|
||
```
|
||
|
||
**`/cookies` — Get cf_clearance cookies only:**
|
||
```bash
|
||
curl "http://localhost:8000/cookies?url=https://protected-site.com"
|
||
# Returns: {"cookies": {"cf_clearance": "..."}, "user_agent": "..."}
|
||
```
|
||
|
||
**Mirror mode — proxy any request through the server:**
|
||
```bash
|
||
curl "http://localhost:8000/api/data" \
|
||
-H "x-hostname: protected-site.com" \
|
||
-H "x-proxy: http://user:pass@proxy:port" # optional upstream proxy
|
||
```
|
||
|
||
### Control headers
|
||
|
||
| Header | Purpose |
|
||
|--------|---------|
|
||
| `x-hostname` | Target hostname (required) |
|
||
| `x-proxy` | Upstream proxy URL (optional) |
|
||
| `x-bypass-cache` | Set to `true` to force fresh cookie generation |
|
||
|
||
### Docker (alternative)
|
||
|
||
```bash
|
||
docker compose pull && docker compose up -d
|
||
```
|
||
|
||
## Integration pattern: email scraper
|
||
|
||
See `templates/email_scraper_cf.py` for a complete example that:
|
||
- Tries `curl_cffi` first (Tier 1)
|
||
- Falls back to CF Bypass Server (Tier 2) via `--backend server`
|
||
- Crawls internal links, extracts emails, saves to JSON
|
||
- Supports `--proxy`, `--quick` (single page), `--depth N`
|
||
|
||
```bash
|
||
# Quick scan with curl_cffi
|
||
PYTHONPATH= python email_scraper_cf.py https://target.com --quick
|
||
|
||
# Full crawl with CF bypass server
|
||
PYTHONPATH= python email_scraper_cf.py https://target.com --backend server
|
||
|
||
# With residential proxy
|
||
PYTHONPATH= python email_scraper_cf.py https://target.com --proxy socks5://user:pass@host:port
|
||
```
|
||
|
||
## Decision flowchart
|
||
|
||
```
|
||
Is the site behind Cloudflare?
|
||
├── No → use requests / httpx directly
|
||
└── Yes → try curl_cffi (impersonate="chrome124")
|
||
├── Got 200 with content → done
|
||
└── Got 403 / "Checking your browser" / challenge page
|
||
└── Start CF Bypass Server → use /html or mirror mode
|
||
```
|
||
|
||
## Pitfalls
|
||
|
||
1. **First CF bypass server start downloads 535 MB** — plan ahead, start it before the scraping session
|
||
2. **Hermes PYTHONPATH locks playwright** — always `PYTHONPATH= python` when running the server or scraper
|
||
3. **curl_cffi impersonate versions** — `chrome124` is current; if sites start detecting it, try `chrome120`, `chrome110`, `safari17_0`
|
||
4. **CloakBrowser may trigger antivirus** — it's a modified Chromium binary; whitelist the directory
|
||
5. **Cookie cache is per-worker** — if using `--workers N`, each worker has its own cache; cookies are regenerated per worker
|
||
6. **Server uses significant RAM** — ~500 MB per worker; use `--workers 1` for scraping
|
||
|
||
## Tier 3: remote stealth render (firecrawl) — для SPA+Turnstile и когда локальный IP режет ассеты
|
||
|
||
Если локальные браузеры показывают белый экран (SPA-бандл висит в pending / ERR_CONNECTION_RESET), а curl и firecrawl качают тот же ассет с 200 — проблема в выходном IP, не в сайте. Работай через `firecrawl_scrape`/`firecrawl_interact` со stealth-прокси. Детали лимитов interact (60s MCP-таймаут, сессия живёт ~1-2 мин, лимит 2 джобов, паттерн Turnstile-регистрации) — `references/firecrawl-interact-turnstile-reg.md`.
|
||
|
||
## Tier 2.5: TRAWL — self-hosted FlareSolverr-killer (Bun + Camoufox + Redis + MCP)
|
||
|
||
**Repo**: https://github.com/germondai/trawl (AGPL-3.0, 979 stars, v1.8.0 08.10.2026 — активная разработка). Docs: https://docs.trawl.germondai.com
|
||
|
||
Когда: нужен ДОЛГОЖИВУЩЙ сервер обхода (не разовый скрипт) — парсинг под нагрузкой, *arr-стек (Prowlarr/Jackett/Sonarr), интеграция с AI-агентами через MCP. Многопрофильный: Cloudflare Turnstile/Interstitial, Akamai Bot Manager, Imperva/Incapsula, GeeTest Slide, Friendly Captcha v1/v2, ALTCHA, reCAPTCHA v2 (бесплатный audio через локальный Whisper), CAP proof-of-work. Всё локально, 2Captcha — опциональный fallback.
|
||
|
||
4-уровневая эскалация: Tier 0 plain HTTP → Tier 1 cached browser session (Redis) → Tier 2 fresh solve в Camoufox → Tier 3/4 (residential) proxy.
|
||
|
||
Деплой: `git clone https://github.com/germondai/trawl && cd trawl && cp .env.example .env && docker compose up -d`. API :8191 (`/health`, `/scrape`, FlareSolverr-compat `/v1`), MITM-прокси :8192 (MITM_ENABLED=true), MCP `/mcp` (MCP_ENABLED=true, Streamable HTTP: markdown/JSON-extract/screenshots/diagnostics). Дашборд :8191/dashboard. Первый старт 15-30с (browser pool warmup).
|
||
|
||
Детали и curl-примеры: `references/trawl-flaresolverr-replacement.md`.
|
||
|
||
## Related skills
|
||
|
||
- `anti-detection-browser` — browser automation for signup/registration flows (DrissionPage, nodriver)
|
||
- `bpproxy` — residential proxy generation for IP rotation
|
||
- `paywall-bypass` — bypassing paywalls on news sites |