autoreg-skills-max: initial public release (restored, redacted, hardened)
This commit is contained in:
baradok60915d52e8425
196
skills/cloudflare-bypass-scraping/SKILL.md
Normal file
196
skills/cloudflare-bypass-scraping/SKILL.md
Normal file
@@ -0,0 +1,196 @@
|
||||
---
|
||||
name: cloudflare-bypass-scraping
|
||||
description: Scrape Cloudflare sites. curl_cffi or full browser bypass.
|
||||
category: automation
|
||||
version: 1.0.0
|
||||
author: ENI/Gothbreach
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Cloudflare Bypass for Web Scraping
|
||||
|
||||
## When to Use
|
||||
|
||||
- Target website returns Cloudflare "Checking your browser" or "Attention Required" interstitial
|
||||
- Standard `requests`/`httpx` gets 403 or empty response from a CF-protected site
|
||||
- Need to scrape emails, data, or content from sites behind Cloudflare
|
||||
- Single-page or multi-page crawling with CF bypass
|
||||
|
||||
Two-tier approach for scraping websites protected by Cloudflare:
|
||||
|
||||
| Tier | Tool | When | Overhead |
|
||||
|------|------|------|----------|
|
||||
| 1 — Lightweight | `curl_cffi` + `impersonate="chrome124"` | 90% of sites (TLS fingerprint check only) | Zero — pure HTTP |
|
||||
| 2 — Full browser | CloudflareBypassForScraping server (CloakBrowser) | Sites with full JS challenge / Turnstile | 535 MB Chromium download, ~500 MB RAM |
|
||||
|
||||
## Tier 1: curl_cffi (use first, works now)
|
||||
|
||||
`curl_cffi` impersonates Chrome's TLS/JA3 fingerprint at the HTTP library level. No browser, no JS engine — just TLS mimicry. Handles the "Checking your browser" interstitial on most CF-protected sites.
|
||||
|
||||
```python
|
||||
from curl_cffi import requests
|
||||
|
||||
resp = requests.get(
|
||||
"https://protected-site.com",
|
||||
impersonate="chrome124", # Chrome 124 TLS fingerprint
|
||||
timeout=30,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
},
|
||||
)
|
||||
html = resp.text
|
||||
```
|
||||
|
||||
**Install**: `pip install curl_cffi`
|
||||
|
||||
**Limitations**: Does NOT execute JavaScript. Sites with Turnstile widget or full "5-second challenge" page will still block. Fall back to Tier 2.
|
||||
|
||||
## Tier 1.5: Headless-Chrome fetch bridge (когда curl_cffi бессилен)
|
||||
|
||||
**Симптом:** API возвращает «invalid api key»/403/пустой ответ из curl, curl_cffi (ВСЕ impersonate-профили: chrome/chrome124/chrome131/safari/edge99), python requests — но тот же ключ через `fetch()` в реальном Chrome работает. IP одинаковый, заголовки не влияют. Значит backend сверяет JA3/TLS строже, чем умеет curl_cffi (проверено live: beta.pvacodes.com, 2026-09-19).
|
||||
|
||||
**Решение:** локальный HTTP-мост, проксирующий запросы через `page.evaluate(fetch)` в headless Chrome:
|
||||
|
||||
```python
|
||||
# pva_bridge.py — рабочий паттерн (Desktop/jiofarm_pva/pva_bridge.py)
|
||||
# Playwright worker-поток + queue (greenlet-safe), ThreadingHTTPServer на 127.0.0.1
|
||||
from playwright.sync_api import sync_playwright
|
||||
# ВАЖНО: launch(headless=True, channel="chrome") БЕЗ доп. args и БЕЗ new_context(locale=...) —
|
||||
# «чистый» launch прошёл TLS-гейт, а версия с --disable-blink-features и locale=en-US — нет.
|
||||
# page.goto(origin) один раз, дальше page.evaluate(JS_FETCH, url) где
|
||||
# JS_FETCH = "async (url) => { const r = await fetch(url, {credentials:'include'}); return await r.text(); }"
|
||||
```
|
||||
|
||||
**Greenlet-ловушка (обязательно):** playwright sync_api НЕЛЬЗЯ вызывать из потоков ThreadingHTTPServer — «Cannot switch to a different thread». Паттерн: один daemon-worker владеет page, HTTP-handler кладёт `(payload, result_box, threading.Event)` в `queue.Queue` и ждёт `event.wait(90)`.
|
||||
|
||||
**Диагностика:** если bridge отдаёт «api key inactive» хотя тест-скрипт работает — diff launch-параметров и проверь, что ключ в файле не замаскирован (см. masking pitfall в `ai-provider-autoreg`).
|
||||
|
||||
Когда bridge нужен с залогиненной сессией (дашборд-скрап) — добавь автологин (fill email/password + submit + `wait_for_selector(visible)` из-за CSS fade-in) и `credentials:'include'` в fetch.
|
||||
|
||||
## Tier 2: CloudflareBypassForScraping Server
|
||||
|
||||
Full Chromium browser (CloakBrowser — stealth-patched Chromium 146) that solves the JS challenge, then caches `cf_clearance` cookies. Subsequent requests are instant.
|
||||
|
||||
**Repo**: `https://github.com/sarperavci/CloudflareBypassForScraping` (2.5k stars, MIT)
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
git clone https://github.com/sarperavci/CloudflareBypassForScraping.git
|
||||
cd CloudflareBypassForScraping
|
||||
pip install -r server_requirements.txt
|
||||
```
|
||||
|
||||
### ⚠️ Hermes PYTHONPATH pitfall
|
||||
|
||||
Hermes sets `PYTHONPATH` to the old `hermes-agent-0.18.2-latest` venv which has playwright files locked (PermissionError). The CF bypass server needs playwright. **Always run with `PYTHONPATH=` cleared:**
|
||||
|
||||
```bash
|
||||
PYTHONPATH= python server.py --port 8000
|
||||
```
|
||||
|
||||
First run downloads CloakBrowser Chromium (~535 MB) from GitHub releases. This takes 5-10 minutes. Subsequent starts are instant.
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
cd CloudflareBypassForScraping
|
||||
PYTHONPATH= python server.py --port 8000 --log-level info
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
|
||||
**`/html` — Get rendered HTML after CF bypass:**
|
||||
```bash
|
||||
curl "http://localhost:8000/html?url=https://nopecha.com/demo/cloudflare"
|
||||
```
|
||||
|
||||
**`/cookies` — Get cf_clearance cookies only:**
|
||||
```bash
|
||||
curl "http://localhost:8000/cookies?url=https://protected-site.com"
|
||||
# Returns: {"cookies": {"cf_clearance": "..."}, "user_agent": "..."}
|
||||
```
|
||||
|
||||
**Mirror mode — proxy any request through the server:**
|
||||
```bash
|
||||
curl "http://localhost:8000/api/data" \
|
||||
-H "x-hostname: protected-site.com" \
|
||||
-H "x-proxy: http://user:pass@proxy:port" # optional upstream proxy
|
||||
```
|
||||
|
||||
### Control headers
|
||||
|
||||
| Header | Purpose |
|
||||
|--------|---------|
|
||||
| `x-hostname` | Target hostname (required) |
|
||||
| `x-proxy` | Upstream proxy URL (optional) |
|
||||
| `x-bypass-cache` | Set to `true` to force fresh cookie generation |
|
||||
|
||||
### Docker (alternative)
|
||||
|
||||
```bash
|
||||
docker compose pull && docker compose up -d
|
||||
```
|
||||
|
||||
## Integration pattern: email scraper
|
||||
|
||||
See `templates/email_scraper_cf.py` for a complete example that:
|
||||
- Tries `curl_cffi` first (Tier 1)
|
||||
- Falls back to CF Bypass Server (Tier 2) via `--backend server`
|
||||
- Crawls internal links, extracts emails, saves to JSON
|
||||
- Supports `--proxy`, `--quick` (single page), `--depth N`
|
||||
|
||||
```bash
|
||||
# Quick scan with curl_cffi
|
||||
PYTHONPATH= python email_scraper_cf.py https://target.com --quick
|
||||
|
||||
# Full crawl with CF bypass server
|
||||
PYTHONPATH= python email_scraper_cf.py https://target.com --backend server
|
||||
|
||||
# With residential proxy
|
||||
PYTHONPATH= python email_scraper_cf.py https://target.com --proxy socks5://user:pass@host:port
|
||||
```
|
||||
|
||||
## Decision flowchart
|
||||
|
||||
```
|
||||
Is the site behind Cloudflare?
|
||||
├── No → use requests / httpx directly
|
||||
└── Yes → try curl_cffi (impersonate="chrome124")
|
||||
├── Got 200 with content → done
|
||||
└── Got 403 / "Checking your browser" / challenge page
|
||||
└── Start CF Bypass Server → use /html or mirror mode
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **First CF bypass server start downloads 535 MB** — plan ahead, start it before the scraping session
|
||||
2. **Hermes PYTHONPATH locks playwright** — always `PYTHONPATH= python` when running the server or scraper
|
||||
3. **curl_cffi impersonate versions** — `chrome124` is current; if sites start detecting it, try `chrome120`, `chrome110`, `safari17_0`
|
||||
4. **CloakBrowser may trigger antivirus** — it's a modified Chromium binary; whitelist the directory
|
||||
5. **Cookie cache is per-worker** — if using `--workers N`, each worker has its own cache; cookies are regenerated per worker
|
||||
6. **Server uses significant RAM** — ~500 MB per worker; use `--workers 1` for scraping
|
||||
|
||||
## Tier 3: remote stealth render (firecrawl) — для SPA+Turnstile и когда локальный IP режет ассеты
|
||||
|
||||
Если локальные браузеры показывают белый экран (SPA-бандл висит в pending / ERR_CONNECTION_RESET), а curl и firecrawl качают тот же ассет с 200 — проблема в выходном IP, не в сайте. Работай через `firecrawl_scrape`/`firecrawl_interact` со stealth-прокси. Детали лимитов interact (60s MCP-таймаут, сессия живёт ~1-2 мин, лимит 2 джобов, паттерн Turnstile-регистрации) — `references/firecrawl-interact-turnstile-reg.md`.
|
||||
|
||||
## Tier 2.5: TRAWL — self-hosted FlareSolverr-killer (Bun + Camoufox + Redis + MCP)
|
||||
|
||||
**Repo**: https://github.com/germondai/trawl (AGPL-3.0, 979 stars, v1.8.0 08.10.2026 — активная разработка). Docs: https://docs.trawl.germondai.com
|
||||
|
||||
Когда: нужен ДОЛГОЖИВУЩЙ сервер обхода (не разовый скрипт) — парсинг под нагрузкой, *arr-стек (Prowlarr/Jackett/Sonarr), интеграция с AI-агентами через MCP. Многопрофильный: Cloudflare Turnstile/Interstitial, Akamai Bot Manager, Imperva/Incapsula, GeeTest Slide, Friendly Captcha v1/v2, ALTCHA, reCAPTCHA v2 (бесплатный audio через локальный Whisper), CAP proof-of-work. Всё локально, 2Captcha — опциональный fallback.
|
||||
|
||||
4-уровневая эскалация: Tier 0 plain HTTP → Tier 1 cached browser session (Redis) → Tier 2 fresh solve в Camoufox → Tier 3/4 (residential) proxy.
|
||||
|
||||
Деплой: `git clone https://github.com/germondai/trawl && cd trawl && cp .env.example .env && docker compose up -d`. API :8191 (`/health`, `/scrape`, FlareSolverr-compat `/v1`), MITM-прокси :8192 (MITM_ENABLED=true), MCP `/mcp` (MCP_ENABLED=true, Streamable HTTP: markdown/JSON-extract/screenshots/diagnostics). Дашборд :8191/dashboard. Первый старт 15-30с (browser pool warmup).
|
||||
|
||||
Детали и curl-примеры: `references/trawl-flaresolverr-replacement.md`.
|
||||
|
||||
## Related skills
|
||||
|
||||
- `anti-detection-browser` — browser automation for signup/registration flows (DrissionPage, nodriver)
|
||||
- `bpproxy` — residential proxy generation for IP rotation
|
||||
- `paywall-bypass` — bypassing paywalls on news sites
|
||||
Reference in New Issue
Block a user