Grok X-Parser Farm

Free Twitter/X parsing pipeline: autoreg Grok accounts → grok2api gateway pool → live X search. Full workflow verified end-to-end 2026-09-20. Repo: github.com/MeshFinancial/grok-x-farm.

When to use

Need real X posts (handle/date/text/likes/URL) without paying Twitter API $200/mo
Scaling a free Grok model pool for chat/search/image
Monitoring X for keywords continuously

Local deployment (this machine)

Gateway: C:\Users\User\grok2api-deploy\grok2api\grok2api.exe on :8000 (restart: run binary with --config config.yaml)
Admin creds: C:\Users\User\grok2api-deploy\SECRETS.local.txt
Client key: C:\Users\User\grok2api-deploy\g2a_key.txt
Autoreg: C:\Users\User\grok-reg\grok-auto (curl_cffi==0.13.0 pinned, patchright chromium installed)
Parser: C:\Users\User\grok2api-deploy\x_parser.py, replenish: replenish.py
Pool: 4 accounts × Web+Build = 8 records, 9 models (grok-4.5/4.6/chat-fast/composer/imagine)
Unified CLI: C:\Users\User\tmp\grok-x-farm\farm.py — doctor / reg --count N / import / keys / parse / crawl (config farm.config.json: email provider, captcha mode, proxy mode+geo_whitelist)

Repo layout v3 (2026-09-21, commit 5d21980)

ДВА каталога: tmp/grok-x-farm = рабочая копия (внутри неё НЕТ своего .git — git rev-parse --show-toplevel показывает C:/Users/User/tmp!), tmp/grok-x-farm-main = чистый git-зеркало для push. Workflow: правки в рабочем → cp изменённых файлов в main-зеркало → push скриптом tmp/push_grokfarm_v3.py (Git Data API, токен tmp/meshfin_pat.json).
v3: gateway.tool_model/reasoning_model = grok-4.6 (4.5 больше не дефолт); geo_whitelist = US/EU (UA убран по требованию Влада); captcha: modes free_browser/yescaptcha/capsolver/nopecha/2captcha + fallback_order, solver_timeout_sec, retry_per_solver, turnstile_sitekey настраиваемый; secrets: YESCAPTCHA_KEY, CAPSOLVER_KEY, NOPETCHA_KEY, TWOCAPTCHA_KEY.
Дашборд dashboard/panel.py (:порт в farm.config) — ENUM/INT/STR_KEYS + INT_RANGES валидация; settings-панель: models, farm.count_per_run/threads, gateway rpm/max_concurrent, captcha-тюнинг. После правки ENUM — обновить tests/test_panel_offline.py (bad-mode фикстура должна остаться НЕвалидной).
Происхождение = ФОРК, не с нуля: gateway chenyme/grok2api (7.7k★ MIT), autoreg AaronL725/grok-register (2.2k★ MIT) — зафиксировано в README ## Credits / upstream. Влад требует именно такую подачу.
AUDIT.md/DOSSIER_AUDIT.md — исторические записи баг-фиксов, упоминания старых моделей/гео там НЕ переписывать (это пруфы прошлых проверок).

Parse X — the ONLY working method

POST http://127.0.0.1:8000/v1/chat/completions
Authorization: Bearer g2a_xxx
{"model": "grok-chat-fast",
 "messages": [{"role":"user","content":"Use your live X search. Find N most recent posts about QUERY from last D days. Reply ONLY with JSON array: [{handle,date,text,likes,url}]"}]}

Speed 8-10s, ~500 tokens. Web pool has NATIVE live X search. NOT tools:[{type:x_search}] — Build upstream has no server-side search (num_server_side_tools_used always 0). Verify results via api.fxtwitter.com/{user}/status/{id} (4/4 matched in tests).

Replenish pool

cd C:\Users\User\grok2api-deploy && python replenish.py --count 5
# reg (tmail + free Turnstile) → import SSO → convert Web→Build; idempotent via imported_sso.txt

Turnstile solved FREE: patchright + manual turnstile.render, sitekey 0x4AAAAAAAhr9JGVDZbrZOo0. ~141-210s/account.

Pitfalls (all hit in practice)

curl_cffi 0.14+ broken on Windows → pin 0.13.0
RU egress geo-blocked by xAI → US/EU residential or clean DC
Signup page in RU breaks AaronL725 selectors → patch Russian variants into registration_browser.py scoreEntry (2 places)
credentialEncryptionKey must be base64(32 bytes) — openssl rand -base64 32
CPA refresh tokens die ~1 month (invalid_grant); SSO stays alive → import by SSO
Web→Build conversion unlocks grok-4.6 (v3 default, 2026-09-21): POST /api/admin/v1/accounts/web/convert-to-build {"all":true,"strategy":"missing"}
Gateway 502 transient → retry with backoff (farm.py has 3 attempts built in)
GitHub push: git-remote-https missing on this machine → Git Data/Contents API (empty repo: Contents API PUT first file bootstraps main branch; blobs/trees return 409 on empty repo)
X filters explicit grey-market queries ('cvv shop', 'account selling' → 0 results, 3/13 in test run); use neutral wording
Model dates drift outside requested window (1/105) → post-filter by date field
Tool use (function calling) = Build pool ONLY (grok-4.5/4.6, verified full cycle: tool_calls → result → answer). Web pool grok-chat-fast rejects tools 400 invalid_tools
doctor geo-check: ip-api countryCode field, not country name

Admin API cheatsheet

POST /api/admin/v1/auth/login                   {"username","password"} → data.tokens.accessToken
POST /api/admin/v1/accounts/web/import          multipart file=<SSO text>
POST /api/admin/v1/accounts/web/convert-to-build
POST /api/admin/v1/client-keys                  {"name":"x"} → data.secret (g2a_...)
GET  /v1/models                                 Bearer g2a_...

5.9 KiB Raw Permalink Blame History Unescape Escape