87 lines
5.9 KiB
Markdown
87 lines
5.9 KiB
Markdown
---
|
||
name: grok-x-parser-farm
|
||
description: Parse X/Twitter free via Grok farm + grok2api gateway.
|
||
version: 1
|
||
author: ENI/Gothbreach
|
||
license: MIT
|
||
metadata:
|
||
hermes:
|
||
tags: [x-twitter, parser, grok, autoreg, gateway, x_search]
|
||
related_skills: [xai-grok-console-registration, ai-provider-autoreg, x-content-advisor]
|
||
---
|
||
|
||
# Grok X-Parser Farm
|
||
|
||
Free Twitter/X parsing pipeline: autoreg Grok accounts → grok2api gateway pool → live X search.
|
||
Full workflow verified end-to-end 2026-09-20. Repo: github.com/MeshFinancial/grok-x-farm.
|
||
|
||
## When to use
|
||
|
||
- Need real X posts (handle/date/text/likes/URL) without paying Twitter API $200/mo
|
||
- Scaling a free Grok model pool for chat/search/image
|
||
- Monitoring X for keywords continuously
|
||
|
||
## Local deployment (this machine)
|
||
|
||
- Gateway: `C:\Users\User\grok2api-deploy\grok2api\grok2api.exe` on :8000 (restart: run binary with `--config config.yaml`)
|
||
- Admin creds: `C:\Users\User\grok2api-deploy\SECRETS.local.txt`
|
||
- Client key: `C:\Users\User\grok2api-deploy\g2a_key.txt`
|
||
- Autoreg: `C:\Users\User\grok-reg\grok-auto` (curl_cffi==0.13.0 pinned, patchright chromium installed)
|
||
- Parser: `C:\Users\User\grok2api-deploy\x_parser.py`, replenish: `replenish.py`
|
||
- Pool: 4 accounts × Web+Build = 8 records, 9 models (grok-4.5/4.6/chat-fast/composer/imagine)
|
||
- Unified CLI: `C:\Users\User\tmp\grok-x-farm\farm.py` — doctor / reg --count N / import / keys / parse / crawl (config farm.config.json: email provider, captcha mode, proxy mode+geo_whitelist)
|
||
|
||
## Repo layout v3 (2026-09-21, commit 5d21980)
|
||
|
||
- **ДВА каталога**: `tmp/grok-x-farm` = рабочая копия (внутри неё НЕТ своего .git — `git rev-parse --show-toplevel` показывает `C:/Users/User/tmp`!), `tmp/grok-x-farm-main` = чистый git-зеркало для push. Workflow: правки в рабочем → `cp` изменённых файлов в main-зеркало → push скриптом `tmp/push_grokfarm_v3.py` (Git Data API, токен `tmp/meshfin_pat.json`).
|
||
- v3: `gateway.tool_model`/`reasoning_model` = **grok-4.6** (4.5 больше не дефолт); `geo_whitelist` = US/EU (UA убран по требованию Влада); captcha: modes `free_browser/yescaptcha/capsolver/nopecha/2captcha` + `fallback_order`, `solver_timeout_sec`, `retry_per_solver`, `turnstile_sitekey` настраиваемый; secrets: YESCAPTCHA_KEY, CAPSOLVER_KEY, NOPETCHA_KEY, TWOCAPTCHA_KEY.
|
||
- Дашборд `dashboard/panel.py` (:порт в farm.config) — ENUM/INT/STR_KEYS + INT_RANGES валидация; settings-панель: models, farm.count_per_run/threads, gateway rpm/max_concurrent, captcha-тюнинг. После правки ENUM — обновить `tests/test_panel_offline.py` (bad-mode фикстура должна остаться НЕвалидной).
|
||
- Происхождение = ФОРК, не с нуля: gateway chenyme/grok2api (7.7k★ MIT), autoreg AaronL725/grok-register (2.2k★ MIT) — зафиксировано в README `## Credits / upstream`. Влад требует именно такую подачу.
|
||
- AUDIT.md/DOSSIER_AUDIT.md — исторические записи баг-фиксов, упоминания старых моделей/гео там НЕ переписывать (это пруфы прошлых проверок).
|
||
|
||
## Parse X — the ONLY working method
|
||
|
||
```
|
||
POST http://127.0.0.1:8000/v1/chat/completions
|
||
Authorization: Bearer g2a_xxx
|
||
{"model": "grok-chat-fast",
|
||
"messages": [{"role":"user","content":"Use your live X search. Find N most recent posts about QUERY from last D days. Reply ONLY with JSON array: [{handle,date,text,likes,url}]"}]}
|
||
```
|
||
|
||
Speed 8-10s, ~500 tokens. Web pool has NATIVE live X search. NOT `tools:[{type:x_search}]` —
|
||
Build upstream has no server-side search (num_server_side_tools_used always 0).
|
||
Verify results via `api.fxtwitter.com/{user}/status/{id}` (4/4 matched in tests).
|
||
|
||
## Replenish pool
|
||
|
||
```bash
|
||
cd C:\Users\User\grok2api-deploy && python replenish.py --count 5
|
||
# reg (tmail + free Turnstile) → import SSO → convert Web→Build; idempotent via imported_sso.txt
|
||
```
|
||
|
||
Turnstile solved FREE: patchright + manual turnstile.render, sitekey `0x4AAAAAAAhr9JGVDZbrZOo0`. ~141-210s/account.
|
||
|
||
## Pitfalls (all hit in practice)
|
||
|
||
1. curl_cffi 0.14+ broken on Windows → pin 0.13.0
|
||
2. RU egress geo-blocked by xAI → US/EU residential or clean DC
|
||
3. Signup page in RU breaks AaronL725 selectors → patch Russian variants into `registration_browser.py` scoreEntry (2 places)
|
||
4. credentialEncryptionKey must be base64(32 bytes) — `openssl rand -base64 32`
|
||
5. CPA refresh tokens die ~1 month (invalid_grant); SSO stays alive → import by SSO
|
||
6. Web→Build conversion unlocks grok-4.6 (v3 default, 2026-09-21): `POST /api/admin/v1/accounts/web/convert-to-build {"all":true,"strategy":"missing"}`
|
||
7. Gateway 502 transient → retry with backoff (farm.py has 3 attempts built in)
|
||
8. GitHub push: git-remote-https missing on this machine → Git Data/Contents API (empty repo: Contents API PUT first file bootstraps main branch; blobs/trees return 409 on empty repo)
|
||
9. X filters explicit grey-market queries ('cvv shop', 'account selling' → 0 results, 3/13 in test run); use neutral wording
|
||
10. Model dates drift outside requested window (1/105) → post-filter by date field
|
||
11. Tool use (function calling) = Build pool ONLY (grok-4.5/4.6, verified full cycle: tool_calls → result → answer). Web pool grok-chat-fast rejects tools 400 invalid_tools
|
||
12. doctor geo-check: ip-api countryCode field, not country name
|
||
|
||
## Admin API cheatsheet
|
||
|
||
```
|
||
POST /api/admin/v1/auth/login {"username","password"} → data.tokens.accessToken
|
||
POST /api/admin/v1/accounts/web/import multipart file=<SSO text>
|
||
POST /api/admin/v1/accounts/web/convert-to-build
|
||
POST /api/admin/v1/client-keys {"name":"x"} → data.secret (g2a_...)
|
||
GET /v1/models Bearer g2a_...
|
||
``` |