156 lines
6.2 KiB
Markdown
156 lines
6.2 KiB
Markdown
# Conol.ai Self-Healing Auto-Repair Daemon
|
|
|
|
**Date:** 2026-07-30
|
|
**Status:** Production — 30 accounts, continuous 5-min repair cycle
|
|
**File:** `C:\Users\User\Desktop\conol_autoreg\eni_repair_daemon.py` (28K)
|
|
|
|
## What It Does
|
|
|
|
A standalone daemon that continuously monitors the conol.ai account pool and auto-repairs individual accounts. Runs as `python eni_repair_daemon.py` (continuous 5-min cycle) or `--once` (single cycle).
|
|
|
|
## Architecture: Three-Phase Repair Cycle
|
|
|
|
### Phase 1: Mass Health Check
|
|
- Checks `/api/sessions` on ALL accounts concurrently (semaphore=5)
|
|
- Each account gets its OWN `httpx.AsyncClient` — sharing client mixes cookies → all 401 (pitfall #1 from conol-pool-gateway.md)
|
|
- Categorizes failures: HTTP 500 (server-side), HTTP 401 (auth expired), status 0 (network/no cookies)
|
|
|
|
### Phase 2: Server-Down Detection (NEW — skips unnecessary repairs)
|
|
**Rule:** If >80% of non-cookie failures return HTTP 500 → conol.ai server is down. **Skip ALL repairs.** Do NOT waste CDP sessions, re-register accounts, or restart gateway — the upstream is broken, not the accounts.
|
|
|
|
```python
|
|
non_cookie_fails = [r for r in health_results if not r.get("ok") and r.get("status") != 0]
|
|
status_500_count = sum(1 for r in non_cookie_fails if r.get("status") == 500)
|
|
server_down_ratio = status_500_count / max(1, len(non_cookie_fails))
|
|
|
|
if server_down_ratio >= 0.8 and non_cookie_fails:
|
|
log_event("server_down", ...)
|
|
return state # skip repairs, skip gateway restart
|
|
```
|
|
|
|
When server comes back (any account returns 201), fires `server_up` event.
|
|
|
|
### Phase 3: Individual Account Repair
|
|
Only runs when server is healthy. Three repair strategies:
|
|
|
|
| Strategy | Trigger | Action |
|
|
|----------|---------|--------|
|
|
| **Quest farming** | credits < 100 | Auto-farm quests via agent sessions (+300cr/quest) |
|
|
| **Re-login** | 401 or cookie error after 3 consecutive fails | CDP sign-in, refresh cookies, update pool |
|
|
| **Re-register** | `no_cookies` after 3 fails | Spawn `eni_conol.py reg 1` |
|
|
|
|
Repairs only trigger after `MAX_FAIL_STREAK=3` consecutive failures — avoids thrashing on transient errors.
|
|
|
|
### Gateway Auto-Restart
|
|
Checks gateway health via `GET /health`. If down:
|
|
1. `taskkill` old python processes
|
|
2. Start `gateway.py` as subprocess
|
|
3. Poll health for 5 attempts
|
|
4. Reload pool via `POST /pool/reload`
|
|
|
|
## Run Modes
|
|
|
|
```bash
|
|
# Single check+repair cycle
|
|
python eni_repair_daemon.py --once
|
|
|
|
# Continuous daemon (5-min interval)
|
|
python eni_repair_daemon.py
|
|
|
|
# Custom interval via env
|
|
ENI_REPAIR_INTERVAL=600 python eni_repair_daemon.py # 10 min
|
|
```
|
|
|
|
## Re-Login via CDP
|
|
|
|
When cookies expire (Better Auth session token times out), the daemon re-logs in via Chrome CDP:
|
|
|
|
```python
|
|
async def relogin_account(email: str) -> dict:
|
|
# 1. Connect CDP (http://127.0.0.1:9228)
|
|
browser = await p.chromium.connect_over_cdp(CDP_URL)
|
|
|
|
# 2. Clear old cookies + navigate to /login
|
|
await context.clear_cookies()
|
|
await page.goto(f"{BASE_URL}/login")
|
|
|
|
# 3. Execute reCAPTCHA v3 (same sitekey as registration)
|
|
captcha = await page.evaluate("""async ({siteKey}) => {
|
|
await new Promise(r => window.grecaptcha.ready(r));
|
|
return await window.grecaptcha.execute(siteKey, {action: 'sign_in'});
|
|
}""", {"siteKey": SITE_KEY})
|
|
|
|
# 4. POST /api/auth/sign-in/email with captcha
|
|
result = await page.evaluate("""...""", {
|
|
"path": "/api/auth/sign-in/email",
|
|
"body": {"email": email, "password": pw, "callbackURL": "/home", "rememberMe": True},
|
|
"captcha": captcha,
|
|
})
|
|
|
|
# 5. Verify session + save fresh cookies
|
|
cookies = await context.cookies([BASE_URL])
|
|
cookies_path.write_text(json.dumps(cookies))
|
|
|
|
# 6. Update pool file with new cookies_path
|
|
```
|
|
|
|
## Re-Registration
|
|
|
|
When cookies are completely gone (deleted/corrupted), spawns a fresh registration:
|
|
|
|
```python
|
|
proc = await asyncio.create_subprocess_exec(
|
|
sys.executable, "eni_conol.py", "reg", "1",
|
|
cwd=str(ROOT),
|
|
)
|
|
```
|
|
|
|
## Quest Auto-Farming
|
|
|
|
When an account has credits < `CREDIT_LOW_THRESHOLD` (100):
|
|
|
|
```python
|
|
QUEST_PROMPTS = {
|
|
"note_written_by_agent": "Please create a new note titled 'ENI Log'...",
|
|
"memory_written": "Save this fact to your persistent memory...",
|
|
"note_edited_by_agent": "Edit the note titled 'ENI Log'...",
|
|
"timer_scheduled": "Schedule a timer/reminder for 5 minutes...",
|
|
"note_shared": "Share the note 'ENI Log' publicly...",
|
|
"skill_installed": "Browse the skill marketplace...",
|
|
}
|
|
```
|
|
|
|
Each successful quest adds ~300 credits. Farm runs sequentially per account with 4s cooldown between quests.
|
|
|
|
## Logging
|
|
|
|
All events logged to `eni_repair_log.jsonl` (JSONL):
|
|
- `health_ok` / `health_fail` — per-account check results
|
|
- `server_down` / `server_up` — server-wide status changes
|
|
- `repair_login` / `repair_register` / `repair_farm` — repair actions
|
|
- `gateway_restart` / `gateway_fail` — gateway lifecycle
|
|
|
|
Repair state (fail streaks, counters) in `eni_repair_state.json`.
|
|
|
|
## Thresholds (configurable)
|
|
|
|
| Setting | Default | Description |
|
|
|---------|---------|-------------|
|
|
| `CHECK_INTERVAL_SEC` | 300 | Seconds between repair cycles |
|
|
| `CREDIT_LOW_THRESHOLD` | 100 | Farm quests when credits < this |
|
|
| `CREDIT_DEAD_THRESHOLD` | 0 | Re-register when credits hit 0 |
|
|
| `MAX_FAIL_STREAK` | 3 | Consecutive failures before repair |
|
|
| `GATEWAY_RESTART_ATTEMPTS` | 3 | Max restart tries per cycle |
|
|
|
|
## Key Pitfalls
|
|
|
|
1. **Don't repair when server is down.** If all accounts return 500, it's conol.ai's fault — not expired cookies. The server-down detection (80% threshold) prevents mass re-registration during server outages.
|
|
|
|
2. **Don't restart gateway when server is down.** Server-down detection returns before the gateway restart phase. No point keeping a gateway alive if the upstream returns 500 on all accounts.
|
|
|
|
3. **Isolate httpx clients per account.** Same pitfall as gateway — sharing `AsyncClient` across accounts merges cookies → all 401. Each health check creates its own client.
|
|
|
|
4. **CDP browser must be running.** Re-login requires Chrome with `--remote-debugging-port=9228`. If CDP is down, re-login fails silently and the daemon falls through to re-registration.
|
|
|
|
5. **Don't reset fail streaks too early.** Only reset after a successful repair. Transient 500s shouldn't reset the counter — the daemon needs to see real recovery.
|