autoreg-skills-max/skills/lobehub-pool/references/health-check-auto-restart.md

LobeHub :20140 Health Check & Auto-Restart

⚡ Quick-First Check: Watchdog Cron Status

When the user says "lobehub не работает" — check the watchdog FIRST before anything else. The proxy may be alive NOW, but if the cron is paused it will stay dead next crash.

cron job: 1b08ee756117 (lobehub-proxy-watchdog, every 5min)

If enabled: false or state: paused → resume it. If proxy is down AND watchdog is paused → resume watchdog, THEN do the health check below.

Health Criteria

GET http://localhost:20140/health → 200
{
  "status": "ok",
  "backend": "lobehub",          // MUST be "lobehub" (NOT "lobehub-cli" or "lobehub-api")
  "models": 20,                   // MUST be 20
  "accounts": {"total": 601, "valid": 601}
}

Detect Wrong Process

Old/wrong processes to watch for:

• lobehub_cli_proxy.py — CLI version, backend="lobehub-cli", 9-11 models
• lobehub_pool_v7.py — v7, backend="closerouter"
• lobehub_api_v8.py — v8, backend="lobehub-api" (NOT "lobehub")
• lobehub_gateway.py — gateway on :20141, not the proxy

Step-by-Step Fix (git-bash on Windows)

1. Health Check

curl -s --max-time 10 http://localhost:20140/health

2. If wrong or down: find the PID owning port 20140

netstat -ano | grep 20140 | grep LISTENING
# → TCP 127.0.0.1:20140 0.0.0.0:0 LISTENING 52268

Extract the PID from the last column.

3. Verify it's a python process

tasklist //FI "PID eq <PID>" //V
# → python.exe  <PID>  ...  NIKITA\Nikita  ...

4. Kill old process by PID

taskkill //F //PID <PID>

Alternative — kill by cmdline filter (broad net, may silently miss):

taskkill //F //IM python.exe //FI "cmdline like *lobehub*"
# Returns exit code 1 if no match — the filter runs but may miss due to cmdline format.

Fallback — find PID via wmic when taskkill //FI misses:

wmic process where "name='python.exe' and CommandLine like '%lobehub%'" get ProcessId
taskkill //F //PID <PID_FROM_WMIC>

5. Verify old process is gone

wmic process where "name='python.exe' and CommandLine like '%lobehub%'" get ProcessId 2>/dev/null
netstat -ano | grep 20140 | grep LISTENING || echo "PORT_FREE"

Both should show nothing if cleanup succeeded.

5. If lobehub_final.py is missing, recreate it

The script is stored in the skill as scripts/lobehub_final.py. Copy it to C:\Users\User\lobehub_final.py if missing. It's based on lobehub_api_v8.py with backend="lobehub" (not "lobehub-api"). 20 models, 601 accounts, LobeHub self-hosted primary + closerouter fallback.

6. Start the correct proxy

Use terminal(background=true) — the server is a long-lived daemon. Use cd + bare filename to avoid Windows path mangling by MSYS/git-bash (backslash paths get mangled to nonsense like UsersUserlobehub_final.py):

# In terminal() call with background=true:
cd /c/Users/User && python -u lobehub_final.py

⚠️ Do NOT use nohup ... & in Hermes terminal() calls — Hermes rejects shell-level background wrappers with "Foreground command uses shell-level background wrappers". Use terminal(background=true) instead. The nohup pattern only works in external bash / cron .sh scripts.

7. Wait for startup (accounts load takes ~5s with 601 accounts)

sleep 5

8. Verify health

curl -s --max-time 10 http://localhost:20140/health

Confirm: backend="lobehub", models=20.

Alternative: PowerShell (more detail)

If netstat doesn't show the process clearly:

powershell -Command "Get-NetTCPConnection -LocalPort 20140 -State Listen | Format-List"
powershell -Command "Get-CimInstance Win32_Process -Filter 'ProcessId=<PID>' | Select-Object ProcessId, Name, CommandLine"

Pitfalls

• wmic may not be available on some Windows installs — use tasklist //FI "PID eq <PID>" //V instead for process info.
• Two processes can fight for port 20140 — if your new process crashes with [WinError 10013] (socket access denied), an old process still holds the port. Kill it first, verify port is free, then restart.
• The old proxy's while True: serve_forever() loop means the process restarts endlessly on crash — you must kill by PID, not just wait for it to die. After killing, verify port is free with netstat -ano | grep 20140 | grep LISTENING before starting new one.
• Health may show stale data if the old process hasn't fully died — always check PID matches your new process via netstat.
• python -u REQUIRED for background HTTP servers — see references/python-unbuffered-background.md.
• taskkill //FI filter may silently miss — the //FI "cmdline like *lobehub*" filter returned exit code 1 (no match) even though a lobehub_cli_proxy.py process was running. Use wmic as fallback (see step 4).