autoreg-skills-max: initial public release (restored, redacted, hardened)

skills/lobehub-pool/references/health-check-auto-restart.md

@@ -0,0 +1,126 @@
+# LobeHub :20140 Health Check & Auto-Restart
+
+## ⚡ Quick-First Check: Watchdog Cron Status
+
+**When the user says "lobehub не работает" — check the watchdog FIRST**
+before anything else. The proxy may be alive NOW, but if the cron is paused
+it will stay dead next crash.
+
+```
+cron job: 1b08ee756117 (lobehub-proxy-watchdog, every 5min)
+```
+
+If `enabled: false` or `state: paused` → resume it. If proxy is down AND watchdog
+is paused → resume watchdog, THEN do the health check below.
+
+## Health Criteria
+
+```
+GET http://localhost:20140/health → 200
+{
+  "status": "ok",
+  "backend": "lobehub",          // MUST be "lobehub" (NOT "lobehub-cli" or "lobehub-api")
+  "models": 20,                   // MUST be 20
+  "accounts": {"total": 601, "valid": 601}
+}
+```
+
+## Detect Wrong Process
+
+Old/wrong processes to watch for:
+- `lobehub_cli_proxy.py` — CLI version, backend="lobehub-cli", 9-11 models
+- `lobehub_pool_v7.py` — v7, backend="closerouter"
+- `lobehub_api_v8.py` — v8, backend="lobehub-api" (NOT "lobehub")
+- `lobehub_gateway.py` — gateway on :20141, not the proxy
+
+## Step-by-Step Fix (git-bash on Windows)
+
+### 1. Health Check
+```bash
+curl -s --max-time 10 http://localhost:20140/health
+```
+
+### 2. If wrong or down: find the PID owning port 20140
+```bash
+netstat -ano | grep 20140 | grep LISTENING
+# → TCP 127.0.0.1:20140 0.0.0.0:0 LISTENING 52268
+```
+Extract the PID from the last column.
+
+### 3. Verify it's a python process
+```bash
+tasklist //FI "PID eq <PID>" //V
+# → python.exe  <PID>  ...  NIKITA\Nikita  ...
+```
+
+### 4. Kill old process by PID
+
+```bash
+taskkill //F //PID <PID>
+```
+
+**Alternative — kill by cmdline filter (broad net, may silently miss):**
+```bash
+taskkill //F //IM python.exe //FI "cmdline like *lobehub*"
+# Returns exit code 1 if no match — the filter runs but may miss due to cmdline format.
+```
+
+**Fallback — find PID via `wmic` when `taskkill //FI` misses:**
+```bash
+wmic process where "name='python.exe' and CommandLine like '%lobehub%'" get ProcessId
+taskkill //F //PID <PID_FROM_WMIC>
+```
+
+### 5. Verify old process is gone
+```bash
+wmic process where "name='python.exe' and CommandLine like '%lobehub%'" get ProcessId 2>/dev/null
+netstat -ano | grep 20140 | grep LISTENING || echo "PORT_FREE"
+```
+Both should show nothing if cleanup succeeded.
+
+### 5. If lobehub_final.py is missing, recreate it
+The script is stored in the skill as `scripts/lobehub_final.py`. Copy it to `C:\Users\User\lobehub_final.py` if missing. It's based on `lobehub_api_v8.py` with `backend="lobehub"` (not `"lobehub-api"`). 20 models, 601 accounts, LobeHub self-hosted primary + closerouter fallback.
+
+### 6. Start the correct proxy
+
+Use `terminal(background=true)` — the server is a long-lived daemon.
+**Use `cd` + bare filename** to avoid Windows path mangling by MSYS/git-bash
+(backslash paths get mangled to nonsense like `UsersUserlobehub_final.py`):
+
+```bash
+# In terminal() call with background=true:
+cd /c/Users/User && python -u lobehub_final.py
+```
+
+**⚠️ Do NOT use `nohup ... &` in Hermes terminal() calls** — Hermes rejects
+shell-level background wrappers with *"Foreground command uses shell-level
+background wrappers"*. Use `terminal(background=true)` instead. The `nohup`
+pattern only works in external bash / cron .sh scripts.
+
+### 7. Wait for startup (accounts load takes ~5s with 601 accounts)
+```bash
+sleep 5
+```
+
+### 8. Verify health
+```bash
+curl -s --max-time 10 http://localhost:20140/health
+```
+Confirm: `backend="lobehub"`, `models=20`.
+
+## Alternative: PowerShell (more detail)
+
+If netstat doesn't show the process clearly:
+```bash
+powershell -Command "Get-NetTCPConnection -LocalPort 20140 -State Listen | Format-List"
+powershell -Command "Get-CimInstance Win32_Process -Filter 'ProcessId=<PID>' | Select-Object ProcessId, Name, CommandLine"
+```
+
+## Pitfalls
+
+- **`wmic` may not be available** on some Windows installs — use `tasklist //FI "PID eq <PID>" //V` instead for process info.
+- **Two processes can fight for port 20140** — if your new process crashes with `[WinError 10013]` (socket access denied), an old process still holds the port. Kill it first, verify port is free, then restart.
+- **The old proxy's `while True: serve_forever()` loop means the process restarts endlessly on crash** — you must kill by PID, not just wait for it to die. After killing, verify port is free with `netstat -ano | grep 20140 | grep LISTENING` before starting new one.
+- **Health may show stale data** if the old process hasn't fully died — always check PID matches your new process via `netstat`.
+- **`python -u` REQUIRED** for background HTTP servers — see `references/python-unbuffered-background.md`.
+- **`taskkill //FI` filter may silently miss** — the `//FI "cmdline like *lobehub*"` filter returned exit code 1 (no match) even though a `lobehub_cli_proxy.py` process was running. Use `wmic` as fallback (see step 4).