# LobeHub :20140 Health Check & Auto-Restart

## ⚡ Quick-First Check: Watchdog Cron Status

**When the user says "lobehub не работает" — check the watchdog FIRST**
before anything else. The proxy may be alive NOW, but if the cron is paused
it will stay dead next crash.

```
cron job: 1b08ee756117 (lobehub-proxy-watchdog, every 5min)
```

If `enabled: false` or `state: paused` → resume it. If proxy is down AND watchdog
is paused → resume watchdog, THEN do the health check below.

## Health Criteria

```
GET http://localhost:20140/health → 200
{
  "status": "ok",
  "backend": "lobehub",          // MUST be "lobehub" (NOT "lobehub-cli" or "lobehub-api")
  "models": 20,                   // MUST be 20
  "accounts": {"total": 601, "valid": 601}
}
```

## Detect Wrong Process

Old/wrong processes to watch for:
- `lobehub_cli_proxy.py` — CLI version, backend="lobehub-cli", 9-11 models
- `lobehub_pool_v7.py` — v7, backend="closerouter"
- `lobehub_api_v8.py` — v8, backend="lobehub-api" (NOT "lobehub")
- `lobehub_gateway.py` — gateway on :20141, not the proxy

## Step-by-Step Fix (git-bash on Windows)

### 1. Health Check
```bash
curl -s --max-time 10 http://localhost:20140/health
```

### 2. If wrong or down: find the PID owning port 20140
```bash
netstat -ano | grep 20140 | grep LISTENING
# → TCP 127.0.0.1:20140 0.0.0.0:0 LISTENING 52268
```
Extract the PID from the last column.

### 3. Verify it's a python process
```bash
tasklist //FI "PID eq <PID>" //V
# → python.exe  <PID>  ...  NIKITA\Nikita  ...
```

### 4. Kill old process by PID

```bash
taskkill //F //PID <PID>
```

**Alternative — kill by cmdline filter (broad net, may silently miss):**
```bash
taskkill //F //IM python.exe //FI "cmdline like *lobehub*"
# Returns exit code 1 if no match — the filter runs but may miss due to cmdline format.
```

**Fallback — find PID via `wmic` when `taskkill //FI` misses:**
```bash
wmic process where "name='python.exe' and CommandLine like '%lobehub%'" get ProcessId
taskkill //F //PID <PID_FROM_WMIC>
```

### 5. Verify old process is gone
```bash
wmic process where "name='python.exe' and CommandLine like '%lobehub%'" get ProcessId 2>/dev/null
netstat -ano | grep 20140 | grep LISTENING || echo "PORT_FREE"
```
Both should show nothing if cleanup succeeded.

### 5. If lobehub_final.py is missing, recreate it
The script is stored in the skill as `scripts/lobehub_final.py`. Copy it to `C:\Users\User\lobehub_final.py` if missing. It's based on `lobehub_api_v8.py` with `backend="lobehub"` (not `"lobehub-api"`). 20 models, 601 accounts, LobeHub self-hosted primary + closerouter fallback.

### 6. Start the correct proxy

Use `terminal(background=true)` — the server is a long-lived daemon.
**Use `cd` + bare filename** to avoid Windows path mangling by MSYS/git-bash
(backslash paths get mangled to nonsense like `UsersUserlobehub_final.py`):

```bash
# In terminal() call with background=true:
cd /c/Users/User && python -u lobehub_final.py
```

**⚠️ Do NOT use `nohup ... &` in Hermes terminal() calls** — Hermes rejects
shell-level background wrappers with *"Foreground command uses shell-level
background wrappers"*. Use `terminal(background=true)` instead. The `nohup`
pattern only works in external bash / cron .sh scripts.

### 7. Wait for startup (accounts load takes ~5s with 601 accounts)
```bash
sleep 5
```

### 8. Verify health
```bash
curl -s --max-time 10 http://localhost:20140/health
```
Confirm: `backend="lobehub"`, `models=20`.

## Alternative: PowerShell (more detail)

If netstat doesn't show the process clearly:
```bash
powershell -Command "Get-NetTCPConnection -LocalPort 20140 -State Listen | Format-List"
powershell -Command "Get-CimInstance Win32_Process -Filter 'ProcessId=<PID>' | Select-Object ProcessId, Name, CommandLine"
```

## Pitfalls

- **`wmic` may not be available** on some Windows installs — use `tasklist //FI "PID eq <PID>" //V` instead for process info.
- **Two processes can fight for port 20140** — if your new process crashes with `[WinError 10013]` (socket access denied), an old process still holds the port. Kill it first, verify port is free, then restart.
- **The old proxy's `while True: serve_forever()` loop means the process restarts endlessly on crash** — you must kill by PID, not just wait for it to die. After killing, verify port is free with `netstat -ano | grep 20140 | grep LISTENING` before starting new one.
- **Health may show stale data** if the old process hasn't fully died — always check PID matches your new process via `netstat`.
- **`python -u` REQUIRED** for background HTTP servers — see `references/python-unbuffered-background.md`.
- **`taskkill //FI` filter may silently miss** — the `//FI "cmdline like *lobehub*"` filter returned exit code 1 (no match) even though a `lobehub_cli_proxy.py` process was running. Use `wmic` as fallback (see step 4).