126 lines
4.8 KiB
Markdown
126 lines
4.8 KiB
Markdown
# LobeHub :20140 Health Check & Auto-Restart
|
|
|
|
## ⚡ Quick-First Check: Watchdog Cron Status
|
|
|
|
**When the user says "lobehub не работает" — check the watchdog FIRST**
|
|
before anything else. The proxy may be alive NOW, but if the cron is paused
|
|
it will stay dead next crash.
|
|
|
|
```
|
|
cron job: 1b08ee756117 (lobehub-proxy-watchdog, every 5min)
|
|
```
|
|
|
|
If `enabled: false` or `state: paused` → resume it. If proxy is down AND watchdog
|
|
is paused → resume watchdog, THEN do the health check below.
|
|
|
|
## Health Criteria
|
|
|
|
```
|
|
GET http://localhost:20140/health → 200
|
|
{
|
|
"status": "ok",
|
|
"backend": "lobehub", // MUST be "lobehub" (NOT "lobehub-cli" or "lobehub-api")
|
|
"models": 20, // MUST be 20
|
|
"accounts": {"total": 601, "valid": 601}
|
|
}
|
|
```
|
|
|
|
## Detect Wrong Process
|
|
|
|
Old/wrong processes to watch for:
|
|
- `lobehub_cli_proxy.py` — CLI version, backend="lobehub-cli", 9-11 models
|
|
- `lobehub_pool_v7.py` — v7, backend="closerouter"
|
|
- `lobehub_api_v8.py` — v8, backend="lobehub-api" (NOT "lobehub")
|
|
- `lobehub_gateway.py` — gateway on :20141, not the proxy
|
|
|
|
## Step-by-Step Fix (git-bash on Windows)
|
|
|
|
### 1. Health Check
|
|
```bash
|
|
curl -s --max-time 10 http://localhost:20140/health
|
|
```
|
|
|
|
### 2. If wrong or down: find the PID owning port 20140
|
|
```bash
|
|
netstat -ano | grep 20140 | grep LISTENING
|
|
# → TCP 127.0.0.1:20140 0.0.0.0:0 LISTENING 52268
|
|
```
|
|
Extract the PID from the last column.
|
|
|
|
### 3. Verify it's a python process
|
|
```bash
|
|
tasklist //FI "PID eq <PID>" //V
|
|
# → python.exe <PID> ... NIKITA\Nikita ...
|
|
```
|
|
|
|
### 4. Kill old process by PID
|
|
|
|
```bash
|
|
taskkill //F //PID <PID>
|
|
```
|
|
|
|
**Alternative — kill by cmdline filter (broad net, may silently miss):**
|
|
```bash
|
|
taskkill //F //IM python.exe //FI "cmdline like *lobehub*"
|
|
# Returns exit code 1 if no match — the filter runs but may miss due to cmdline format.
|
|
```
|
|
|
|
**Fallback — find PID via `wmic` when `taskkill //FI` misses:**
|
|
```bash
|
|
wmic process where "name='python.exe' and CommandLine like '%lobehub%'" get ProcessId
|
|
taskkill //F //PID <PID_FROM_WMIC>
|
|
```
|
|
|
|
### 5. Verify old process is gone
|
|
```bash
|
|
wmic process where "name='python.exe' and CommandLine like '%lobehub%'" get ProcessId 2>/dev/null
|
|
netstat -ano | grep 20140 | grep LISTENING || echo "PORT_FREE"
|
|
```
|
|
Both should show nothing if cleanup succeeded.
|
|
|
|
### 5. If lobehub_final.py is missing, recreate it
|
|
The script is stored in the skill as `scripts/lobehub_final.py`. Copy it to `C:\Users\User\lobehub_final.py` if missing. It's based on `lobehub_api_v8.py` with `backend="lobehub"` (not `"lobehub-api"`). 20 models, 601 accounts, LobeHub self-hosted primary + closerouter fallback.
|
|
|
|
### 6. Start the correct proxy
|
|
|
|
Use `terminal(background=true)` — the server is a long-lived daemon.
|
|
**Use `cd` + bare filename** to avoid Windows path mangling by MSYS/git-bash
|
|
(backslash paths get mangled to nonsense like `UsersUserlobehub_final.py`):
|
|
|
|
```bash
|
|
# In terminal() call with background=true:
|
|
cd /c/Users/User && python -u lobehub_final.py
|
|
```
|
|
|
|
**⚠️ Do NOT use `nohup ... &` in Hermes terminal() calls** — Hermes rejects
|
|
shell-level background wrappers with *"Foreground command uses shell-level
|
|
background wrappers"*. Use `terminal(background=true)` instead. The `nohup`
|
|
pattern only works in external bash / cron .sh scripts.
|
|
|
|
### 7. Wait for startup (accounts load takes ~5s with 601 accounts)
|
|
```bash
|
|
sleep 5
|
|
```
|
|
|
|
### 8. Verify health
|
|
```bash
|
|
curl -s --max-time 10 http://localhost:20140/health
|
|
```
|
|
Confirm: `backend="lobehub"`, `models=20`.
|
|
|
|
## Alternative: PowerShell (more detail)
|
|
|
|
If netstat doesn't show the process clearly:
|
|
```bash
|
|
powershell -Command "Get-NetTCPConnection -LocalPort 20140 -State Listen | Format-List"
|
|
powershell -Command "Get-CimInstance Win32_Process -Filter 'ProcessId=<PID>' | Select-Object ProcessId, Name, CommandLine"
|
|
```
|
|
|
|
## Pitfalls
|
|
|
|
- **`wmic` may not be available** on some Windows installs — use `tasklist //FI "PID eq <PID>" //V` instead for process info.
|
|
- **Two processes can fight for port 20140** — if your new process crashes with `[WinError 10013]` (socket access denied), an old process still holds the port. Kill it first, verify port is free, then restart.
|
|
- **The old proxy's `while True: serve_forever()` loop means the process restarts endlessly on crash** — you must kill by PID, not just wait for it to die. After killing, verify port is free with `netstat -ano | grep 20140 | grep LISTENING` before starting new one.
|
|
- **Health may show stale data** if the old process hasn't fully died — always check PID matches your new process via `netstat`.
|
|
- **`python -u` REQUIRED** for background HTTP servers — see `references/python-unbuffered-background.md`.
|
|
- **`taskkill //FI` filter may silently miss** — the `//FI "cmdline like *lobehub*"` filter returned exit code 1 (no match) even though a `lobehub_cli_proxy.py` process was running. Use `wmic` as fallback (see step 4). |