171 lines
6.4 KiB
Markdown
171 lines
6.4 KiB
Markdown
# python -u + exec: Critical for Background HTTP Servers
|
|
|
|
## Problem
|
|
|
|
Python HTTP servers (http.server, uvicorn, Flask, etc.) crash with exit code 1 or 127 when started in background mode via Hermes `terminal(background=true)`. The server appears to start (prints "Ready."), then exits moments later.
|
|
|
|
## Root Causes (two separate issues)
|
|
|
|
### 1. stdout buffering → exit code 1
|
|
|
|
Python's default stdout/stderr buffering. When running in background (non-TTY), Python buffers output. The buffered I/O blocks the event loop, the server can't respond to requests, and crashes.
|
|
|
|
### 2. bash profile interference → exit code 127
|
|
|
|
The bash profile (`~/.bashrc`, `~/.bash_profile`) injects environment banners and tool setup that interferes with Python's process lifecycle. The shell wrapper around `python script.py` gets killed by bash's job control, taking the Python process with it. Exit code 127 is bash's "command not found" — but the real cause is the shell process dying and taking the child with it.
|
|
|
|
**Detection:** Log shows `bash: no job control in this shell` and the ENI hacker environment banner, then immediate exit.
|
|
|
|
## Fix — Working Patterns (2026-07-02 verified)
|
|
|
|
### ⚠️ Hermes blocks `nohup`/`&`/`disown` in foreground terminal
|
|
|
|
As of 2026-07-02, Hermes rejects shell-level background wrappers in foreground
|
|
`terminal()` calls with: *"Foreground command uses shell-level background
|
|
wrappers (nohup/disown/setsid). Use terminal(background=true)..."*. The old
|
|
`nohup ... & disown` pattern (Option 1 below) is therefore **ONLY usable from
|
|
cron job shell scripts or external bash**, NOT from Hermes `terminal()` calls.
|
|
|
|
### Option 1 (PREFERRED): `terminal(background=true)` + `cd` + relative path
|
|
|
|
```bash
|
|
# CORRECT — use background=true and cd into the script's directory first:
|
|
# (in terminal() call, set background=true)
|
|
cd /c/Users/User && python -u lobehub_final.py
|
|
```
|
|
|
|
**Why `cd` + bare filename instead of absolute path:** Windows backslash
|
|
paths (`C:\Users\User\lobehub_final.py`) get mangled by MSYS/git-bash when
|
|
passed to `python` — the working directory gets concatenated with the script
|
|
path, producing nonsense like `UsersUserlobehub_final.py`. Even POSIX-form
|
|
absolute paths (`/c/Users/User/lobehub_final.py`) can be mangled when the
|
|
bash profile runs first. The `cd` + relative-path pattern sidesteps all path
|
|
translation issues — `python -u lobehub_final.py` has no path to mangle.
|
|
|
|
This is the canonical pattern for starting long-lived Python servers from
|
|
Hermes `terminal()` calls as of 2026-07-02.
|
|
|
|
### Option 2: `nohup + disown` (cron scripts / external bash only)
|
|
|
|
```bash
|
|
# CORRECT in a cron shell script or external bash — NOT in Hermes terminal():
|
|
nohup python -u /c/Users/User/lobehub_final.py > /dev/null 2>&1 & disown
|
|
```
|
|
|
|
`nohup` makes the process immune to SIGHUP, `disown` removes it from bash's
|
|
job table. Use this in cron job `.sh` scripts where Hermes' foreground
|
|
wrapper restrictions don't apply.
|
|
|
|
### Option 3: `exec python -u` (WORKS BUT FRAGILE)
|
|
|
|
```bash
|
|
# CORRECT — replaces shell process, survives bash profile:
|
|
exec python -u lobehub_final.py
|
|
```
|
|
|
|
`exec` replaces the current shell process with Python. No bash job control,
|
|
no profile interference. However, this makes the process the terminal session
|
|
leader — if the terminal is closed, the process may still die. Observed exit
|
|
code 1 after several minutes in some runs.
|
|
|
|
### Option 4: `cmd.exe /c` (DOES NOT WORK in background)
|
|
|
|
```bash
|
|
# WRONG in background mode — opens an interactive cmd shell and hangs:
|
|
cmd.exe /c "python -u C:\path\to\server.py"
|
|
```
|
|
|
|
This works in foreground but in background mode, cmd.exe opens an interactive
|
|
prompt instead of executing the command. Output shows `C:\Users\User\...>`
|
|
prompt. DO NOT use in `terminal(background=true)`.
|
|
|
|
### WRONG patterns
|
|
|
|
```bash
|
|
# WRONG (buffering) — will crash with exit 1:
|
|
python lobehub_final.py
|
|
|
|
# WRONG (bash profile) — will crash with exit 127:
|
|
python -u lobehub_final.py
|
|
|
|
# WRONG (Hermes blocks shell background wrappers in foreground terminal):
|
|
nohup python -u lobehub_final.py > /dev/null 2>&1 &
|
|
# Error: "Foreground command uses shell-level background wrappers"
|
|
# Use terminal(background=true) instead, or use from cron .sh script
|
|
|
|
# WRONG (Windows backslash path mangled by MSYS):
|
|
python -u C:\Users\User\lobehub_final.py
|
|
# Becomes: UsersUserlobehub_final.py (working dir concatenated)
|
|
# Fix: cd /c/Users/User && python -u lobehub_final.py
|
|
```
|
|
|
|
### Cron Watchdog Restart Pattern
|
|
|
|
When a watchdog cron restarts the service, the command depends on context:
|
|
|
|
**From Hermes `terminal(background=true)` (preferred for cron jobs that run
|
|
as Hermes cron tasks):**
|
|
```bash
|
|
# In terminal() call with background=true:
|
|
cd /c/Users/User && python -u lobehub_final.py
|
|
```
|
|
|
|
**From external bash / cron .sh script (nohup still works here):**
|
|
```bash
|
|
taskkill //F //FI "IMAGENAME eq python.exe" //FI "WINDOWTITLE eq *lobehub*" 2>/dev/null
|
|
nohup python -u /c/Users/User/lobehub_final.py > /dev/null 2>&1 & disown
|
|
```
|
|
|
|
### Startup Command Template
|
|
|
|
For any long-lived Python HTTP server on Windows bash:
|
|
|
|
```bash
|
|
# From Hermes terminal() — use background=true:
|
|
cd /c/Users/User && python -u server.py
|
|
|
|
# From external bash / cron .sh:
|
|
nohup python -u /path/to/server.py > /dev/null 2>&1 & disown
|
|
```
|
|
|
|
## Verification
|
|
|
|
After starting with `exec python -u`, wait 3-5 seconds and curl the health endpoint. If it responds, the server is stable.
|
|
|
|
## Affected Scripts
|
|
|
|
- `lobehub_final.py` (:20140) — confirmed crash without `exec python -u`
|
|
- Any `http.server.HTTPServer` based proxy
|
|
- Any uvicorn/FastAPI server
|
|
- Any Flask with `app.run()`
|
|
|
|
## Old Process Conflicts
|
|
|
|
Another common issue: an OLD version of the same service is already running on the port. Before starting the new version, kill all old processes:
|
|
|
|
```bash
|
|
# Find all lobehub python processes
|
|
powershell -Command "Get-WmiObject Win32_Process -Filter \"Name='python.exe'\" | Where-Object { $_.CommandLine -like '*lobehub*' } | Select-Object ProcessId, CommandLine | Format-List"
|
|
|
|
# Kill them all
|
|
taskkill //F //PID <pid1> //PID <pid2> ...
|
|
```
|
|
|
|
Then start the new version with `exec python -u`.
|
|
|
|
## Cron Watchdog Pattern
|
|
|
|
When a watchdog cron restarts a service, it MUST use the nohup pattern:
|
|
|
|
```bash
|
|
taskkill //F //FI "IMAGENAME eq python.exe" //FI "WINDOWTITLE eq *lobehub*" 2>/dev/null
|
|
nohup python -u /c/Users/User/lobehub_final.py > /dev/null 2>&1 & disown
|
|
```
|
|
|
|
### Startup Command Template
|
|
|
|
For any long-lived Python HTTP server on Windows bash:
|
|
|
|
```bash
|
|
nohup python -u /path/to/server.py > /dev/null 2>&1 & disown
|
|
``` |