68 lines
4.7 KiB
Markdown
68 lines
4.7 KiB
Markdown
# Provider Key Fleet Audit and Semantic Health
|
||
|
||
Use this when a gateway has many OpenAI-compatible providers/keys and its admin panel reports misleading states such as `ConnectTimeout`, `online`, or a large healthy-key count.
|
||
|
||
## Principle
|
||
|
||
A provider/key is production-healthy only when a representative `POST /v1/chat/completions` succeeds semantically. DNS, TLS, `GET /v1/models == 200`, or HTTP 200 with an empty/soft-error payload are not enough.
|
||
|
||
## Safe audit sequence
|
||
|
||
1. Inventory provider name, exact base URL, number of keys, configured models, aliases, priority, and soft-error markers. Never print full keys; use an index plus a short SHA-256 fingerprint.
|
||
2. Attribute network failures separately:
|
||
- resolve DNS;
|
||
- measure TCP/TLS with an unauthenticated request;
|
||
- compare regional sibling hosts when official docs list them;
|
||
- distinguish connect timeout from read timeout and application errors.
|
||
3. Run `GET {base}/v1/models` for every key as an authentication/catalog pass.
|
||
4. Intersect configured model IDs with IDs actually returned by the catalog. A plausible but absent alias is a configuration bug, not a dead network.
|
||
5. Run a real completion for every key using a model known to exist on that exact route.
|
||
6. For reasoning models, use `max_tokens` 64–256 and accept non-empty `content` **or** `reasoning_content`. Tiny budgets such as 8 tokens can produce HTTP 200 with empty `content` and create false negatives.
|
||
7. Detect semantic failures inside HTTP 200 responses: recharge notices, insufficient-credit text, disabled-account messages, placeholder answers, and configured reject markers.
|
||
8. Classify precisely:
|
||
- 200 + usable assistant payload: usable;
|
||
- 200 + empty payload: retry with realistic output budget, then inconclusive/dead;
|
||
- 401: invalid key or wrong host/region;
|
||
- 403: valid identity but missing model entitlement;
|
||
- 402: insufficient balance;
|
||
- 429: quota/rate/billing state; quarantine rather than calling permanently invalid unless the message proves billing exhaustion;
|
||
- 404: often duplicated `/v1`, wrong API version, or wrong path;
|
||
- 5xx: provider/upstream failure; retry before pruning.
|
||
9. Prune unusable keys atomically from the private pool. Remove a whole provider only when no completion-verified key/model remains.
|
||
10. Restart safely, then verify the gateway's sanitized admin endpoint and one end-to-end completion per active provider family.
|
||
11. Sync the cleaned private pool to staging/production using secret-safe transport; verify file permissions and never copy it to a public repository.
|
||
|
||
## Health endpoint design
|
||
|
||
Do not make provider health depend only on `/models`. A sanitized health probe should:
|
||
|
||
- use one explicit configured model (after applying route aliases);
|
||
- send a tiny but realistic completion (`max_tokens` around 32–64);
|
||
- require non-empty `content` or `reasoning_content`;
|
||
- reject configured soft-error text;
|
||
- report only provider name, host, key count, model count, latency, status, and sanitized error class/code;
|
||
- use separate connect/read timeouts;
|
||
- cache results briefly so health checks do not consume excessive quota.
|
||
|
||
For very large pools, first audit every key with `/models`, then completion-probe every surviving key with bounded concurrency and incremental result saves. Do not infer that all keys work from one successful key.
|
||
|
||
## Durable pitfalls
|
||
|
||
- **Catalog/model mismatch:** A configured public ID may not exist even though the host and most keys are healthy. Compare exact IDs and add a verified alias only when the provider accepts it.
|
||
- **Wrong API prefix:** If the stored base already includes `/v1` or `/v4`, blindly appending `/v1/chat/completions` can produce `/v1/v1/...` or `/v4/v1/...` and false 404s. Probe both documented canonical forms before changing keys.
|
||
- **Reasoning false negatives:** HTTP 200 plus empty `content` at `max_tokens=8` is not proof of failure. Inspect `reasoning_content` and retry with 64–256 tokens.
|
||
- **Health says online while billing is dead:** `/models` often succeeds for keys whose completion returns 402/429. Semantic completion is the health gate.
|
||
- **One timeout label can hide several causes:** Verify DNS/TCP/TLS directly before blaming a provider. The actual defect may be an invalid model ID, dead keys, regional entitlement, or an overly short health-probe timeout.
|
||
|
||
## Reporting
|
||
|
||
Report counts, not secrets:
|
||
|
||
- provider/key/model inventory;
|
||
- catalog-authenticated keys;
|
||
- completion-verified keys;
|
||
- failure counts by status/reason;
|
||
- keys/providers removed or quarantined;
|
||
- final runtime providers online and end-to-end smoke results;
|
||
- any research limitation (for example, a public search backend unavailable) without treating it as a durable tool limitation.
|