Provider Key Fleet Audit and Semantic Health

Use this when a gateway has many OpenAI-compatible providers/keys and its admin panel reports misleading states such as ConnectTimeout, online, or a large healthy-key count.

Principle

A provider/key is production-healthy only when a representative POST /v1/chat/completions succeeds semantically. DNS, TLS, GET /v1/models == 200, or HTTP 200 with an empty/soft-error payload are not enough.

Safe audit sequence

Inventory provider name, exact base URL, number of keys, configured models, aliases, priority, and soft-error markers. Never print full keys; use an index plus a short SHA-256 fingerprint.
Attribute network failures separately:
- resolve DNS;
- measure TCP/TLS with an unauthenticated request;
- compare regional sibling hosts when official docs list them;
- distinguish connect timeout from read timeout and application errors.
Run GET {base}/v1/models for every key as an authentication/catalog pass.
Intersect configured model IDs with IDs actually returned by the catalog. A plausible but absent alias is a configuration bug, not a dead network.
Run a real completion for every key using a model known to exist on that exact route.
For reasoning models, use max_tokens 64–256 and accept non-empty content or reasoning_content. Tiny budgets such as 8 tokens can produce HTTP 200 with empty content and create false negatives.
Detect semantic failures inside HTTP 200 responses: recharge notices, insufficient-credit text, disabled-account messages, placeholder answers, and configured reject markers.
Classify precisely:
- 200 + usable assistant payload: usable;
- 200 + empty payload: retry with realistic output budget, then inconclusive/dead;
- 401: invalid key or wrong host/region;
- 403: valid identity but missing model entitlement;
- 402: insufficient balance;
- 429: quota/rate/billing state; quarantine rather than calling permanently invalid unless the message proves billing exhaustion;
- 404: often duplicated /v1, wrong API version, or wrong path;
- 5xx: provider/upstream failure; retry before pruning.
Prune unusable keys atomically from the private pool. Remove a whole provider only when no completion-verified key/model remains.
Restart safely, then verify the gateway's sanitized admin endpoint and one end-to-end completion per active provider family.
Sync the cleaned private pool to staging/production using secret-safe transport; verify file permissions and never copy it to a public repository.

Health endpoint design

Do not make provider health depend only on /models. A sanitized health probe should:

use one explicit configured model (after applying route aliases);
send a tiny but realistic completion (max_tokens around 32–64);
require non-empty content or reasoning_content;
reject configured soft-error text;
report only provider name, host, key count, model count, latency, status, and sanitized error class/code;
use separate connect/read timeouts;
cache results briefly so health checks do not consume excessive quota.

For very large pools, first audit every key with /models, then completion-probe every surviving key with bounded concurrency and incremental result saves. Do not infer that all keys work from one successful key.

Durable pitfalls

Catalog/model mismatch: A configured public ID may not exist even though the host and most keys are healthy. Compare exact IDs and add a verified alias only when the provider accepts it.
Wrong API prefix: If the stored base already includes /v1 or /v4, blindly appending /v1/chat/completions can produce /v1/v1/... or /v4/v1/... and false 404s. Probe both documented canonical forms before changing keys.
Reasoning false negatives: HTTP 200 plus empty content at max_tokens=8 is not proof of failure. Inspect reasoning_content and retry with 64–256 tokens.
Health says online while billing is dead: /models often succeeds for keys whose completion returns 402/429. Semantic completion is the health gate.
One timeout label can hide several causes: Verify DNS/TCP/TLS directly before blaming a provider. The actual defect may be an invalid model ID, dead keys, regional entitlement, or an overly short health-probe timeout.

Reporting

Report counts, not secrets:

provider/key/model inventory;
catalog-authenticated keys;
completion-verified keys;
failure counts by status/reason;
keys/providers removed or quarantined;
final runtime providers online and end-to-end smoke results;
any research limitation (for example, a public search backend unavailable) without treating it as a durable tool limitation.

4.7 KiB Raw Blame History Unescape Escape