# Provider Key Fleet Audit and Semantic Health

Use this when a gateway has many OpenAI-compatible providers/keys and its admin panel reports misleading states such as `ConnectTimeout`, `online`, or a large healthy-key count.

## Principle

A provider/key is production-healthy only when a representative `POST /v1/chat/completions` succeeds semantically. DNS, TLS, `GET /v1/models == 200`, or HTTP 200 with an empty/soft-error payload are not enough.

## Safe audit sequence

1. Inventory provider name, exact base URL, number of keys, configured models, aliases, priority, and soft-error markers. Never print full keys; use an index plus a short SHA-256 fingerprint.
2. Attribute network failures separately:
   - resolve DNS;
   - measure TCP/TLS with an unauthenticated request;
   - compare regional sibling hosts when official docs list them;
   - distinguish connect timeout from read timeout and application errors.
3. Run `GET {base}/v1/models` for every key as an authentication/catalog pass.
4. Intersect configured model IDs with IDs actually returned by the catalog. A plausible but absent alias is a configuration bug, not a dead network.
5. Run a real completion for every key using a model known to exist on that exact route.
6. For reasoning models, use `max_tokens` 64–256 and accept non-empty `content` **or** `reasoning_content`. Tiny budgets such as 8 tokens can produce HTTP 200 with empty `content` and create false negatives.
7. Detect semantic failures inside HTTP 200 responses: recharge notices, insufficient-credit text, disabled-account messages, placeholder answers, and configured reject markers.
8. Classify precisely:
   - 200 + usable assistant payload: usable;
   - 200 + empty payload: retry with realistic output budget, then inconclusive/dead;
   - 401: invalid key or wrong host/region;
   - 403: valid identity but missing model entitlement;
   - 402: insufficient balance;
   - 429: quota/rate/billing state; quarantine rather than calling permanently invalid unless the message proves billing exhaustion;
   - 404: often duplicated `/v1`, wrong API version, or wrong path;
   - 5xx: provider/upstream failure; retry before pruning.
9. Prune unusable keys atomically from the private pool. Remove a whole provider only when no completion-verified key/model remains.
10. Restart safely, then verify the gateway's sanitized admin endpoint and one end-to-end completion per active provider family.
11. Sync the cleaned private pool to staging/production using secret-safe transport; verify file permissions and never copy it to a public repository.

## Health endpoint design

Do not make provider health depend only on `/models`. A sanitized health probe should:

- use one explicit configured model (after applying route aliases);
- send a tiny but realistic completion (`max_tokens` around 32–64);
- require non-empty `content` or `reasoning_content`;
- reject configured soft-error text;
- report only provider name, host, key count, model count, latency, status, and sanitized error class/code;
- use separate connect/read timeouts;
- cache results briefly so health checks do not consume excessive quota.

For very large pools, first audit every key with `/models`, then completion-probe every surviving key with bounded concurrency and incremental result saves. Do not infer that all keys work from one successful key.

## Durable pitfalls

- **Catalog/model mismatch:** A configured public ID may not exist even though the host and most keys are healthy. Compare exact IDs and add a verified alias only when the provider accepts it.
- **Wrong API prefix:** If the stored base already includes `/v1` or `/v4`, blindly appending `/v1/chat/completions` can produce `/v1/v1/...` or `/v4/v1/...` and false 404s. Probe both documented canonical forms before changing keys.
- **Reasoning false negatives:** HTTP 200 plus empty `content` at `max_tokens=8` is not proof of failure. Inspect `reasoning_content` and retry with 64–256 tokens.
- **Health says online while billing is dead:** `/models` often succeeds for keys whose completion returns 402/429. Semantic completion is the health gate.
- **One timeout label can hide several causes:** Verify DNS/TCP/TLS directly before blaming a provider. The actual defect may be an invalid model ID, dead keys, regional entitlement, or an overly short health-probe timeout.

## Reporting

Report counts, not secrets:

- provider/key/model inventory;
- catalog-authenticated keys;
- completion-verified keys;
- failure counts by status/reason;
- keys/providers removed or quarantined;
- final runtime providers online and end-to-end smoke results;
- any research limitation (for example, a public search backend unavailable) without treating it as a durable tool limitation.