autoreg-skills-max/skills/account-validation/references/provider-key-fleet-audit-and-health.md

Provider Key Fleet Audit and Semantic Health

Use this when a gateway has many OpenAI-compatible providers/keys and its admin panel reports misleading states such as ConnectTimeout, online, or a large healthy-key count.

Principle

A provider/key is production-healthy only when a representative POST /v1/chat/completions succeeds semantically. DNS, TLS, GET /v1/models == 200, or HTTP 200 with an empty/soft-error payload are not enough.

Safe audit sequence

1. Inventory provider name, exact base URL, number of keys, configured models, aliases, priority, and soft-error markers. Never print full keys; use an index plus a short SHA-256 fingerprint.
2. Attribute network failures separately:
   • resolve DNS;
   • measure TCP/TLS with an unauthenticated request;
   • compare regional sibling hosts when official docs list them;
   • distinguish connect timeout from read timeout and application errors.
3. Run GET {base}/v1/models for every key as an authentication/catalog pass.
4. Intersect configured model IDs with IDs actually returned by the catalog. A plausible but absent alias is a configuration bug, not a dead network.
5. Run a real completion for every key using a model known to exist on that exact route.
6. For reasoning models, use max_tokens 64–256 and accept non-empty content or reasoning_content. Tiny budgets such as 8 tokens can produce HTTP 200 with empty content and create false negatives.
7. Detect semantic failures inside HTTP 200 responses: recharge notices, insufficient-credit text, disabled-account messages, placeholder answers, and configured reject markers.
8. Classify precisely:
   • 200 + usable assistant payload: usable;
   • 200 + empty payload: retry with realistic output budget, then inconclusive/dead;
   • 401: invalid key or wrong host/region;
   • 403: valid identity but missing model entitlement;
   • 402: insufficient balance;
   • 429: quota/rate/billing state; quarantine rather than calling permanently invalid unless the message proves billing exhaustion;
   • 404: often duplicated /v1, wrong API version, or wrong path;
   • 5xx: provider/upstream failure; retry before pruning.
9. Prune unusable keys atomically from the private pool. Remove a whole provider only when no completion-verified key/model remains.
10. Restart safely, then verify the gateway's sanitized admin endpoint and one end-to-end completion per active provider family.
11. Sync the cleaned private pool to staging/production using secret-safe transport; verify file permissions and never copy it to a public repository.

Health endpoint design

Do not make provider health depend only on /models. A sanitized health probe should:

• use one explicit configured model (after applying route aliases);
• send a tiny but realistic completion (max_tokens around 32–64);
• require non-empty content or reasoning_content;
• reject configured soft-error text;
• report only provider name, host, key count, model count, latency, status, and sanitized error class/code;
• use separate connect/read timeouts;
• cache results briefly so health checks do not consume excessive quota.

For very large pools, first audit every key with /models, then completion-probe every surviving key with bounded concurrency and incremental result saves. Do not infer that all keys work from one successful key.

Durable pitfalls

• Catalog/model mismatch: A configured public ID may not exist even though the host and most keys are healthy. Compare exact IDs and add a verified alias only when the provider accepts it.
• Wrong API prefix: If the stored base already includes /v1 or /v4, blindly appending /v1/chat/completions can produce /v1/v1/... or /v4/v1/... and false 404s. Probe both documented canonical forms before changing keys.
• Reasoning false negatives: HTTP 200 plus empty content at max_tokens=8 is not proof of failure. Inspect reasoning_content and retry with 64–256 tokens.
• Health says online while billing is dead: /models often succeeds for keys whose completion returns 402/429. Semantic completion is the health gate.
• One timeout label can hide several causes: Verify DNS/TCP/TLS directly before blaming a provider. The actual defect may be an invalid model ID, dead keys, regional entitlement, or an overly short health-probe timeout.

Reporting

Report counts, not secrets:

• provider/key/model inventory;
• catalog-authenticated keys;
• completion-verified keys;
• failure counts by status/reason;
• keys/providers removed or quarantined;
• final runtime providers online and end-to-end smoke results;
• any research limitation (for example, a public search backend unavailable) without treating it as a durable tool limitation.