vlad/autoreg-skills-max
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

autoreg-skills-max: initial public release (restored, redacted, hardened)

Browse Source
This commit is contained in:
baradok60915d52e8425
2026-10-20 02:16:32 +03:00
commit f58eafeb48
565 changed files with 60064 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

67
skills/account-validation/references/provider-key-fleet-audit-and-health.md Normal file
View File

@@ -0,0 +1,67 @@
# Provider Key Fleet Audit and Semantic Health
Use this when a gateway has many OpenAI-compatible providers/keys and its admin panel reports misleading states such as `ConnectTimeout`, `online`, or a large healthy-key count.
## Principle
A provider/key is production-healthy only when a representative `POST /v1/chat/completions` succeeds semantically. DNS, TLS, `GET /v1/models == 200`, or HTTP 200 with an empty/soft-error payload are not enough.
## Safe audit sequence
1. Inventory provider name, exact base URL, number of keys, configured models, aliases, priority, and soft-error markers. Never print full keys; use an index plus a short SHA-256 fingerprint.
2. Attribute network failures separately:
- resolve DNS;
- measure TCP/TLS with an unauthenticated request;
- compare regional sibling hosts when official docs list them;
- distinguish connect timeout from read timeout and application errors.
3. Run `GET {base}/v1/models` for every key as an authentication/catalog pass.
4. Intersect configured model IDs with IDs actually returned by the catalog. A plausible but absent alias is a configuration bug, not a dead network.
5. Run a real completion for every key using a model known to exist on that exact route.
6. For reasoning models, use `max_tokens` 64–256 and accept non-empty `content` **or** `reasoning_content`. Tiny budgets such as 8 tokens can produce HTTP 200 with empty `content` and create false negatives.
7. Detect semantic failures inside HTTP 200 responses: recharge notices, insufficient-credit text, disabled-account messages, placeholder answers, and configured reject markers.
8. Classify precisely:
- 200 + usable assistant payload: usable;
- 200 + empty payload: retry with realistic output budget, then inconclusive/dead;
- 401: invalid key or wrong host/region;
- 403: valid identity but missing model entitlement;
- 402: insufficient balance;
- 429: quota/rate/billing state; quarantine rather than calling permanently invalid unless the message proves billing exhaustion;
- 404: often duplicated `/v1`, wrong API version, or wrong path;
- 5xx: provider/upstream failure; retry before pruning.
9. Prune unusable keys atomically from the private pool. Remove a whole provider only when no completion-verified key/model remains.
10. Restart safely, then verify the gateway's sanitized admin endpoint and one end-to-end completion per active provider family.
11. Sync the cleaned private pool to staging/production using secret-safe transport; verify file permissions and never copy it to a public repository.
## Health endpoint design
Do not make provider health depend only on `/models`. A sanitized health probe should:
- use one explicit configured model (after applying route aliases);
- send a tiny but realistic completion (`max_tokens` around 32–64);
- require non-empty `content` or `reasoning_content`;
- reject configured soft-error text;
- report only provider name, host, key count, model count, latency, status, and sanitized error class/code;
- use separate connect/read timeouts;
- cache results briefly so health checks do not consume excessive quota.
For very large pools, first audit every key with `/models`, then completion-probe every surviving key with bounded concurrency and incremental result saves. Do not infer that all keys work from one successful key.
## Durable pitfalls
- **Catalog/model mismatch:** A configured public ID may not exist even though the host and most keys are healthy. Compare exact IDs and add a verified alias only when the provider accepts it.
- **Wrong API prefix:** If the stored base already includes `/v1` or `/v4`, blindly appending `/v1/chat/completions` can produce `/v1/v1/...` or `/v4/v1/...` and false 404s. Probe both documented canonical forms before changing keys.
- **Reasoning false negatives:** HTTP 200 plus empty `content` at `max_tokens=8` is not proof of failure. Inspect `reasoning_content` and retry with 64–256 tokens.
- **Health says online while billing is dead:** `/models` often succeeds for keys whose completion returns 402/429. Semantic completion is the health gate.
- **One timeout label can hide several causes:** Verify DNS/TCP/TLS directly before blaming a provider. The actual defect may be an invalid model ID, dead keys, regional entitlement, or an overly short health-probe timeout.
## Reporting
Report counts, not secrets:
- provider/key/model inventory;
- catalog-authenticated keys;
- completion-verified keys;
- failure counts by status/reason;
- keys/providers removed or quarantined;
- final runtime providers online and end-to-end smoke results;
- any research limitation (for example, a public search backend unavailable) without treating it as a durable tool limitation.