58 lines
5.0 KiB
Markdown
58 lines
5.0 KiB
Markdown
# OpenAI-Compatible Provider Pool: Live Audit and Safe Integration
|
||
|
||
Use this procedure when an operator supplies one or more base URLs and API keys and asks whether *all models really work* or whether a key should join a gateway pool.
|
||
|
||
## Core rule
|
||
|
||
`GET /v1/models == 200` proves only that a catalog is readable. It does **not** prove that a model can complete requests. A provider may advertise models while returning 401, 402, 429, 503, an empty assistant message, or a provider-specific routing error for every completion.
|
||
|
||
A model is eligible for a production catalog only after a real `POST /v1/chat/completions` returns HTTP 200 and a usable assistant payload. Validate through the final gateway too, because direct-provider success does not prove routing, auth, rank, quota, or response adaptation works end to end.
|
||
|
||
## Procedure
|
||
|
||
1. **Normalize the base URL carefully.** Test whether the supplied URL already ends in `/v1`; avoid accidentally calling `/v1/v1/...`. A token may be valid on one host and invalid on a sibling host, even when both expose the same model catalog.
|
||
2. **Fetch `/models` with the supplied credential.** Record status and model IDs, but label this as catalog discovery only.
|
||
3. **Probe completions, not just catalogs.** For every candidate public model, send a minimal non-stream request:
|
||
- one user message such as `Reply exactly OK`;
|
||
- `stream: false`;
|
||
- a realistic `max_tokens` (64–256 for reasoning models, not 8–16);
|
||
- provider-compatible temperature (some models accept only `1`).
|
||
4. **Classify results precisely:**
|
||
- `200 + non-empty content/reasoning_content`: completion-verified;
|
||
- `200 + empty content`: inconclusive; retry with a larger output budget and inspect the full payload;
|
||
- `401/403`: wrong host, invalid/deactivated credential, or authorization mismatch;
|
||
- `402`: credential authentic but lacks credits/entitlement;
|
||
- `429`: currently rate-limited; do not call it dead, but do not enable it as a dependable route without a successful retry;
|
||
- `400 invalid model`: advertised alias is not accepted by that endpoint/request shape;
|
||
- `503/no active upstream`: catalog-only route; do not enable it.
|
||
5. **Retry ambiguous models.** A reasoning model can consume a tiny token budget before emitting `content`. Retry 2–3 times with `max_tokens >= 128` before declaring it unusable.
|
||
6. **Build an allowlist by intersection:** public catalog ∩ provider `/models` ∩ live completion successes. Never attach a new credential as a generic route with an empty model list; that can make it receive unrelated traffic.
|
||
7. **Respect scarce-credit instructions.** Add low-balance credentials as explicit, low-priority failover routes. Do not round-robin them with primary routes. Prefer a separate provider record (for example `provider-reserve`) instead of mixing the key into a primary provider's key rotation.
|
||
8. **Verify routing order.** Inspect the gateway's computed route list for at least one covered model. The scarce reserve must appear after primary routes.
|
||
9. **End-to-end verification.** Restart/reload safely, call the final gateway with a temporary authorized test user/key, verify HTTP 200 and assistant output, then remove temporary DB rows and usage records.
|
||
10. **Prune honestly.** If the public catalog contains models that consistently return route-unavailable, remove them rather than reporting the catalog as healthy. Report `N/N completion-verified`, not merely `N listed`.
|
||
11. **Protect secrets.** Store keys only in ignored/private config or secret files. Do not print full keys, commit private pool files, copy them into publication repos, logs, reports, or backups without encryption/access controls.
|
||
|
||
## Routing pitfall
|
||
|
||
Many gateways order routes as `preferred + explicit + generic`. A newly added explicit reserve may still be attempted before generic environment routes, depending on the implementation. Verify the actual computed order rather than assuming append order in the private JSON is sufficient. If strict scarcity protection is required, add an explicit priority/tier field and sort by it, or keep the reserve outside rotation until earlier routes fail.
|
||
|
||
## Evidence to report
|
||
|
||
A useful operator report includes:
|
||
|
||
- catalog status and model count per host;
|
||
- real completion status for sampled or all public models;
|
||
- exact number of completion-verified public models;
|
||
- which credentials were enabled, rejected, or reserved;
|
||
- route priority and covered model count;
|
||
- final gateway health and end-to-end completion proof;
|
||
- tests and backup path;
|
||
- no secret values.
|
||
|
||
## Session-derived provider quirks (examples, not permanent truths)
|
||
|
||
- A Forge-prefixed token can be invalid on a Forge-branded gateway but valid on an EchoGate host. Always test the exact host/token pair.
|
||
- Catalog endpoints may return 200 even when completions return `Invalid or deactivated API key` or `No active upstream keys available`.
|
||
- Reasoning models may return HTTP 200 with empty `content` at very small `max_tokens`; retry with a larger budget and inspect `reasoning_content` before pruning.
|