51 lines
3.9 KiB
Markdown
51 lines
3.9 KiB
Markdown
# OpenAI-Compatible Provider Capability Probe
|
||
|
||
Use this for a read-only audit of an unfamiliar OpenAI-compatible LLM provider when the user wants the base URL, `/models`, free models, exact model IDs, and a proven chat completion.
|
||
|
||
## Read-only workflow
|
||
|
||
1. **Discover documentation without guessing paths.** Probe the host root, `/docs`, and `/openapi.json`. Prefer the OpenAPI document as route evidence. A vendor UI at `/models` may be HTML; the API route is commonly `/v1/models`.
|
||
2. **Establish the OpenAI base URL.** If OpenAPI exposes `/v1/chat/completions`, report the base as `https://HOST/v1`, not the full completion endpoint.
|
||
3. **Use the user-authorized key from context without displaying it.** Keep it in memory/environment, send it as `Authorization: Bearer ...`, and redact command output. Never persist the secret in a report, reference, log, or long-term memory.
|
||
4. **Query `/v1/models` with the key.** Record HTTP status, content type, total models, exact IDs, and owner metadata. Treat IDs returned by this live endpoint as authoritative for that key at that time.
|
||
5. **Classify catalog-free models conservatively.** A `-free` suffix proves catalog designation, not current usability or remaining quota. Say “catalog-designated free” until completion-tested.
|
||
6. **Completion-test representative models.** POST `/v1/chat/completions` with a deterministic prompt such as `Reply exactly PROVIDER_OK`, `stream:false`, and a realistic token budget.
|
||
7. **Validate semantic success, not only HTTP 200.** Require a non-empty assistant answer matching the task. Inspect `choices[0].message.content`, `finish_reason`, returned model, and usage. Some gateways return quota/top-up notices as successful `chat.completion` objects with HTTP 200 and zero token usage.
|
||
8. **Handle reasoning models correctly.** A reasoning model can consume a tiny `max_tokens` budget on hidden reasoning and return `content:null`. Retry once with 128–256 tokens before classifying it as broken.
|
||
9. **Report three separate sets:** (a) IDs advertised by `/models`, (b) catalog-designated free IDs, and (c) models proven live on the current key. Do not generalize from one live model to all free models.
|
||
10. **Leave the project untouched.** Use one-off in-memory probes or an existing external scratch script. Do not create or modify project files when the user requested research only.
|
||
|
||
## Minimal probe shape
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(api_key=KEY, base_url="https://HOST/v1")
|
||
models = client.models.list()
|
||
reply = client.chat.completions.create(
|
||
model="EXACT_ID_FROM_MODELS",
|
||
messages=[{"role": "user", "content": "Reply exactly PROVIDER_OK"}],
|
||
temperature=0,
|
||
max_tokens=128,
|
||
)
|
||
assert reply.choices[0].message.content == "PROVIDER_OK"
|
||
```
|
||
|
||
## Status taxonomy
|
||
|
||
- **Live:** HTTP 200, meaningful non-empty assistant content, plausible usage.
|
||
- **Reasoning-budget retry needed:** HTTP 200, `content:null`, completion/reasoning tokens consumed, tiny token cap.
|
||
- **Quota-gated:** HTTP 200 or 4xx with top-up/quota message; often zero usage.
|
||
- **Auth failure:** 401/403 attributable to key authorization.
|
||
- **Wrong route:** HTML content or UI shell where JSON was expected.
|
||
- **Advertised only:** present in `/models`, not completion-tested.
|
||
|
||
## Inferera observation (2026-07-29)
|
||
|
||
- OpenAI base: `https://api.inferera.com/v1`
|
||
- API models route: `GET /v1/models`; `/models` is a web UI.
|
||
- OpenAPI route evidence: `/openapi.json` describes `/v1/chat/completions`.
|
||
- The live catalog contained hundreds of models and dozens of `-free` IDs; exact counts are time-sensitive and must be refreshed.
|
||
- `ling-3.0-flash-free` produced a real completion after increasing `max_tokens` from 32 to 128; the response mapped it to an upstream InclusionAI model ID.
|
||
- Several other `-free` IDs returned a top-up/free-trial exhaustion message inside HTTP 200 `chat.completion` responses with zero usage. Therefore, never equate `-free`, presence in `/models`, or HTTP 200 with usable inference.
|