autoreg-skills-max/skills/account-validation/references/openai-compatible-provider-capability-probe.md

OpenAI-Compatible Provider Capability Probe

Use this for a read-only audit of an unfamiliar OpenAI-compatible LLM provider when the user wants the base URL, /models, free models, exact model IDs, and a proven chat completion.

Read-only workflow

1. Discover documentation without guessing paths. Probe the host root, /docs, and /openapi.json. Prefer the OpenAPI document as route evidence. A vendor UI at /models may be HTML; the API route is commonly /v1/models.
2. Establish the OpenAI base URL. If OpenAPI exposes /v1/chat/completions, report the base as https://HOST/v1, not the full completion endpoint.
3. Use the user-authorized key from context without displaying it. Keep it in memory/environment, send it as Authorization: Bearer ..., and redact command output. Never persist the secret in a report, reference, log, or long-term memory.
4. Query /v1/models with the key. Record HTTP status, content type, total models, exact IDs, and owner metadata. Treat IDs returned by this live endpoint as authoritative for that key at that time.
5. Classify catalog-free models conservatively. A -free suffix proves catalog designation, not current usability or remaining quota. Say “catalog-designated free” until completion-tested.
6. Completion-test representative models. POST /v1/chat/completions with a deterministic prompt such as Reply exactly PROVIDER_OK, stream:false, and a realistic token budget.
7. Validate semantic success, not only HTTP 200. Require a non-empty assistant answer matching the task. Inspect choices[0].message.content, finish_reason, returned model, and usage. Some gateways return quota/top-up notices as successful chat.completion objects with HTTP 200 and zero token usage.
8. Handle reasoning models correctly. A reasoning model can consume a tiny max_tokens budget on hidden reasoning and return content:null. Retry once with 128–256 tokens before classifying it as broken.
9. Report three separate sets: (a) IDs advertised by /models, (b) catalog-designated free IDs, and (c) models proven live on the current key. Do not generalize from one live model to all free models.
10. Leave the project untouched. Use one-off in-memory probes or an existing external scratch script. Do not create or modify project files when the user requested research only.

Minimal probe shape

from openai import OpenAI

client = OpenAI(api_key=KEY, base_url="https://HOST/v1")
models = client.models.list()
reply = client.chat.completions.create(
    model="EXACT_ID_FROM_MODELS",
    messages=[{"role": "user", "content": "Reply exactly PROVIDER_OK"}],
    temperature=0,
    max_tokens=128,
)
assert reply.choices[0].message.content == "PROVIDER_OK"

Status taxonomy

• Live: HTTP 200, meaningful non-empty assistant content, plausible usage.
• Reasoning-budget retry needed: HTTP 200, content:null, completion/reasoning tokens consumed, tiny token cap.
• Quota-gated: HTTP 200 or 4xx with top-up/quota message; often zero usage.
• Auth failure: 401/403 attributable to key authorization.
• Wrong route: HTML content or UI shell where JSON was expected.
• Advertised only: present in /models, not completion-tested.

Inferera observation (2026-07-29)

• OpenAI base: https://api.inferera.com/v1
• API models route: GET /v1/models; /models is a web UI.
• OpenAPI route evidence: /openapi.json describes /v1/chat/completions.
• The live catalog contained hundreds of models and dozens of -free IDs; exact counts are time-sensitive and must be refreshed.
• ling-3.0-flash-free produced a real completion after increasing max_tokens from 32 to 128; the response mapped it to an upstream InclusionAI model ID.
• Several other -free IDs returned a top-up/free-trial exhaustion message inside HTTP 200 chat.completion responses with zero usage. Therefore, never equate -free, presence in /models, or HTTP 200 with usable inference.