autoreg-skills-max/skills/multi-provider-gateway/SKILL.md

name, description

name	description
multi-provider-gateway	Build and optimize FastAPI OpenAI-compatible gateways that aggregate multiple upstream AI providers (sh00t, Firepass, dashscope, etc.) into a single endpoint. Use when building API proxies, merging providers, or optimizing gateway latency.

Multi-Provider API Gateway

Trigger

Use when:

• Building an OpenAI-compatible proxy that fronts multiple upstream AI providers
• Merging provider catalogs (sh00t.host + Firepass + etc.) into one /v1/models
• Gateway latency is high — models timeout through proxy but work direct
• Need to strip rate limits / quotas / referral checks for admin keys
• Adding live/dead model status to a Telegram bot

Architecture

Client → Gateway (:8080) → PRIMARY (provider A)
                          → FIREPASS (provider B)
                          → ... (N providers)

Key Patterns

1. Model→Provider Routing (critical for speed)

Without routing, every request tries ALL providers in rotation — wasting time on providers that don't host the model. Add a MODEL_PROVIDER_MAP:

MODEL_PROVIDER_MAP = {
    "gpt-5.6-sol": "primary",
    "qwen/qwen3.8-max-preview:free": "firepass",
    ...
}

Then filter upstreams before the request:

def _filter_upstreams_for_model(upstreams, model):
    provider = MODEL_PROVIDER_MAP.get(model)
    if not provider:
        return upstreams
    prefix = _PROVIDER_PREFIX.get(provider, "")
    return [u for u in upstreams if prefix.lower() in u["url"].lower()]

Result: deepseek-v4-pro went from 45s timeout → 2.0s (22x speedup).

2. Shared httpx Client

Use a single httpx.AsyncClient instance across the FastAPI app — not per-request:

app.state.http_client = httpx.AsyncClient(
    timeout=httpx.Timeout(300.0, connect=2.0, read=300.0),
    limits=httpx.Limits(max_connections=200, max_keepalive_connections=50),
    http2=True,
)

Pass it to _handle_nonstream and _handle_stream — reuse the pool.

Pitfall: HTTP/2 can cause issues with some upstreams. If models timeout through the gateway but work direct, test with --http1.1 on curl. If that fixes it, set http2=False or use per-upstream config.

3. Strip All Limits for Admin

To disable rate limits, quotas, referral checks, and daily limits for admin keys:

# In chat_completions handler:
# ── ALL LIMITS DISABLED for admin key ──
# (remove: quota check, rate-limit check, daily grok limit, referral bonus check)

Keep the auth check, model validation, and upstream routing — strip only the throttling.

4. Upstream Parsing from Env Vars

UPSTREAM_PRIMARY_URL="https://sh00t.host"
UPSTREAM_PRIMARY_KEY="sk_..."
UPSTREAM_FIREPASS_URL="https://api.firepass.space"
UPSTREAM_FIREPASS_KEY="fp_live_..."

Parse with prefix convention:

for prefix in ("PRIMARY", "FIREPASS", "THIRD"):
    url = os.environ.get(f"UPSTREAM_{prefix}_URL")
    key = os.environ.get(f"UPSTREAM_{prefix}_KEY")
    if url and key:
        upstreams.append({"url": url.rstrip("/"), "key": key})

5. Rotation with per-request start index

Use threading.Lock + _rotation_index to distribute load across upstreams that share the same model. Each request starts from the next provider in the ring.

6. Live/Dead Status in Telegram Bot

Add a /status command that probes all models through the gateway:

async def status_cmd(update, context):
    for model, provider in MODEL_PROVIDER_MAP.items():
        resp = await client.post(
            f"{GATEWAY_URL}/v1/chat/completions",
            json={"model": model, "messages": [{"role": "user", "content": "1"}], "max_tokens": 5},
            timeout=15.0,
        )
        # categorize as live/dead

Pitfalls

Pitfall	Fix
HTTP/2 causes upstream timeouts	Test with curl --http1.1; if faster, disable http2
Rotation wastes time on wrong providers	Add MODEL_PROVIDER_MAP
IndentationError when stripping limit blocks	Check all if/else blocks after deletion — keep 4-space consistency
PTB 22.6 doesn't support icon_custom_emoji_id	Upgrade to PTB 22.8+
Ngrok URL changes on restart	Use paid ngrok ($8/mo) or cloudflared with own domain

Verification

# Gateway health
curl http://127.0.0.1:8080/health

# Model list
curl http://127.0.0.1:8080/v1/models

# Chat test
curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.6-sol","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'

References

• references/firepass-models.md — Firepass model catalog and status
• references/ngrok-tunnel-fallback.md — Ngrok as port-forwarding fallback (linked from eni-operator)