---
name: multi-provider-gateway
description: Build and optimize FastAPI OpenAI-compatible gateways that aggregate multiple upstream AI providers (sh00t, Firepass, dashscope, etc.) into a single endpoint. Use when building API proxies, merging providers, or optimizing gateway latency.
---

# Multi-Provider API Gateway

## Trigger
Use when:
- Building an OpenAI-compatible proxy that fronts multiple upstream AI providers
- Merging provider catalogs (sh00t.host + Firepass + etc.) into one `/v1/models`
- Gateway latency is high — models timeout through proxy but work direct
- Need to strip rate limits / quotas / referral checks for admin keys
- Adding live/dead model status to a Telegram bot

## Architecture

```
Client → Gateway (:8080) → PRIMARY (provider A)
                          → FIREPASS (provider B)
                          → ... (N providers)
```

## Key Patterns

### 1. Model→Provider Routing (critical for speed)

Without routing, every request tries ALL providers in rotation — wasting time on providers that don't host the model. Add a `MODEL_PROVIDER_MAP`:

```python
MODEL_PROVIDER_MAP = {
    "gpt-5.6-sol": "primary",
    "qwen/qwen3.8-max-preview:free": "firepass",
    ...
}
```

Then filter upstreams before the request:

```python
def _filter_upstreams_for_model(upstreams, model):
    provider = MODEL_PROVIDER_MAP.get(model)
    if not provider:
        return upstreams
    prefix = _PROVIDER_PREFIX.get(provider, "")
    return [u for u in upstreams if prefix.lower() in u["url"].lower()]
```

**Result:** deepseek-v4-pro went from 45s timeout → 2.0s (22x speedup).

### 2. Shared httpx Client

Use a single `httpx.AsyncClient` instance across the FastAPI app — not per-request:

```python
app.state.http_client = httpx.AsyncClient(
    timeout=httpx.Timeout(300.0, connect=2.0, read=300.0),
    limits=httpx.Limits(max_connections=200, max_keepalive_connections=50),
    http2=True,
)
```

Pass it to `_handle_nonstream` and `_handle_stream` — reuse the pool.

**Pitfall:** HTTP/2 can cause issues with some upstreams. If models timeout through the gateway but work direct, test with `--http1.1` on curl. If that fixes it, set `http2=False` or use per-upstream config.

### 3. Strip All Limits for Admin

To disable rate limits, quotas, referral checks, and daily limits for admin keys:

```python
# In chat_completions handler:
# ── ALL LIMITS DISABLED for admin key ──
# (remove: quota check, rate-limit check, daily grok limit, referral bonus check)
```

Keep the auth check, model validation, and upstream routing — strip only the throttling.

### 4. Upstream Parsing from Env Vars

```python
UPSTREAM_PRIMARY_URL="https://sh00t.host"
UPSTREAM_PRIMARY_KEY="sk_..."
UPSTREAM_FIREPASS_URL="https://api.firepass.space"
UPSTREAM_FIREPASS_KEY="fp_live_..."
```

Parse with prefix convention:
```python
for prefix in ("PRIMARY", "FIREPASS", "THIRD"):
    url = os.environ.get(f"UPSTREAM_{prefix}_URL")
    key = os.environ.get(f"UPSTREAM_{prefix}_KEY")
    if url and key:
        upstreams.append({"url": url.rstrip("/"), "key": key})
```

### 5. Rotation with per-request start index

Use `threading.Lock` + `_rotation_index` to distribute load across upstreams that share the same model. Each request starts from the next provider in the ring.

### 6. Live/Dead Status in Telegram Bot

Add a `/status` command that probes all models through the gateway:

```python
async def status_cmd(update, context):
    for model, provider in MODEL_PROVIDER_MAP.items():
        resp = await client.post(
            f"{GATEWAY_URL}/v1/chat/completions",
            json={"model": model, "messages": [{"role": "user", "content": "1"}], "max_tokens": 5},
            timeout=15.0,
        )
        # categorize as live/dead
```

## Pitfalls

| Pitfall | Fix |
|---------|-----|
| HTTP/2 causes upstream timeouts | Test with `curl --http1.1`; if faster, disable http2 |
| Rotation wastes time on wrong providers | Add MODEL_PROVIDER_MAP |
| IndentationError when stripping limit blocks | Check all `if/else` blocks after deletion — keep 4-space consistency |
| PTB 22.6 doesn't support icon_custom_emoji_id | Upgrade to PTB 22.8+ |
| Ngrok URL changes on restart | Use paid ngrok ($8/mo) or cloudflared with own domain |

## Verification

```bash
# Gateway health
curl http://127.0.0.1:8080/health

# Model list
curl http://127.0.0.1:8080/v1/models

# Chat test
curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.6-sol","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'
```

## References

- `references/firepass-models.md` — Firepass model catalog and status
- `references/ngrok-tunnel-fallback.md` — Ngrok as port-forwarding fallback (linked from eni-operator)