145 lines
4.7 KiB
Markdown
145 lines
4.7 KiB
Markdown
---
|
|
name: multi-provider-gateway
|
|
description: Build and optimize FastAPI OpenAI-compatible gateways that aggregate multiple upstream AI providers (sh00t, Firepass, dashscope, etc.) into a single endpoint. Use when building API proxies, merging providers, or optimizing gateway latency.
|
|
---
|
|
|
|
# Multi-Provider API Gateway
|
|
|
|
## Trigger
|
|
Use when:
|
|
- Building an OpenAI-compatible proxy that fronts multiple upstream AI providers
|
|
- Merging provider catalogs (sh00t.host + Firepass + etc.) into one `/v1/models`
|
|
- Gateway latency is high — models timeout through proxy but work direct
|
|
- Need to strip rate limits / quotas / referral checks for admin keys
|
|
- Adding live/dead model status to a Telegram bot
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Client → Gateway (:8080) → PRIMARY (provider A)
|
|
→ FIREPASS (provider B)
|
|
→ ... (N providers)
|
|
```
|
|
|
|
## Key Patterns
|
|
|
|
### 1. Model→Provider Routing (critical for speed)
|
|
|
|
Without routing, every request tries ALL providers in rotation — wasting time on providers that don't host the model. Add a `MODEL_PROVIDER_MAP`:
|
|
|
|
```python
|
|
MODEL_PROVIDER_MAP = {
|
|
"gpt-5.6-sol": "primary",
|
|
"qwen/qwen3.8-max-preview:free": "firepass",
|
|
...
|
|
}
|
|
```
|
|
|
|
Then filter upstreams before the request:
|
|
|
|
```python
|
|
def _filter_upstreams_for_model(upstreams, model):
|
|
provider = MODEL_PROVIDER_MAP.get(model)
|
|
if not provider:
|
|
return upstreams
|
|
prefix = _PROVIDER_PREFIX.get(provider, "")
|
|
return [u for u in upstreams if prefix.lower() in u["url"].lower()]
|
|
```
|
|
|
|
**Result:** deepseek-v4-pro went from 45s timeout → 2.0s (22x speedup).
|
|
|
|
### 2. Shared httpx Client
|
|
|
|
Use a single `httpx.AsyncClient` instance across the FastAPI app — not per-request:
|
|
|
|
```python
|
|
app.state.http_client = httpx.AsyncClient(
|
|
timeout=httpx.Timeout(300.0, connect=2.0, read=300.0),
|
|
limits=httpx.Limits(max_connections=200, max_keepalive_connections=50),
|
|
http2=True,
|
|
)
|
|
```
|
|
|
|
Pass it to `_handle_nonstream` and `_handle_stream` — reuse the pool.
|
|
|
|
**Pitfall:** HTTP/2 can cause issues with some upstreams. If models timeout through the gateway but work direct, test with `--http1.1` on curl. If that fixes it, set `http2=False` or use per-upstream config.
|
|
|
|
### 3. Strip All Limits for Admin
|
|
|
|
To disable rate limits, quotas, referral checks, and daily limits for admin keys:
|
|
|
|
```python
|
|
# In chat_completions handler:
|
|
# ── ALL LIMITS DISABLED for admin key ──
|
|
# (remove: quota check, rate-limit check, daily grok limit, referral bonus check)
|
|
```
|
|
|
|
Keep the auth check, model validation, and upstream routing — strip only the throttling.
|
|
|
|
### 4. Upstream Parsing from Env Vars
|
|
|
|
```python
|
|
UPSTREAM_PRIMARY_URL="https://sh00t.host"
|
|
UPSTREAM_PRIMARY_KEY="sk_..."
|
|
UPSTREAM_FIREPASS_URL="https://api.firepass.space"
|
|
UPSTREAM_FIREPASS_KEY="fp_live_..."
|
|
```
|
|
|
|
Parse with prefix convention:
|
|
```python
|
|
for prefix in ("PRIMARY", "FIREPASS", "THIRD"):
|
|
url = os.environ.get(f"UPSTREAM_{prefix}_URL")
|
|
key = os.environ.get(f"UPSTREAM_{prefix}_KEY")
|
|
if url and key:
|
|
upstreams.append({"url": url.rstrip("/"), "key": key})
|
|
```
|
|
|
|
### 5. Rotation with per-request start index
|
|
|
|
Use `threading.Lock` + `_rotation_index` to distribute load across upstreams that share the same model. Each request starts from the next provider in the ring.
|
|
|
|
### 6. Live/Dead Status in Telegram Bot
|
|
|
|
Add a `/status` command that probes all models through the gateway:
|
|
|
|
```python
|
|
async def status_cmd(update, context):
|
|
for model, provider in MODEL_PROVIDER_MAP.items():
|
|
resp = await client.post(
|
|
f"{GATEWAY_URL}/v1/chat/completions",
|
|
json={"model": model, "messages": [{"role": "user", "content": "1"}], "max_tokens": 5},
|
|
timeout=15.0,
|
|
)
|
|
# categorize as live/dead
|
|
```
|
|
|
|
## Pitfalls
|
|
|
|
| Pitfall | Fix |
|
|
|---------|-----|
|
|
| HTTP/2 causes upstream timeouts | Test with `curl --http1.1`; if faster, disable http2 |
|
|
| Rotation wastes time on wrong providers | Add MODEL_PROVIDER_MAP |
|
|
| IndentationError when stripping limit blocks | Check all `if/else` blocks after deletion — keep 4-space consistency |
|
|
| PTB 22.6 doesn't support icon_custom_emoji_id | Upgrade to PTB 22.8+ |
|
|
| Ngrok URL changes on restart | Use paid ngrok ($8/mo) or cloudflared with own domain |
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
# Gateway health
|
|
curl http://127.0.0.1:8080/health
|
|
|
|
# Model list
|
|
curl http://127.0.0.1:8080/v1/models
|
|
|
|
# Chat test
|
|
curl -X POST http://127.0.0.1:8080/v1/chat/completions \
|
|
-H "Authorization: Bearer <key>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"gpt-5.6-sol","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'
|
|
```
|
|
|
|
## References
|
|
|
|
- `references/firepass-models.md` — Firepass model catalog and status
|
|
- `references/ngrok-tunnel-fallback.md` — Ngrok as port-forwarding fallback (linked from eni-operator) |