autoreg-skills-max: initial public release (restored, redacted, hardened)

skills/multi-provider-gateway/SKILL.md

@@ -0,0 +1,145 @@
+---
+name: multi-provider-gateway
+description: Build and optimize FastAPI OpenAI-compatible gateways that aggregate multiple upstream AI providers (sh00t, Firepass, dashscope, etc.) into a single endpoint. Use when building API proxies, merging providers, or optimizing gateway latency.
+---
+
+# Multi-Provider API Gateway
+
+## Trigger
+Use when:
+- Building an OpenAI-compatible proxy that fronts multiple upstream AI providers
+- Merging provider catalogs (sh00t.host + Firepass + etc.) into one `/v1/models`
+- Gateway latency is high — models timeout through proxy but work direct
+- Need to strip rate limits / quotas / referral checks for admin keys
+- Adding live/dead model status to a Telegram bot
+
+## Architecture
+
+```
+Client → Gateway (:8080) → PRIMARY (provider A)
+                          → FIREPASS (provider B)
+                          → ... (N providers)
+```
+
+## Key Patterns
+
+### 1. Model→Provider Routing (critical for speed)
+
+Without routing, every request tries ALL providers in rotation — wasting time on providers that don't host the model. Add a `MODEL_PROVIDER_MAP`:
+
+```python
+MODEL_PROVIDER_MAP = {
+    "gpt-5.6-sol": "primary",
+    "qwen/qwen3.8-max-preview:free": "firepass",
+    ...
+}
+```
+
+Then filter upstreams before the request:
+
+```python
+def _filter_upstreams_for_model(upstreams, model):
+    provider = MODEL_PROVIDER_MAP.get(model)
+    if not provider:
+        return upstreams
+    prefix = _PROVIDER_PREFIX.get(provider, "")
+    return [u for u in upstreams if prefix.lower() in u["url"].lower()]
+```
+
+**Result:** deepseek-v4-pro went from 45s timeout → 2.0s (22x speedup).
+
+### 2. Shared httpx Client
+
+Use a single `httpx.AsyncClient` instance across the FastAPI app — not per-request:
+
+```python
+app.state.http_client = httpx.AsyncClient(
+    timeout=httpx.Timeout(300.0, connect=2.0, read=300.0),
+    limits=httpx.Limits(max_connections=200, max_keepalive_connections=50),
+    http2=True,
+)
+```
+
+Pass it to `_handle_nonstream` and `_handle_stream` — reuse the pool.
+
+**Pitfall:** HTTP/2 can cause issues with some upstreams. If models timeout through the gateway but work direct, test with `--http1.1` on curl. If that fixes it, set `http2=False` or use per-upstream config.
+
+### 3. Strip All Limits for Admin
+
+To disable rate limits, quotas, referral checks, and daily limits for admin keys:
+
+```python
+# In chat_completions handler:
+# ── ALL LIMITS DISABLED for admin key ──
+# (remove: quota check, rate-limit check, daily grok limit, referral bonus check)
+```
+
+Keep the auth check, model validation, and upstream routing — strip only the throttling.
+
+### 4. Upstream Parsing from Env Vars
+
+```python
+UPSTREAM_PRIMARY_URL="https://sh00t.host"
+UPSTREAM_PRIMARY_KEY="sk_..."
+UPSTREAM_FIREPASS_URL="https://api.firepass.space"
+UPSTREAM_FIREPASS_KEY="fp_live_..."
+```
+
+Parse with prefix convention:
+```python
+for prefix in ("PRIMARY", "FIREPASS", "THIRD"):
+    url = os.environ.get(f"UPSTREAM_{prefix}_URL")
+    key = os.environ.get(f"UPSTREAM_{prefix}_KEY")
+    if url and key:
+        upstreams.append({"url": url.rstrip("/"), "key": key})
+```
+
+### 5. Rotation with per-request start index
+
+Use `threading.Lock` + `_rotation_index` to distribute load across upstreams that share the same model. Each request starts from the next provider in the ring.
+
+### 6. Live/Dead Status in Telegram Bot
+
+Add a `/status` command that probes all models through the gateway:
+
+```python
+async def status_cmd(update, context):
+    for model, provider in MODEL_PROVIDER_MAP.items():
+        resp = await client.post(
+            f"{GATEWAY_URL}/v1/chat/completions",
+            json={"model": model, "messages": [{"role": "user", "content": "1"}], "max_tokens": 5},
+            timeout=15.0,
+        )
+        # categorize as live/dead
+```
+
+## Pitfalls
+
+| Pitfall | Fix |
+|---------|-----|
+| HTTP/2 causes upstream timeouts | Test with `curl --http1.1`; if faster, disable http2 |
+| Rotation wastes time on wrong providers | Add MODEL_PROVIDER_MAP |
+| IndentationError when stripping limit blocks | Check all `if/else` blocks after deletion — keep 4-space consistency |
+| PTB 22.6 doesn't support icon_custom_emoji_id | Upgrade to PTB 22.8+ |
+| Ngrok URL changes on restart | Use paid ngrok ($8/mo) or cloudflared with own domain |
+
+## Verification
+
+```bash
+# Gateway health
+curl http://127.0.0.1:8080/health
+
+# Model list
+curl http://127.0.0.1:8080/v1/models
+
+# Chat test
+curl -X POST http://127.0.0.1:8080/v1/chat/completions \
+  -H "Authorization: Bearer <key>" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"gpt-5.6-sol","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'
+```
+
+## References
+
+- `references/firepass-models.md` — Firepass model catalog and status
+- `references/ngrok-tunnel-fallback.md` — Ngrok as port-forwarding fallback (linked from eni-operator)