--- name: reseller-api-gateway description: Build anonymous OpenAI-compatible API gateways for reselling — hide upstream provider, alias model IDs, strip metadata, apply speed optimizations (HTTP/2, connection pool, gzip, TCP_NODELAY), and deploy via immutable releases. triggers: - "resell API" - "hide provider" - "anonymous gateway" - "opaque proxy" - "api reseller" - "model aliasing" - "[REDACTED-NAME] API" - "apioid gateway" - "speed up API" --- # Reseller API Gateway Build an anonymous OpenAI-compatible API gateway that proxies an upstream OpenAI-compatible backend while hiding its identity — for reselling. Model IDs are aliased via HMAC, metadata is stripped, and speed optimizations eliminate unnecessary latency. ## Architecture ``` Client (any OpenAI client) → Nginx (TLS termination, HTTP/2, gzip, error masking) → Gateway (Python http.server on localhost, connection pool) → Upstream (echogate.one, OpenRouter, any /v1 backend) ``` ## Core Patterns ### 1. Model ID Aliasing (HMAC-SHA256) Every upstream model ID gets a deterministic public alias. Upstream never sees the alias; clients never see the real ID. ```python import hmac, hashlib def public_model_id(private_id: str, secret: str) -> str: digest = hmac.new(secret.encode(), private_id.encode(), hashlib.sha256).hexdigest() return f"rb-{digest[:20]}" ``` - Private IDs ONLY accepted on upstream requests (gateway looks up alias → real ID) - Direct private IDs in client requests → HTTP 400 - Forbidden models filtered in catalog (e.g. `gpt-5-5` fragment match) ### 2. Metadata Sanitization Strip all upstream metadata from responses: ```python PRIVATE_METADATA_KEYS = { "backend", "provider", "provider_id", "provider_name", "route", "routing", "system_fingerprint", "upstream", "upstream_id", "upstream_model", "upstream_provider", } ``` - `owned_by` always `[REDACTED-NAME]` (or user's brand) - `system_fingerprint` removed - `model` field rewritten to public alias in both buffered and SSE responses - All hop-by-hop headers stripped (`Connection`, `Keep-Alive`, etc.) ### 3. Connection Pool with Retry Thread-safe HTTPS connection pool — eliminates TCP+TLS handshake per request: ```python class ConnectionPool: def __init__(self, host, port, scheme, timeout, max_idle=4): self._lock = threading.Lock() self._idle: list[http.client.HTTPConnection] = [] self._conn_type = HTTPSConnection if scheme == "https" else HTTPConnection def _create(self): conn = self._conn_type(host, port, timeout=timeout) conn.connect() conn.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) return conn def get(self): with self._lock: while self._idle: conn = self._idle.pop() # select() health check: if socket readable, connection is dead r, _, _ = select.select([conn.sock], [], [], 0) if not r: return conn conn.close() return self._create() def put(self, conn): with self._lock: if len(self._idle) < self._max_idle and conn.sock is not None: self._idle.append(conn) return conn.close() ``` **Retry pattern** — stale connections fail on first use, retry with fresh: ```python def upstream_request(self, method, suffix, body): for attempt in (1, 2): conn = self._pool.get() try: conn.request(method, path, body=body, headers=headers) response = conn.getresponse() body = response.read() if response.status < 500: self._pool.put(conn) # healthy → return to pool else: conn.close() # 5xx → discard return response.status, response.getheaders(), body except (HTTPException, OSError, TimeoutError): conn.close() # transport error → discard if attempt == 2: raise # retry exhausted ``` - Only `5xx` and transport errors discard connections; `4xx` connections go back to pool - Pre-warm 2 connections + force catalog fetch on startup ### 4. Speed Optimizations (Nginx) ```nginx server { listen 443 ssl; http2 on; # Multiplexing gzip on; gzip_min_length 256; gzip_types application/json text/event-stream; gzip_comp_level 1; location / { proxy_pass http://127.0.0.1:18181; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_read_timeout 600s; proxy_request_buffering off; proxy_buffering off; } # Mask ALL nginx/upstream errors as neutral JSON error_page 400 408 411 414 431 = @invalid_request; error_page 405 = @method_not_allowed; error_page 500 502 503 504 = @service_unavailable; } ``` ### 5. Anti-Fingerprint - **UA rotation**: rotate through `OpenAI/Python`, `Go-http-client/2.0`, `python-requests/2.32`, `curl/8.12` — no custom User-Agent - **Server header**: stripped via `more_clear_headers Server;` or `server_tokens off;` - **Error responses**: always neutral JSON, never upstream error bodies or HTML - **Unknown routes**: 404 JSON, never nginx default HTML - **Echo/legacy paths**: explicit 404 (e.g. `^~ /echo/` → JSON 404) ### 6. Immutable Release Deployment Each deploy creates a timestamped release directory with symlink: ```bash RELEASE_ID=$(date +%Y%m%d_%H%M%S) mkdir -p /opt/apioid-echo/releases/${RELEASE_ID} scp gateway.py root@host:/opt/apioid-echo/releases/${RELEASE_ID}/ ssh root@host " ln -sfn /opt/apioid-echo/releases/${RELEASE_ID} /opt/apioid-echo/releases/current systemctl restart apioid-echo-gateway.service " ``` - Backup before deploy: copy current → `/var/backups/apioid-echo/gateway.before-${RELEASE_ID}.py` - Verify: `systemctl is-active`, `NRestarts=0`, live smoke with curl - Env file: `/etc/apioid-echo/gateway.env` chmod 600 (keys + `MODEL_ALIAS_SECRET`) ## Error Masking All errors → neutral JSON, never HTML or upstream metadata: | Upstream status | Client response | |---|---| | 429 | `{"error":{"message":"Rate limit exceeded","code":"rate_limit_exceeded"}}` | | 402 | `{"error":{"message":"Daily usage limit reached","code":"daily_limit_reached"}}` | | 400/404/409/422 | `{"error":{"message":"Request was rejected","code":"request_rejected"}}` | | Other 5xx | `{"error":{"message":"Service temporarily unavailable","code":"service_unavailable"}}` | | Auth fail | `{"error":{"message":"Invalid or missing API key","code":"invalid_api_key"}}` | ## Forbidden Models Block specific model fragments from reaching upstream: ```python FORBIDDEN_MODEL_FRAGMENT = "gpt-5-5" @staticmethod def _is_allowed(model_id: str) -> bool: normalized = re.sub(r"[\s._/]+", "-", model_id.casefold()) return FORBIDDEN_MODEL_FRAGMENT not in normalized ``` ## Transparency Audit After deploying or modifying the gateway, verify it is a **pure passthrough** — no system prompt injection, no content modification, no metadata leak. ### 1. System Prompt Injection Detection Send a system prompt with a **unique marker string** through the gateway. The model must echo it exactly — any deviation proves injection/modification: ```bash curl -s -X POST https:///v1/chat/completions \ -H "Content-Type: application/json" -H "x-api-key: $KEY" \ -d '{"model":"claude-sonnet-5","messages":[ {"role":"system","content":"Answer every message with exactly: PROBE_OK"}, {"role":"user","content":"answer"} ],"max_tokens":20,"temperature":0}' | jq '.choices[0].message.content' # Expected: "PROBE_OK" — verbatim, no added words ``` For Anthropic format, same test on `/v1/messages` with `system` field. ### 2. Content Integrity — Gateway vs Upstream 1:1 Send identical requests to gateway AND directly to upstream. Responses must match: ```bash BODY='{"model":"claude-sonnet-5","messages":[{"role":"user","content":"hi"}],"max_tokens":20,"temperature":0}' GATEWAY=$(curl -s -X POST https:///v1/chat/completions \ -H "Content-Type: application/json" -H "x-api-key: $KEY" -d "$BODY") UPSTREAM=$(curl -s -X POST https:///v1/chat/completions \ -H "Content-Type: application/json" -H "Authorization: Bearer $UK" -d "$BODY") diff <(echo "$GATEWAY" | jq -S '.choices[0].message.content') \ <(echo "$UPSTREAM" | jq -S '.choices[0].message.content') ``` For dual-key setups, test BOTH upstream keys independently — all three sources (key1, key2, gateway) must agree. ### 3. Model List Integrity Gateway `/v1/models` count must match upstream count. If filtering is intentional, document it: ```bash GC=$(curl -s -H "x-api-key: $KEY" https:///v1/models | jq '.data | length') UC=$(curl -s -H "Authorization: Bearer $UK" https:///v1/models | jq '.data | length') [ "$GC" -eq "$UC" ] && echo "OK: $GC models" || echo "MISMATCH: gateway=$GC upstream=$UC" ``` ### 4. Provider Leak Scan No upstream-identifying strings in responses: ```bash curl -s -H "x-api-key: $KEY" https:///v1/models | \ grep -iq "provider_id\|system_fingerprint\|upstream" && echo "LEAK" || echo "CLEAN" ``` ### 5. Error Opaqueness Invalid key → neutral JSON, not HTML. Bad model → generic rejection, not provider-specific: ```bash curl -s https:///v1/models | jq '.error.code' # "invalid_api_key" curl -s -H "x-api-key: $KEY" -H "Content-Type: application/json" \ -d '{"model":"nonexistent","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \ https:///v1/chat/completions | jq '.error.type' # "invalid_request_error" ``` ## References - `references/deployment-commands.md` — immutable release deployment workflow + live smoke - `references/nginx-config.md` — production nginx config with HTTP/2, gzip, error masking - `references/transparency-audit.md` — full audit checklist with probe scripts ## Verification ```bash # Health (no auth) curl -sk https://api.[REDACTED-NAME].loc.cc/health # Models (auth required) curl -sk -H "Authorization: Bearer $KEY" https://api.[REDACTED-NAME].loc.cc/v1/models # Verify: owned_by=[REDACTED-NAME], no provider/system_fingerprint # Chat (warm pool — should be <3s) time curl -sk -H "Authorization: Bearer $KEY" \ -H 'Content-Type: application/json' \ -d '{"model":"rb-902d95ce37cee7204c89","messages":[{"role":"user","content":"hi"}],"max_tokens":3}' \ https://api.[REDACTED-NAME].loc.cc/v1/chat/completions # Verify: model=rb-*, fp=ABSENT, content present # HTTP/2 curl -skI --http2 https://api.[REDACTED-NAME].loc.cc/health 2>&1 | head -1 # Expected: HTTP/2 200 # Provider leak test curl -sk -H "Authorization: Bearer $KEY" https://api.[REDACTED-NAME].loc.cc/v1/models \ | python3 -c "import json,sys; d=json.load(sys.stdin); print(any('provider' in m or 'system_fingerprint' in m for m in d['data']))" # Expected: False ``` ## Anthropic Direct Proxying (Multimodal) When the upstream supports Anthropic `/v1/messages` natively, forward Anthropic requests **directly** — do NOT convert Anthropic→OpenAI→Anthropic. The conversion round-trip strips multimodal content (images, files, PDFs). ```python # do_POST — route Anthropic natively is_anthropic = path == "/v1/messages" # NO _anthropic_to_openai conversion — forward as-is suffix = "messages" if is_anthropic else path.removeprefix("/v1/") # NO _openai_to_anthropic response transform # sanitize_json_body handles model name replacement for both formats ``` The response sanitizer (`sanitize_json_body`) already handles both OpenAI and Anthropic response structures — it recursively replaces the `model` field and strips `PRIVATE_METADATA_KEYS`. ## Single-Model Endpoint OpenAI clients expect `GET /v1/models/{id}` to return a single model object. Add after the `/v1/models` list handler: ```python m = re.match(r"^/v1/models/(.+)$", path) if m: model_id = m.group(1) if model_id not in self.gateway.catalog.allowed_ids(): self._error(404, "Model not found", "model_not_found", request_id) return self._send_json(200, {"id": model_id, "object": "model", "owned_by": "[REDACTED-NAME]"}, request_id=request_id) return ``` ## Transparency Audit Addendum: Multimodal After the standard text-only audit, verify multimodal passthrough. Use a 1×1 red PNG (`iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg==`): ```bash IMG="iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg==" # Anthropic vision (gateway must forward image blocks) curl -s -X POST https:///v1/messages \ -H "Content-Type: application/json" -H "x-api-key: $KEY" \ -H "anthropic-version: 2023-06-01" \ -d "{\"model\":\"claude-sonnet-5\",\"messages\":[{\"role\":\"user\",\"content\":[ {\"type\":\"text\",\"text\":\"What color is this image?\"}, {\"type\":\"image\",\"source\":{\"type\":\"base64\",\"media_type\":\"image/png\",\"data\":\"$IMG\"}} ]}],\"max_tokens\":50}" # If the model says \"I don't see an image\" — image was stripped during conversion # Fix: forward Anthropic directly, do not convert to OpenAI ``` ## Pitfalls 1. **Stale connection hang**: pooled HTTPS connections die silently (upstream FIN). Health-check with `select.select(sock, [], [], 0)` — if readable with 0 timeout, connection is dead. Always retry on transport error. 2. **TCP_NODELAY placement**: must be called AFTER `conn.connect()` and before first `request()`. On http.client, set `conn.sock.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)`. 3. **HTTP/2 on nginx 1.28**: use `http2 on;` directive, NOT `listen ... http2` (deprecated, generates warnings). 4. **Gzip on SSE**: `gzip_types text/event-stream` is correct. SSE is text, gzip compresses both JSON and SSE. 5. **Catalog pre-warm failure**: wrap in try/except on startup, log warning. Catalog retries on first authenticated request. 6. **`_jitter()` removal**: if user says "as fast as possible", remove ALL jitter — TTL jitter AND request jitter. Keep only functional variation (UA rotation). 7. **Stream connections**: never return SSE connections to pool — they're live for the stream duration. Only buffered request connections go back. 8. **Reasoning models (Claude Fable 5, etc.)**: these consume tokens for internal reasoning before producing visible output. If `max_tokens` is too low (e.g. 10), ALL tokens go to thinking and `content` comes back `null` with `finish_reason: "length"`. Always use `max_tokens >= 50` when probing reasoning models, and check `usage.completion_tokens_details.reasoning_tokens` to confirm thinking consumption. 9. **Dual-key verification**: when gateway uses multiple upstream keys, test EACH key directly against the upstream before testing through the gateway. All keys + gateway must return structurally identical responses. A mismatch between keys may indicate per-key model availability differences or rate-limit state. 10. **Anthropic→OpenAI conversion strips multimodal**: the `_anthropic_to_openai` converter joins text blocks from content arrays but discards image/file blocks. If the upstream supports Anthropic natively, forward `/v1/messages` requests directly. Only use the converter when the upstream ONLY speaks OpenAI. 11. **base64 newlines break curl**: when piping base64 from a file, `cat file.txt` includes a trailing newline that invalidates the base64 in JSON. Use `tr -d '\n\r'` to strip.