366 lines
15 KiB
Markdown
366 lines
15 KiB
Markdown
---
|
||
name: reseller-api-gateway
|
||
description: Build anonymous OpenAI-compatible API gateways for reselling — hide upstream provider, alias model IDs, strip metadata, apply speed optimizations (HTTP/2, connection pool, gzip, TCP_NODELAY), and deploy via immutable releases.
|
||
triggers:
|
||
- "resell API"
|
||
- "hide provider"
|
||
- "anonymous gateway"
|
||
- "opaque proxy"
|
||
- "api reseller"
|
||
- "model aliasing"
|
||
- "[REDACTED-NAME] API"
|
||
- "apioid gateway"
|
||
- "speed up API"
|
||
---
|
||
|
||
# Reseller API Gateway
|
||
|
||
Build an anonymous OpenAI-compatible API gateway that proxies an upstream OpenAI-compatible backend while hiding its identity — for reselling. Model IDs are aliased via HMAC, metadata is stripped, and speed optimizations eliminate unnecessary latency.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Client (any OpenAI client)
|
||
→ Nginx (TLS termination, HTTP/2, gzip, error masking)
|
||
→ Gateway (Python http.server on localhost, connection pool)
|
||
→ Upstream (echogate.one, OpenRouter, any /v1 backend)
|
||
```
|
||
|
||
## Core Patterns
|
||
|
||
### 1. Model ID Aliasing (HMAC-SHA256)
|
||
|
||
Every upstream model ID gets a deterministic public alias. Upstream never sees the alias; clients never see the real ID.
|
||
|
||
```python
|
||
import hmac, hashlib
|
||
|
||
def public_model_id(private_id: str, secret: str) -> str:
|
||
digest = hmac.new(secret.encode(), private_id.encode(), hashlib.sha256).hexdigest()
|
||
return f"rb-{digest[:20]}"
|
||
```
|
||
|
||
- Private IDs ONLY accepted on upstream requests (gateway looks up alias → real ID)
|
||
- Direct private IDs in client requests → HTTP 400
|
||
- Forbidden models filtered in catalog (e.g. `gpt-5-5` fragment match)
|
||
|
||
### 2. Metadata Sanitization
|
||
|
||
Strip all upstream metadata from responses:
|
||
|
||
```python
|
||
PRIVATE_METADATA_KEYS = {
|
||
"backend", "provider", "provider_id", "provider_name",
|
||
"route", "routing", "system_fingerprint",
|
||
"upstream", "upstream_id", "upstream_model", "upstream_provider",
|
||
}
|
||
```
|
||
|
||
- `owned_by` always `[REDACTED-NAME]` (or user's brand)
|
||
- `system_fingerprint` removed
|
||
- `model` field rewritten to public alias in both buffered and SSE responses
|
||
- All hop-by-hop headers stripped (`Connection`, `Keep-Alive`, etc.)
|
||
|
||
### 3. Connection Pool with Retry
|
||
|
||
Thread-safe HTTPS connection pool — eliminates TCP+TLS handshake per request:
|
||
|
||
```python
|
||
class ConnectionPool:
|
||
def __init__(self, host, port, scheme, timeout, max_idle=4):
|
||
self._lock = threading.Lock()
|
||
self._idle: list[http.client.HTTPConnection] = []
|
||
self._conn_type = HTTPSConnection if scheme == "https" else HTTPConnection
|
||
|
||
def _create(self):
|
||
conn = self._conn_type(host, port, timeout=timeout)
|
||
conn.connect()
|
||
conn.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
|
||
return conn
|
||
|
||
def get(self):
|
||
with self._lock:
|
||
while self._idle:
|
||
conn = self._idle.pop()
|
||
# select() health check: if socket readable, connection is dead
|
||
r, _, _ = select.select([conn.sock], [], [], 0)
|
||
if not r:
|
||
return conn
|
||
conn.close()
|
||
return self._create()
|
||
|
||
def put(self, conn):
|
||
with self._lock:
|
||
if len(self._idle) < self._max_idle and conn.sock is not None:
|
||
self._idle.append(conn)
|
||
return
|
||
conn.close()
|
||
```
|
||
|
||
**Retry pattern** — stale connections fail on first use, retry with fresh:
|
||
|
||
```python
|
||
def upstream_request(self, method, suffix, body):
|
||
for attempt in (1, 2):
|
||
conn = self._pool.get()
|
||
try:
|
||
conn.request(method, path, body=body, headers=headers)
|
||
response = conn.getresponse()
|
||
body = response.read()
|
||
if response.status < 500:
|
||
self._pool.put(conn) # healthy → return to pool
|
||
else:
|
||
conn.close() # 5xx → discard
|
||
return response.status, response.getheaders(), body
|
||
except (HTTPException, OSError, TimeoutError):
|
||
conn.close() # transport error → discard
|
||
if attempt == 2:
|
||
raise # retry exhausted
|
||
```
|
||
|
||
- Only `5xx` and transport errors discard connections; `4xx` connections go back to pool
|
||
- Pre-warm 2 connections + force catalog fetch on startup
|
||
|
||
### 4. Speed Optimizations (Nginx)
|
||
|
||
```nginx
|
||
server {
|
||
listen 443 ssl;
|
||
http2 on; # Multiplexing
|
||
gzip on;
|
||
gzip_min_length 256;
|
||
gzip_types application/json text/event-stream;
|
||
gzip_comp_level 1;
|
||
|
||
location / {
|
||
proxy_pass http://127.0.0.1:18181;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Connection "";
|
||
proxy_read_timeout 600s;
|
||
proxy_request_buffering off;
|
||
proxy_buffering off;
|
||
}
|
||
|
||
# Mask ALL nginx/upstream errors as neutral JSON
|
||
error_page 400 408 411 414 431 = @invalid_request;
|
||
error_page 405 = @method_not_allowed;
|
||
error_page 500 502 503 504 = @service_unavailable;
|
||
}
|
||
```
|
||
|
||
### 5. Anti-Fingerprint
|
||
|
||
- **UA rotation**: rotate through `OpenAI/Python`, `Go-http-client/2.0`, `python-requests/2.32`, `curl/8.12` — no custom User-Agent
|
||
- **Server header**: stripped via `more_clear_headers Server;` or `server_tokens off;`
|
||
- **Error responses**: always neutral JSON, never upstream error bodies or HTML
|
||
- **Unknown routes**: 404 JSON, never nginx default HTML
|
||
- **Echo/legacy paths**: explicit 404 (e.g. `^~ /echo/` → JSON 404)
|
||
|
||
### 6. Immutable Release Deployment
|
||
|
||
Each deploy creates a timestamped release directory with symlink:
|
||
|
||
```bash
|
||
RELEASE_ID=$(date +%Y%m%d_%H%M%S)
|
||
mkdir -p /opt/apioid-echo/releases/${RELEASE_ID}
|
||
scp gateway.py root@host:/opt/apioid-echo/releases/${RELEASE_ID}/
|
||
ssh root@host "
|
||
ln -sfn /opt/apioid-echo/releases/${RELEASE_ID} /opt/apioid-echo/releases/current
|
||
systemctl restart apioid-echo-gateway.service
|
||
"
|
||
```
|
||
|
||
- Backup before deploy: copy current → `/var/backups/apioid-echo/gateway.before-${RELEASE_ID}.py`
|
||
- Verify: `systemctl is-active`, `NRestarts=0`, live smoke with curl
|
||
- Env file: `/etc/apioid-echo/gateway.env` chmod 600 (keys + `MODEL_ALIAS_SECRET`)
|
||
|
||
## Error Masking
|
||
|
||
All errors → neutral JSON, never HTML or upstream metadata:
|
||
|
||
| Upstream status | Client response |
|
||
|---|---|
|
||
| 429 | `{"error":{"message":"Rate limit exceeded","code":"rate_limit_exceeded"}}` |
|
||
| 402 | `{"error":{"message":"Daily usage limit reached","code":"daily_limit_reached"}}` |
|
||
| 400/404/409/422 | `{"error":{"message":"Request was rejected","code":"request_rejected"}}` |
|
||
| Other 5xx | `{"error":{"message":"Service temporarily unavailable","code":"service_unavailable"}}` |
|
||
| Auth fail | `{"error":{"message":"Invalid or missing API key","code":"invalid_api_key"}}` |
|
||
|
||
## Forbidden Models
|
||
|
||
Block specific model fragments from reaching upstream:
|
||
|
||
```python
|
||
FORBIDDEN_MODEL_FRAGMENT = "gpt-5-5"
|
||
|
||
@staticmethod
|
||
def _is_allowed(model_id: str) -> bool:
|
||
normalized = re.sub(r"[\s._/]+", "-", model_id.casefold())
|
||
return FORBIDDEN_MODEL_FRAGMENT not in normalized
|
||
```
|
||
|
||
## Transparency Audit
|
||
|
||
After deploying or modifying the gateway, verify it is a **pure passthrough** — no system prompt injection, no content modification, no metadata leak.
|
||
|
||
### 1. System Prompt Injection Detection
|
||
|
||
Send a system prompt with a **unique marker string** through the gateway. The model must echo it exactly — any deviation proves injection/modification:
|
||
|
||
```bash
|
||
curl -s -X POST https://<gateway>/v1/chat/completions \
|
||
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
||
-d '{"model":"claude-sonnet-5","messages":[
|
||
{"role":"system","content":"Answer every message with exactly: PROBE_OK"},
|
||
{"role":"user","content":"answer"}
|
||
],"max_tokens":20,"temperature":0}' | jq '.choices[0].message.content'
|
||
# Expected: "PROBE_OK" — verbatim, no added words
|
||
```
|
||
|
||
For Anthropic format, same test on `/v1/messages` with `system` field.
|
||
|
||
### 2. Content Integrity — Gateway vs Upstream 1:1
|
||
|
||
Send identical requests to gateway AND directly to upstream. Responses must match:
|
||
|
||
```bash
|
||
BODY='{"model":"claude-sonnet-5","messages":[{"role":"user","content":"hi"}],"max_tokens":20,"temperature":0}'
|
||
|
||
GATEWAY=$(curl -s -X POST https://<gateway>/v1/chat/completions \
|
||
-H "Content-Type: application/json" -H "x-api-key: $KEY" -d "$BODY")
|
||
UPSTREAM=$(curl -s -X POST https://<upstream>/v1/chat/completions \
|
||
-H "Content-Type: application/json" -H "Authorization: Bearer $UK" -d "$BODY")
|
||
|
||
diff <(echo "$GATEWAY" | jq -S '.choices[0].message.content') \
|
||
<(echo "$UPSTREAM" | jq -S '.choices[0].message.content')
|
||
```
|
||
|
||
For dual-key setups, test BOTH upstream keys independently — all three sources (key1, key2, gateway) must agree.
|
||
|
||
### 3. Model List Integrity
|
||
|
||
Gateway `/v1/models` count must match upstream count. If filtering is intentional, document it:
|
||
|
||
```bash
|
||
GC=$(curl -s -H "x-api-key: $KEY" https://<gateway>/v1/models | jq '.data | length')
|
||
UC=$(curl -s -H "Authorization: Bearer $UK" https://<upstream>/v1/models | jq '.data | length')
|
||
[ "$GC" -eq "$UC" ] && echo "OK: $GC models" || echo "MISMATCH: gateway=$GC upstream=$UC"
|
||
```
|
||
|
||
### 4. Provider Leak Scan
|
||
|
||
No upstream-identifying strings in responses:
|
||
|
||
```bash
|
||
curl -s -H "x-api-key: $KEY" https://<gateway>/v1/models | \
|
||
grep -iq "provider_id\|system_fingerprint\|upstream" && echo "LEAK" || echo "CLEAN"
|
||
```
|
||
|
||
### 5. Error Opaqueness
|
||
|
||
Invalid key → neutral JSON, not HTML. Bad model → generic rejection, not provider-specific:
|
||
|
||
```bash
|
||
curl -s https://<gateway>/v1/models | jq '.error.code' # "invalid_api_key"
|
||
curl -s -H "x-api-key: $KEY" -H "Content-Type: application/json" \
|
||
-d '{"model":"nonexistent","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
|
||
https://<gateway>/v1/chat/completions | jq '.error.type' # "invalid_request_error"
|
||
```
|
||
|
||
## References
|
||
|
||
- `references/deployment-commands.md` — immutable release deployment workflow + live smoke
|
||
- `references/nginx-config.md` — production nginx config with HTTP/2, gzip, error masking
|
||
- `references/transparency-audit.md` — full audit checklist with probe scripts
|
||
|
||
## Verification
|
||
|
||
```bash
|
||
# Health (no auth)
|
||
curl -sk https://api.[REDACTED-NAME].loc.cc/health
|
||
|
||
# Models (auth required)
|
||
curl -sk -H "Authorization: Bearer $KEY" https://api.[REDACTED-NAME].loc.cc/v1/models
|
||
# Verify: owned_by=[REDACTED-NAME], no provider/system_fingerprint
|
||
|
||
# Chat (warm pool — should be <3s)
|
||
time curl -sk -H "Authorization: Bearer $KEY" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"model":"rb-902d95ce37cee7204c89","messages":[{"role":"user","content":"hi"}],"max_tokens":3}' \
|
||
https://api.[REDACTED-NAME].loc.cc/v1/chat/completions
|
||
# Verify: model=rb-*, fp=ABSENT, content present
|
||
|
||
# HTTP/2
|
||
curl -skI --http2 https://api.[REDACTED-NAME].loc.cc/health 2>&1 | head -1
|
||
# Expected: HTTP/2 200
|
||
|
||
# Provider leak test
|
||
curl -sk -H "Authorization: Bearer $KEY" https://api.[REDACTED-NAME].loc.cc/v1/models \
|
||
| python3 -c "import json,sys; d=json.load(sys.stdin); print(any('provider' in m or 'system_fingerprint' in m for m in d['data']))"
|
||
# Expected: False
|
||
```
|
||
|
||
## Anthropic Direct Proxying (Multimodal)
|
||
|
||
When the upstream supports Anthropic `/v1/messages` natively, forward Anthropic requests **directly** — do NOT convert Anthropic→OpenAI→Anthropic. The conversion round-trip strips multimodal content (images, files, PDFs).
|
||
|
||
```python
|
||
# do_POST — route Anthropic natively
|
||
is_anthropic = path == "/v1/messages"
|
||
# NO _anthropic_to_openai conversion — forward as-is
|
||
suffix = "messages" if is_anthropic else path.removeprefix("/v1/")
|
||
# NO _openai_to_anthropic response transform
|
||
# sanitize_json_body handles model name replacement for both formats
|
||
```
|
||
|
||
The response sanitizer (`sanitize_json_body`) already handles both OpenAI and Anthropic response structures — it recursively replaces the `model` field and strips `PRIVATE_METADATA_KEYS`.
|
||
|
||
## Single-Model Endpoint
|
||
|
||
OpenAI clients expect `GET /v1/models/{id}` to return a single model object. Add after the `/v1/models` list handler:
|
||
|
||
```python
|
||
m = re.match(r"^/v1/models/(.+)$", path)
|
||
if m:
|
||
model_id = m.group(1)
|
||
if model_id not in self.gateway.catalog.allowed_ids():
|
||
self._error(404, "Model not found", "model_not_found", request_id)
|
||
return
|
||
self._send_json(200, {"id": model_id, "object": "model", "owned_by": "[REDACTED-NAME]"}, request_id=request_id)
|
||
return
|
||
```
|
||
|
||
## Transparency Audit Addendum: Multimodal
|
||
|
||
After the standard text-only audit, verify multimodal passthrough. Use a 1×1 red PNG (`iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg==`):
|
||
|
||
```bash
|
||
IMG="iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg=="
|
||
|
||
# Anthropic vision (gateway must forward image blocks)
|
||
curl -s -X POST https://<gateway>/v1/messages \
|
||
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
||
-H "anthropic-version: 2023-06-01" \
|
||
-d "{\"model\":\"claude-sonnet-5\",\"messages\":[{\"role\":\"user\",\"content\":[
|
||
{\"type\":\"text\",\"text\":\"What color is this image?\"},
|
||
{\"type\":\"image\",\"source\":{\"type\":\"base64\",\"media_type\":\"image/png\",\"data\":\"$IMG\"}}
|
||
]}],\"max_tokens\":50}"
|
||
|
||
# If the model says \"I don't see an image\" — image was stripped during conversion
|
||
# Fix: forward Anthropic directly, do not convert to OpenAI
|
||
```
|
||
|
||
## Pitfalls
|
||
|
||
1. **Stale connection hang**: pooled HTTPS connections die silently (upstream FIN). Health-check with `select.select(sock, [], [], 0)` — if readable with 0 timeout, connection is dead. Always retry on transport error.
|
||
2. **TCP_NODELAY placement**: must be called AFTER `conn.connect()` and before first `request()`. On http.client, set `conn.sock.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)`.
|
||
3. **HTTP/2 on nginx 1.28**: use `http2 on;` directive, NOT `listen ... http2` (deprecated, generates warnings).
|
||
4. **Gzip on SSE**: `gzip_types text/event-stream` is correct. SSE is text, gzip compresses both JSON and SSE.
|
||
5. **Catalog pre-warm failure**: wrap in try/except on startup, log warning. Catalog retries on first authenticated request.
|
||
6. **`_jitter()` removal**: if user says "as fast as possible", remove ALL jitter — TTL jitter AND request jitter. Keep only functional variation (UA rotation).
|
||
7. **Stream connections**: never return SSE connections to pool — they're live for the stream duration. Only buffered request connections go back.
|
||
8. **Reasoning models (Claude Fable 5, etc.)**: these consume tokens for internal reasoning before producing visible output. If `max_tokens` is too low (e.g. 10), ALL tokens go to thinking and `content` comes back `null` with `finish_reason: "length"`. Always use `max_tokens >= 50` when probing reasoning models, and check `usage.completion_tokens_details.reasoning_tokens` to confirm thinking consumption.
|
||
9. **Dual-key verification**: when gateway uses multiple upstream keys, test EACH key directly against the upstream before testing through the gateway. All keys + gateway must return structurally identical responses. A mismatch between keys may indicate per-key model availability differences or rate-limit state.
|
||
10. **Anthropic→OpenAI conversion strips multimodal**: the `_anthropic_to_openai` converter joins text blocks from content arrays but discards image/file blocks. If the upstream supports Anthropic natively, forward `/v1/messages` requests directly. Only use the converter when the upstream ONLY speaks OpenAI.
|
||
11. **base64 newlines break curl**: when piping base64 from a file, `cat file.txt` includes a trailing newline that invalidates the base64 in JSON. Use `tr -d '\n\r'` to strip.
|