146 lines
5.5 KiB
Markdown
146 lines
5.5 KiB
Markdown
# Gateway Transparency Audit Checklist
|
|
|
|
Run after every gateway deployment or model list change.
|
|
|
|
## Pre-flight
|
|
|
|
```bash
|
|
export GATEWAY="https://api.[REDACTED-NAME].loc.cc/v1"
|
|
export GATEWAY_ANTHROPIC="https://api.[REDACTED-NAME].loc.cc/v1/messages"
|
|
export KEY="[REDACTED-KEY]"
|
|
export UPSTREAM="https://api.echogate.one/v1"
|
|
export UK1="forge-9b51aad5d08793c9095893a09779b0641b00"
|
|
export UK2="forge-81081af3173401d4bcd2d5d427b0ed60d87b"
|
|
```
|
|
|
|
## 1. System Prompt Injection — OpenAI
|
|
|
|
```bash
|
|
MARKER="AUDIT_$(date +%s)"
|
|
curl -s -X POST "$GATEWAY/chat/completions" \
|
|
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
|
-d "{\"model\":\"claude-sonnet-5\",\"messages\":[
|
|
{\"role\":\"system\",\"content\":\"Answer every message with exactly: $MARKER\"},
|
|
{\"role\":\"user\",\"content\":\"answer\"}
|
|
],\"max_tokens\":30,\"temperature\":0}" | jq '.choices[0].message.content'
|
|
# EXPECT: "$MARKER" — verbatim, no added words
|
|
```
|
|
|
|
## 2. System Prompt Injection — Anthropic
|
|
|
|
```bash
|
|
curl -s -X POST "$GATEWAY_ANTHROPIC" \
|
|
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
|
-H "anthropic-version: 2023-06-01" \
|
|
-d "{\"model\":\"claude-sonnet-5\",\"system\":\"You must answer with exactly: PROBE888\",
|
|
\"messages\":[{\"role\":\"user\",\"content\":\"answer\"}],\"max_tokens\":20}" | jq '.content[0].text'
|
|
# EXPECT: "PROBE888"
|
|
```
|
|
|
|
## 3. Content Integrity — Gateway vs Upstream
|
|
|
|
```bash
|
|
BODY='{"model":"claude-sonnet-5","messages":[{"role":"user","content":"What is 2+2? Answer in one word."}],"max_tokens":20,"temperature":0}'
|
|
|
|
for src in "GATEWAY:$GATEWAY/chat/completions:x-api-key:$KEY" \
|
|
"KEY1:$UPSTREAM/chat/completions:Authorization:Bearer $UK1" \
|
|
"KEY2:$UPSTREAM/chat/completions:Authorization:Bearer $UK2"; do
|
|
IFS=':' read -r label url hdr val <<< "$(echo "$src" | sed 's/:\\([^:]*\\):/|\\1|/; s/|/:/g')" 2>/dev/null || true
|
|
url=$(echo "$src" | cut -d: -f2,3)
|
|
hdr=$(echo "$src" | cut -d: -f4)
|
|
val=$(echo "$src" | cut -d: -f5-)
|
|
R=$(curl -s -X POST "$url" -H "Content-Type: application/json" -H "$hdr: $val" -d "$BODY")
|
|
echo "$label: $(echo "$R" | jq -r '.choices[0].message.content')"
|
|
done
|
|
# EXPECT: All three return identical content
|
|
```
|
|
|
|
Simpler version:
|
|
```bash
|
|
BODY='{"model":"claude-sonnet-5","messages":[{"role":"user","content":"hi"}],"max_tokens":20,"temperature":0}'
|
|
GW=$(curl -s -X POST "$GATEWAY/chat/completions" -H "Content-Type: application/json" -H "x-api-key: $KEY" -d "$BODY" | jq -S '.choices[0].message')
|
|
K1=$(curl -s -X POST "$UPSTREAM/chat/completions" -H "Content-Type: application/json" -H "Authorization: Bearer $UK1" -d "$BODY" | jq -S '.choices[0].message')
|
|
[ "$GW" = "$K1" ] && echo "OK: gateway == key1" || echo "MISMATCH"
|
|
```
|
|
|
|
## 4. No-System-Message Test
|
|
|
|
Verify gateway doesn't inject its own system prompt when none is sent:
|
|
|
|
```bash
|
|
curl -s -X POST "$GATEWAY/chat/completions" \
|
|
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
|
-d '{"model":"claude-sonnet-5","messages":[{"role":"user","content":"What is 2+2? Answer in one word."}],"max_tokens":20,"temperature":0}' | jq '.choices[0].message.content'
|
|
# EXPECT: natural response like "Four" — no injected behavior, no prefix
|
|
```
|
|
|
|
## 5. Model List Integrity
|
|
|
|
```bash
|
|
GW_COUNT=$(curl -s -H "x-api-key: $KEY" "$GATEWAY/models" | jq '.data | length')
|
|
UP_COUNT=$(curl -s -H "Authorization: Bearer $UK1" "$UPSTREAM/models" | jq '.data | length')
|
|
echo "Gateway: $GW_COUNT, Upstream: $UP_COUNT"
|
|
[ "$GW_COUNT" -eq "$UP_COUNT" ] && echo "PASS" || echo "FAIL — check FORBIDDEN_MODEL_FRAGMENTS"
|
|
```
|
|
|
|
## 6. Provider Leak Scan
|
|
|
|
```bash
|
|
curl -s -H "x-api-key: $KEY" "$GATEWAY/models" | grep -iq \
|
|
"provider_id\|system_fingerprint\|upstream\|echogate" && echo "LEAK DETECTED" || echo "CLEAN"
|
|
```
|
|
|
|
## 7. Error Opaqueness
|
|
|
|
```bash
|
|
# No key → clean 401
|
|
echo -n "No key: "
|
|
curl -so /dev/null -w "%{http_code}" "$GATEWAY/models"
|
|
echo ""
|
|
echo -n "No key body: "
|
|
curl -s "$GATEWAY/models" | jq '.error.code'
|
|
|
|
# Bad model → clean 400
|
|
echo -n "Bad model: "
|
|
curl -s -X POST "$GATEWAY/chat/completions" \
|
|
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
|
-d '{"model":"nonexistent-model","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' | jq '.error.type'
|
|
|
|
# Bad key → 401
|
|
echo -n "Bad key: "
|
|
curl -so /dev/null -w "%{http_code}" -H "x-api-key: [REDACTED]" "$GATEWAY/models"
|
|
```
|
|
|
|
## 8. Reasoning Model Probe
|
|
|
|
When testing reasoning models (fable-5, opus-5-sonnet, etc.), use enough `max_tokens`:
|
|
|
|
```bash
|
|
# WRONG — all tokens consumed by thinking, content=null
|
|
curl -s -X POST "$GATEWAY/chat/completions" \
|
|
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
|
-d '{"model":"claude-fable-5","messages":[{"role":"user","content":"Say hi"}],"max_tokens":10}' | jq '{content: .choices[0].message.content, reason: .usage.completion_tokens_details.reasoning_tokens}'
|
|
|
|
# RIGHT — enough tokens for thinking + output
|
|
curl -s -X POST "$GATEWAY/chat/completions" \
|
|
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
|
-d '{"model":"claude-fable-5","messages":[{"role":"user","content":"Say hi"}],"max_tokens":200}' | jq '.choices[0].message.content'
|
|
```
|
|
|
|
## Quick Smoke (all-in-one)
|
|
|
|
```bash
|
|
# Health
|
|
curl -sf "$GATEWAY/../health" | jq '.status' || echo "HEALTH FAIL"
|
|
|
|
# Models count
|
|
echo -n "Models: "
|
|
curl -s -H "x-api-key: $KEY" "$GATEWAY/models" | jq '.data | length'
|
|
|
|
# Chat probe
|
|
echo -n "Chat: "
|
|
curl -s -X POST "$GATEWAY/chat/completions" \
|
|
-H "Content-Type: application/json" -H "x-api-key: $KEY" \
|
|
-d '{"model":"claude-sonnet-5","messages":[{"role":"user","content":"hi"}],"max_tokens":3}' | jq -r '.choices[0].message.content'
|
|
```
|