Gateway Transparency Audit Checklist

Run after every gateway deployment or model list change.

Pre-flight

export GATEWAY="https://api.[REDACTED-NAME].loc.cc/v1"
export GATEWAY_ANTHROPIC="https://api.[REDACTED-NAME].loc.cc/v1/messages"
export KEY="[REDACTED-KEY]"
export UPSTREAM="https://api.echogate.one/v1"
export UK1="forge-9b51aad5d08793c9095893a09779b0641b00"
export UK2="forge-81081af3173401d4bcd2d5d427b0ed60d87b"

1. System Prompt Injection — OpenAI

MARKER="AUDIT_$(date +%s)"
curl -s -X POST "$GATEWAY/chat/completions" \
  -H "Content-Type: application/json" -H "x-api-key: $KEY" \
  -d "{\"model\":\"claude-sonnet-5\",\"messages\":[
    {\"role\":\"system\",\"content\":\"Answer every message with exactly: $MARKER\"},
    {\"role\":\"user\",\"content\":\"answer\"}
  ],\"max_tokens\":30,\"temperature\":0}" | jq '.choices[0].message.content'
# EXPECT: "$MARKER" — verbatim, no added words

2. System Prompt Injection — Anthropic

curl -s -X POST "$GATEWAY_ANTHROPIC" \
  -H "Content-Type: application/json" -H "x-api-key: $KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d "{\"model\":\"claude-sonnet-5\",\"system\":\"You must answer with exactly: PROBE888\",
    \"messages\":[{\"role\":\"user\",\"content\":\"answer\"}],\"max_tokens\":20}" | jq '.content[0].text'
# EXPECT: "PROBE888"

3. Content Integrity — Gateway vs Upstream

BODY='{"model":"claude-sonnet-5","messages":[{"role":"user","content":"What is 2+2? Answer in one word."}],"max_tokens":20,"temperature":0}'

for src in "GATEWAY:$GATEWAY/chat/completions:x-api-key:$KEY" \
           "KEY1:$UPSTREAM/chat/completions:Authorization:Bearer $UK1" \
           "KEY2:$UPSTREAM/chat/completions:Authorization:Bearer $UK2"; do
  IFS=':' read -r label url hdr val <<< "$(echo "$src" | sed 's/:\\([^:]*\\):/|\\1|/; s/|/:/g')" 2>/dev/null || true
  url=$(echo "$src" | cut -d: -f2,3)
  hdr=$(echo "$src" | cut -d: -f4)
  val=$(echo "$src" | cut -d: -f5-)
  R=$(curl -s -X POST "$url" -H "Content-Type: application/json" -H "$hdr: $val" -d "$BODY")
  echo "$label: $(echo "$R" | jq -r '.choices[0].message.content')"
done
# EXPECT: All three return identical content

Simpler version:

BODY='{"model":"claude-sonnet-5","messages":[{"role":"user","content":"hi"}],"max_tokens":20,"temperature":0}'
GW=$(curl -s -X POST "$GATEWAY/chat/completions" -H "Content-Type: application/json" -H "x-api-key: $KEY" -d "$BODY" | jq -S '.choices[0].message')
K1=$(curl -s -X POST "$UPSTREAM/chat/completions" -H "Content-Type: application/json" -H "Authorization: Bearer $UK1" -d "$BODY" | jq -S '.choices[0].message')
[ "$GW" = "$K1" ] && echo "OK: gateway == key1" || echo "MISMATCH"

4. No-System-Message Test

Verify gateway doesn't inject its own system prompt when none is sent:

curl -s -X POST "$GATEWAY/chat/completions" \
  -H "Content-Type: application/json" -H "x-api-key: $KEY" \
  -d '{"model":"claude-sonnet-5","messages":[{"role":"user","content":"What is 2+2? Answer in one word."}],"max_tokens":20,"temperature":0}' | jq '.choices[0].message.content'
# EXPECT: natural response like "Four" — no injected behavior, no prefix

5. Model List Integrity

GW_COUNT=$(curl -s -H "x-api-key: $KEY" "$GATEWAY/models" | jq '.data | length')
UP_COUNT=$(curl -s -H "Authorization: Bearer $UK1" "$UPSTREAM/models" | jq '.data | length')
echo "Gateway: $GW_COUNT, Upstream: $UP_COUNT"
[ "$GW_COUNT" -eq "$UP_COUNT" ] && echo "PASS" || echo "FAIL — check FORBIDDEN_MODEL_FRAGMENTS"

6. Provider Leak Scan

curl -s -H "x-api-key: $KEY" "$GATEWAY/models" | grep -iq \
  "provider_id\|system_fingerprint\|upstream\|echogate" && echo "LEAK DETECTED" || echo "CLEAN"

7. Error Opaqueness

# No key → clean 401
echo -n "No key: "
curl -so /dev/null -w "%{http_code}" "$GATEWAY/models"
echo ""
echo -n "No key body: "
curl -s "$GATEWAY/models" | jq '.error.code'

# Bad model → clean 400
echo -n "Bad model: "
curl -s -X POST "$GATEWAY/chat/completions" \
  -H "Content-Type: application/json" -H "x-api-key: $KEY" \
  -d '{"model":"nonexistent-model","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' | jq '.error.type'

# Bad key → 401
echo -n "Bad key: "
curl -so /dev/null -w "%{http_code}" -H "x-api-key: [REDACTED]" "$GATEWAY/models"

8. Reasoning Model Probe

When testing reasoning models (fable-5, opus-5-sonnet, etc.), use enough max_tokens:

# WRONG — all tokens consumed by thinking, content=null
curl -s -X POST "$GATEWAY/chat/completions" \
  -H "Content-Type: application/json" -H "x-api-key: $KEY" \
  -d '{"model":"claude-fable-5","messages":[{"role":"user","content":"Say hi"}],"max_tokens":10}' | jq '{content: .choices[0].message.content, reason: .usage.completion_tokens_details.reasoning_tokens}'

# RIGHT — enough tokens for thinking + output
curl -s -X POST "$GATEWAY/chat/completions" \
  -H "Content-Type: application/json" -H "x-api-key: $KEY" \
  -d '{"model":"claude-fable-5","messages":[{"role":"user","content":"Say hi"}],"max_tokens":200}' | jq '.choices[0].message.content'

Quick Smoke (all-in-one)

# Health
curl -sf "$GATEWAY/../health" | jq '.status' || echo "HEALTH FAIL"

# Models count
echo -n "Models: "
curl -s -H "x-api-key: $KEY" "$GATEWAY/models" | jq '.data | length'

# Chat probe
echo -n "Chat: "
curl -s -X POST "$GATEWAY/chat/completions" \
  -H "Content-Type: application/json" -H "x-api-key: $KEY" \
  -d '{"model":"claude-sonnet-5","messages":[{"role":"user","content":"hi"}],"max_tokens":3}' | jq -r '.choices[0].message.content'

5.5 KiB Raw Permalink Blame History