autoreg-skills-max: initial public release (restored, redacted, hardened)

skills/ai-freebie-scanner/references/scanner-v3.1-architecture.md

@@ -0,0 +1,84 @@
+# Scanner v3.1 Architecture & Quality Scoring
+
+## Flow
+```
+19 sources (parallel asyncio.gather)
+  → raw items (185-238 per first run)
+  → quality_score() filter
+  → dedup by URL (seen_v3.json)
+  → top 15 by score
+  → build_post() with @ikhdev1 format
+  → send_tg() via Telegram Bot API
+```
+
+## quality_score() Algorithm
+
+```python
+def quality_score(item: dict) -> int:
+    score = 1  # base
+    
+    # Source quality
+    if src in direct_sources:    score += 3  # NodeLoc, Lolz
+    if src in premium_sources:   score += 4  # KeyLeaks, BazaarLink, Airforce
+    
+    # Content signals
+    if has_model_name(full_text):     score += 1  # claude, gpt, deepseek...
+    if has_instructions(full_text):   score += 1  # register, signup, setup...
+    if has_keys_tokens(full_text):    score += 1  # api key, token, credits...
+    if is_abuse_jailbreak(full_text): score += 2  # jailbreak, bypass, exploit
+    
+    # Detail richness
+    if len(snippet) > 50:   score += 2
+    if len(snippet) > 150:  score += 1
+    
+    return min(score, 10)
+```
+
+## Source Performance Matrix
+
+| Source | First Run Items | Subsequent | Reliability | Notes |
+|--------|----------------|------------|-------------|-------|
+| GitHub awesome-lists | 150-180 | 0-5 | High | Read raw README.md from 4 repos |
+| DuckDuckGo | 5-15 | 3-10 | Medium | HTML scraping, rate limited |
+| NodeLoc | 10-30 | 5-15 | High | Needs fresh cookies |
+| HN Algolia | 5-10 | 2-5 | High | JSON API, fast |
+| KeyLeaks | 1 | 0 | Low | Static page, DNS issues locally |
+| Blogs (8 sources) | 8 | 0-2 | Low | One-time checks via state keys |
+| Lolz.live | 2-8 | 2-8 | Medium | HTML selectors fragile |
+| GitHub Search | 1-5 | 0-2 | Low | Rate limited without token |
+| Linux.do | 0 | 0 | Dead | CF JS Challenge blocks everything |
+| Reddit | 0 | 0 | Dead | 403 on all datacenter IPs |
+
+## State Management
+
+- `seen_v3.json`: MD5 hashes of visited URLs, max 10000 entries
+- Each source marks URLs as seen via `mark(url)`
+- `is_new(url)` checks hash not in set
+- Clear state for full rescan: `rm seen_v3.json`
+
+## Telegram Delivery
+
+- Bot: @dfhjdfghfjdhfhbot (token: [REDACTED])
+- Chat: @[REDACTED-NAME] ([REDACTED-TGID])
+- Method: `sendMessage` with `parse_mode=HTML`
+- Limit: 4096 chars (message auto-truncates at 3950 + footer)
+- HTML escaping via `safe_html()` is MANDATORY — any unescaped `<` or `&` causes "Unclosed start tag" 400 error
+
+## Server Deployment
+
+Both servers run identical `/root/ai_scanner.py` via cron every 15 min:
+
+```
+gothbreach-abuse-1 (SGP): [REDACTED-IP]
+eni-nyc1 (NYC): [REDACTED-IP]
+```
+
+Deploy command:
+```bash
+cat scanner.py | ssh root@<IP> "cat > /root/ai_scanner.py"
+```
+
+Cron:
+```bash
+echo '*/15 * * * * cd /root && python3 ai_scanner.py >> /root/ai_scanner_cron.log 2>&1' | crontab -
+```