# Scanner v3.1 Architecture & Quality Scoring

## Flow
```
19 sources (parallel asyncio.gather)
  → raw items (185-238 per first run)
  → quality_score() filter
  → dedup by URL (seen_v3.json)
  → top 15 by score
  → build_post() with @ikhdev1 format
  → send_tg() via Telegram Bot API
```

## quality_score() Algorithm

```python
def quality_score(item: dict) -> int:
    score = 1  # base
    
    # Source quality
    if src in direct_sources:    score += 3  # NodeLoc, Lolz
    if src in premium_sources:   score += 4  # KeyLeaks, BazaarLink, Airforce
    
    # Content signals
    if has_model_name(full_text):     score += 1  # claude, gpt, deepseek...
    if has_instructions(full_text):   score += 1  # register, signup, setup...
    if has_keys_tokens(full_text):    score += 1  # api key, token, credits...
    if is_abuse_jailbreak(full_text): score += 2  # jailbreak, bypass, exploit
    
    # Detail richness
    if len(snippet) > 50:   score += 2
    if len(snippet) > 150:  score += 1
    
    return min(score, 10)
```

## Source Performance Matrix

| Source | First Run Items | Subsequent | Reliability | Notes |
|--------|----------------|------------|-------------|-------|
| GitHub awesome-lists | 150-180 | 0-5 | High | Read raw README.md from 4 repos |
| DuckDuckGo | 5-15 | 3-10 | Medium | HTML scraping, rate limited |
| NodeLoc | 10-30 | 5-15 | High | Needs fresh cookies |
| HN Algolia | 5-10 | 2-5 | High | JSON API, fast |
| KeyLeaks | 1 | 0 | Low | Static page, DNS issues locally |
| Blogs (8 sources) | 8 | 0-2 | Low | One-time checks via state keys |
| Lolz.live | 2-8 | 2-8 | Medium | HTML selectors fragile |
| GitHub Search | 1-5 | 0-2 | Low | Rate limited without token |
| Linux.do | 0 | 0 | Dead | CF JS Challenge blocks everything |
| Reddit | 0 | 0 | Dead | 403 on all datacenter IPs |

## State Management

- `seen_v3.json`: MD5 hashes of visited URLs, max 10000 entries
- Each source marks URLs as seen via `mark(url)`
- `is_new(url)` checks hash not in set
- Clear state for full rescan: `rm seen_v3.json`

## Telegram Delivery

- Bot: @dfhjdfghfjdhfhbot (token: [REDACTED])
- Chat: @[REDACTED-NAME] ([REDACTED-TGID])
- Method: `sendMessage` with `parse_mode=HTML`
- Limit: 4096 chars (message auto-truncates at 3950 + footer)
- HTML escaping via `safe_html()` is MANDATORY — any unescaped `<` or `&` causes "Unclosed start tag" 400 error

## Server Deployment

Both servers run identical `/root/ai_scanner.py` via cron every 15 min:

```
gothbreach-abuse-1 (SGP): [REDACTED-IP]
eni-nyc1 (NYC): [REDACTED-IP]
```

Deploy command:
```bash
cat scanner.py | ssh root@<IP> "cat > /root/ai_scanner.py"
```

Cron:
```bash
echo '*/15 * * * * cd /root && python3 ai_scanner.py >> /root/ai_scanner_cron.log 2>&1' | crontab -
```