autoreg-skills-max/skills/ai-freebie-scanner/references/scanner-v3.1-architecture.md

Scanner v3.1 Architecture & Quality Scoring

Flow

19 sources (parallel asyncio.gather)
  → raw items (185-238 per first run)
  → quality_score() filter
  → dedup by URL (seen_v3.json)
  → top 15 by score
  → build_post() with @ikhdev1 format
  → send_tg() via Telegram Bot API

quality_score() Algorithm

def quality_score(item: dict) -> int:
    score = 1  # base
    
    # Source quality
    if src in direct_sources:    score += 3  # NodeLoc, Lolz
    if src in premium_sources:   score += 4  # KeyLeaks, BazaarLink, Airforce
    
    # Content signals
    if has_model_name(full_text):     score += 1  # claude, gpt, deepseek...
    if has_instructions(full_text):   score += 1  # register, signup, setup...
    if has_keys_tokens(full_text):    score += 1  # api key, token, credits...
    if is_abuse_jailbreak(full_text): score += 2  # jailbreak, bypass, exploit
    
    # Detail richness
    if len(snippet) > 50:   score += 2
    if len(snippet) > 150:  score += 1
    
    return min(score, 10)

Source Performance Matrix

Source	First Run Items	Subsequent	Reliability	Notes
GitHub awesome-lists	150-180	0-5	High	Read raw README.md from 4 repos
DuckDuckGo	5-15	3-10	Medium	HTML scraping, rate limited
NodeLoc	10-30	5-15	High	Needs fresh cookies
HN Algolia	5-10	2-5	High	JSON API, fast
KeyLeaks	1	0	Low	Static page, DNS issues locally
Blogs (8 sources)	8	0-2	Low	One-time checks via state keys
Lolz.live	2-8	2-8	Medium	HTML selectors fragile
GitHub Search	1-5	0-2	Low	Rate limited without token
Linux.do	0	0	Dead	CF JS Challenge blocks everything
Reddit	0	0	Dead	403 on all datacenter IPs

State Management

• seen_v3.json: MD5 hashes of visited URLs, max 10000 entries
• Each source marks URLs as seen via mark(url)
• is_new(url) checks hash not in set
• Clear state for full rescan: rm seen_v3.json

Telegram Delivery

• Bot: @dfhjdfghfjdhfhbot (token: [REDACTED])
• Chat: @[REDACTED-NAME] ([REDACTED-TGID])
• Method: sendMessage with parse_mode=HTML
• Limit: 4096 chars (message auto-truncates at 3950 + footer)
• HTML escaping via safe_html() is MANDATORY — any unescaped < or & causes "Unclosed start tag" 400 error

Server Deployment

Both servers run identical /root/ai_scanner.py via cron every 15 min:

gothbreach-abuse-1 (SGP): [REDACTED-IP]
eni-nyc1 (NYC): [REDACTED-IP]

Deploy command:

cat scanner.py | ssh root@<IP> "cat > /root/ai_scanner.py"

Cron:

echo '*/15 * * * * cd /root && python3 ai_scanner.py >> /root/ai_scanner_cron.log 2>&1' | crontab -