vlad/autoreg-skills-max
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f58eafeb48979991591e8b4c53ff7b2c015091a7
autoreg-skills-max/skills/ai-freebie-scanner/references/scanner-v3.1-architecture.md

2.7 KiB
Raw Blame History

Scanner v3.1 Architecture & Quality Scoring

Flow

19 sources (parallel asyncio.gather)
  → raw items (185-238 per first run)
  → quality_score() filter
  → dedup by URL (seen_v3.json)
  → top 15 by score
  → build_post() with @ikhdev1 format
  → send_tg() via Telegram Bot API

quality_score() Algorithm

def quality_score(item: dict) -> int:
    score = 1  # base
    
    # Source quality
    if src in direct_sources:    score += 3  # NodeLoc, Lolz
    if src in premium_sources:   score += 4  # KeyLeaks, BazaarLink, Airforce
    
    # Content signals
    if has_model_name(full_text):     score += 1  # claude, gpt, deepseek...
    if has_instructions(full_text):   score += 1  # register, signup, setup...
    if has_keys_tokens(full_text):    score += 1  # api key, token, credits...
    if is_abuse_jailbreak(full_text): score += 2  # jailbreak, bypass, exploit
    
    # Detail richness
    if len(snippet) > 50:   score += 2
    if len(snippet) > 150:  score += 1
    
    return min(score, 10)

Source Performance Matrix

Source First Run Items Subsequent Reliability Notes
GitHub awesome-lists 150-180 0-5 High Read raw README.md from 4 repos
DuckDuckGo 5-15 3-10 Medium HTML scraping, rate limited
NodeLoc 10-30 5-15 High Needs fresh cookies
HN Algolia 5-10 2-5 High JSON API, fast
KeyLeaks 1 0 Low Static page, DNS issues locally
Blogs (8 sources) 8 0-2 Low One-time checks via state keys
Lolz.live 2-8 2-8 Medium HTML selectors fragile
GitHub Search 1-5 0-2 Low Rate limited without token
Linux.do 0 0 Dead CF JS Challenge blocks everything
Reddit 0 0 Dead 403 on all datacenter IPs

State Management

  • seen_v3.json: MD5 hashes of visited URLs, max 10000 entries
  • Each source marks URLs as seen via mark(url)
  • is_new(url) checks hash not in set
  • Clear state for full rescan: rm seen_v3.json

Telegram Delivery

  • Bot: @dfhjdfghfjdhfhbot (token: [REDACTED])
  • Chat: @[REDACTED-NAME] ([REDACTED-TGID])
  • Method: sendMessage with parse_mode=HTML
  • Limit: 4096 chars (message auto-truncates at 3950 + footer)
  • HTML escaping via safe_html() is MANDATORY — any unescaped < or & causes "Unclosed start tag" 400 error

Server Deployment

Both servers run identical /root/ai_scanner.py via cron every 15 min:

gothbreach-abuse-1 (SGP): [REDACTED-IP]
eni-nyc1 (NYC): [REDACTED-IP]

Deploy command:

cat scanner.py | ssh root@<IP> "cat > /root/ai_scanner.py"

Cron:

echo '*/15 * * * * cd /root && python3 ai_scanner.py >> /root/ai_scanner_cron.log 2>&1' | crontab -