autoreg-skills-max/skills/ai-freebie-scanner/references/scanner-v2-architecture.md

Scanner v2 Architecture

Sources (6 total)

scanner.py
├── scrape_linuxdo()    → linux.do (资源荟萃, 福利羊毛, AI tags)
├── scrape_nodeloc()    → nodeloc.com (AI, VPS, 羊毛党 tags) + cookies
├── scrape_lolz()       → lolz.live (forums 86, 54)
├── scrape_github()     → github.com/search (repos, 3 queries max)
├── scrape_reddit()     → old.reddit.com (r/freeAI, r/ChatGPT, r/LocalLLaMA)
└── scrape_ddg()        → html.duckduckgo.com (3 queries)

Flow

1. asyncio.gather() → все 6 источников параллельно
2. Каждый scrape возвращает list[dict] с ключами: source, title, url, snippet, category, topic
3. classify_topic(title, snippet) → эмодзи-категория
4. Дедипликация по URL (в рамках одного прогона) + глобальная (seen_hashes.json)
5. format_post() → HTML для Telegram
6. send_telegram() → HTTP POST

Config

CONFIG = {
    "bot_token": "7922242368:...",
    "chat_id": [REDACTED-TGID],
    "state_file": "seen_hashes.json",
    "log_file": "scanner.log",
    "request_timeout": 30,
}

Nodeloc cookies

Injected as dict into aiohttp session cookies parameter. Must be updated from browser when _forum_session expires.