39 lines
1.3 KiB
Markdown
39 lines
1.3 KiB
Markdown
# Scanner v2 Architecture
|
||
|
||
## Sources (6 total)
|
||
|
||
```
|
||
scanner.py
|
||
├── scrape_linuxdo() → linux.do (资源荟萃, 福利羊毛, AI tags)
|
||
├── scrape_nodeloc() → nodeloc.com (AI, VPS, 羊毛党 tags) + cookies
|
||
├── scrape_lolz() → lolz.live (forums 86, 54)
|
||
├── scrape_github() → github.com/search (repos, 3 queries max)
|
||
├── scrape_reddit() → old.reddit.com (r/freeAI, r/ChatGPT, r/LocalLLaMA)
|
||
└── scrape_ddg() → html.duckduckgo.com (3 queries)
|
||
```
|
||
|
||
## Flow
|
||
|
||
1. `asyncio.gather()` → все 6 источников параллельно
|
||
2. Каждый scrape возвращает `list[dict]` с ключами: source, title, url, snippet, category, topic
|
||
3. `classify_topic(title, snippet)` → эмодзи-категория
|
||
4. Дедипликация по URL (в рамках одного прогона) + глобальная (seen_hashes.json)
|
||
5. `format_post()` → HTML для Telegram
|
||
6. `send_telegram()` → HTTP POST
|
||
|
||
## Config
|
||
|
||
```python
|
||
CONFIG = {
|
||
"bot_token": "7922242368:...",
|
||
"chat_id": [REDACTED-TGID],
|
||
"state_file": "seen_hashes.json",
|
||
"log_file": "scanner.log",
|
||
"request_timeout": 30,
|
||
}
|
||
```
|
||
|
||
## Nodeloc cookies
|
||
|
||
Injected as dict into aiohttp session cookies parameter. Must be updated from browser when `_forum_session` expires.
|