autoreg-skills-max: initial public release (restored, redacted, hardened)
This commit is contained in:
@@ -0,0 +1,38 @@
|
||||
# Scanner v2 Architecture
|
||||
|
||||
## Sources (6 total)
|
||||
|
||||
```
|
||||
scanner.py
|
||||
├── scrape_linuxdo() → linux.do (资源荟萃, 福利羊毛, AI tags)
|
||||
├── scrape_nodeloc() → nodeloc.com (AI, VPS, 羊毛党 tags) + cookies
|
||||
├── scrape_lolz() → lolz.live (forums 86, 54)
|
||||
├── scrape_github() → github.com/search (repos, 3 queries max)
|
||||
├── scrape_reddit() → old.reddit.com (r/freeAI, r/ChatGPT, r/LocalLLaMA)
|
||||
└── scrape_ddg() → html.duckduckgo.com (3 queries)
|
||||
```
|
||||
|
||||
## Flow
|
||||
|
||||
1. `asyncio.gather()` → все 6 источников параллельно
|
||||
2. Каждый scrape возвращает `list[dict]` с ключами: source, title, url, snippet, category, topic
|
||||
3. `classify_topic(title, snippet)` → эмодзи-категория
|
||||
4. Дедипликация по URL (в рамках одного прогона) + глобальная (seen_hashes.json)
|
||||
5. `format_post()` → HTML для Telegram
|
||||
6. `send_telegram()` → HTTP POST
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
CONFIG = {
|
||||
"bot_token": "7922242368:...",
|
||||
"chat_id": [REDACTED-TGID],
|
||||
"state_file": "seen_hashes.json",
|
||||
"log_file": "scanner.log",
|
||||
"request_timeout": 30,
|
||||
}
|
||||
```
|
||||
|
||||
## Nodeloc cookies
|
||||
|
||||
Injected as dict into aiohttp session cookies parameter. Must be updated from browser when `_forum_session` expires.
|
||||
Reference in New Issue
Block a user