85 lines
2.7 KiB
Markdown
85 lines
2.7 KiB
Markdown
# Scanner v3.1 Architecture & Quality Scoring
|
|
|
|
## Flow
|
|
```
|
|
19 sources (parallel asyncio.gather)
|
|
→ raw items (185-238 per first run)
|
|
→ quality_score() filter
|
|
→ dedup by URL (seen_v3.json)
|
|
→ top 15 by score
|
|
→ build_post() with @ikhdev1 format
|
|
→ send_tg() via Telegram Bot API
|
|
```
|
|
|
|
## quality_score() Algorithm
|
|
|
|
```python
|
|
def quality_score(item: dict) -> int:
|
|
score = 1 # base
|
|
|
|
# Source quality
|
|
if src in direct_sources: score += 3 # NodeLoc, Lolz
|
|
if src in premium_sources: score += 4 # KeyLeaks, BazaarLink, Airforce
|
|
|
|
# Content signals
|
|
if has_model_name(full_text): score += 1 # claude, gpt, deepseek...
|
|
if has_instructions(full_text): score += 1 # register, signup, setup...
|
|
if has_keys_tokens(full_text): score += 1 # api key, token, credits...
|
|
if is_abuse_jailbreak(full_text): score += 2 # jailbreak, bypass, exploit
|
|
|
|
# Detail richness
|
|
if len(snippet) > 50: score += 2
|
|
if len(snippet) > 150: score += 1
|
|
|
|
return min(score, 10)
|
|
```
|
|
|
|
## Source Performance Matrix
|
|
|
|
| Source | First Run Items | Subsequent | Reliability | Notes |
|
|
|--------|----------------|------------|-------------|-------|
|
|
| GitHub awesome-lists | 150-180 | 0-5 | High | Read raw README.md from 4 repos |
|
|
| DuckDuckGo | 5-15 | 3-10 | Medium | HTML scraping, rate limited |
|
|
| NodeLoc | 10-30 | 5-15 | High | Needs fresh cookies |
|
|
| HN Algolia | 5-10 | 2-5 | High | JSON API, fast |
|
|
| KeyLeaks | 1 | 0 | Low | Static page, DNS issues locally |
|
|
| Blogs (8 sources) | 8 | 0-2 | Low | One-time checks via state keys |
|
|
| Lolz.live | 2-8 | 2-8 | Medium | HTML selectors fragile |
|
|
| GitHub Search | 1-5 | 0-2 | Low | Rate limited without token |
|
|
| Linux.do | 0 | 0 | Dead | CF JS Challenge blocks everything |
|
|
| Reddit | 0 | 0 | Dead | 403 on all datacenter IPs |
|
|
|
|
## State Management
|
|
|
|
- `seen_v3.json`: MD5 hashes of visited URLs, max 10000 entries
|
|
- Each source marks URLs as seen via `mark(url)`
|
|
- `is_new(url)` checks hash not in set
|
|
- Clear state for full rescan: `rm seen_v3.json`
|
|
|
|
## Telegram Delivery
|
|
|
|
- Bot: @dfhjdfghfjdhfhbot (token: [REDACTED])
|
|
- Chat: @[REDACTED-NAME] ([REDACTED-TGID])
|
|
- Method: `sendMessage` with `parse_mode=HTML`
|
|
- Limit: 4096 chars (message auto-truncates at 3950 + footer)
|
|
- HTML escaping via `safe_html()` is MANDATORY — any unescaped `<` or `&` causes "Unclosed start tag" 400 error
|
|
|
|
## Server Deployment
|
|
|
|
Both servers run identical `/root/ai_scanner.py` via cron every 15 min:
|
|
|
|
```
|
|
gothbreach-abuse-1 (SGP): [REDACTED-IP]
|
|
eni-nyc1 (NYC): [REDACTED-IP]
|
|
```
|
|
|
|
Deploy command:
|
|
```bash
|
|
cat scanner.py | ssh root@<IP> "cat > /root/ai_scanner.py"
|
|
```
|
|
|
|
Cron:
|
|
```bash
|
|
echo '*/15 * * * * cd /root && python3 ai_scanner.py >> /root/ai_scanner_cron.log 2>&1' | crontab -
|
|
```
|