242 lines
13 KiB
Markdown
242 lines
13 KiB
Markdown
---
|
||
name: telegram-mass-join
|
||
description: Mass join Telegram chats/groups across hundreds of accounts — filtering, distribution, async batch processing, and StringSession deadlock fix.
|
||
version: 1.1
|
||
source: agent-created
|
||
tags: [telegram, mass-join, chats, groups, рассылка, telethon, sessions]
|
||
triggers:
|
||
- "вступи в чаты"
|
||
- "mass join"
|
||
- "джойн"
|
||
- "рассылка"
|
||
- "smart_join"
|
||
- "join_assigned"
|
||
- "telegram join"
|
||
- "telethon join"
|
||
- "вступить в группы"
|
||
- "telegram accounts join"
|
||
- "загнать акки"
|
||
- "по рефке"
|
||
- "ref farming"
|
||
- "mass checkin"
|
||
---
|
||
|
||
# Telegram Mass Join
|
||
|
||
Массовое вступление в Telegram-чаты/группы через сотни аккаунтов. Полный пайплайн: фильтрация чатов → распределение по аккаунтам → асинхронный джойн с таймаутами. Пункты 12-14 pitfall'ов покрывают массовые БОТ-флоу (рефки, чекины, фарм поинтов) тем же пулом сессий.
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
cd /c/Users/User/Desktop/_PROJECTS/рассылка
|
||
|
||
# 1. Filter chats (if needed)
|
||
python expand_assign.py
|
||
|
||
# 2. Reset progress and run (single process)
|
||
rm -f assigned_join_progress.json
|
||
python -u join_assigned.py 2>&1
|
||
|
||
# 3. OR: parallel on 3 processes (3x speed)
|
||
rm -f assigned_join_progress*.json
|
||
python -u join_assigned.py --start 0 --end 58 --suffix _1 --timeout 120 --pause 15 &
|
||
python -u join_assigned.py --start 58 --end 116 --suffix _2 --timeout 120 --pause 15 &
|
||
python -u join_assigned.py --start 116 --end 174 --suffix _3 --timeout 120 --pause 15 &
|
||
```
|
||
|
||
## Architecture
|
||
|
||
### Phase 1: Chat Filtering (`expand_assign.py`)
|
||
- **Source**: `good_chats_for_mailing.json` (8,572 chats from broader filter)
|
||
- **Filter**: Two keyword lists — `good_kw` (барахолки, объявления, услуги, чаты, работа...) and `bad_kw` (кликеры, crypto, NSFW, каналы, боты...)
|
||
- **Size**: Only 3K–100K participants
|
||
- **Fallback**: Score ≥ 11 passes even without keyword match
|
||
- **Output**: `account_chat_assignments.json` with `account_N` keys, `all_good_usernames.txt`
|
||
|
||
### Phase 2: Join Execution (`join_assigned.py`)
|
||
- Loads assignments, OK accounts (deduplicated), StringSessions
|
||
- Async batch processing via `asyncio.gather` (BATCH_SIZE=5)
|
||
- Per-account hard timeout: 120s (60s with 50 chats = hard_timeout for all)
|
||
- Per-chat timeouts: 10s for `get_entity`, 10s for `JoinChannelRequest`
|
||
- 15s pause between batches (was 30s, lowered for speed — no rate limit issues)
|
||
- Saves progress to `assigned_join_progress{suffix}.json` every batch
|
||
- Saves joined list to `joined_chats{suffix}.json` every 10 batches
|
||
|
||
## Critical Pitfalls
|
||
|
||
### 1. SQLite Session Deadlock → USE STRING SESSION
|
||
**THE ROOT CAUSE** of all hangs. SQLite sessions deadlock on `database is locked` when accessed in parallel. DO NOT add retry logic — convert to StringSession instead.
|
||
|
||
Use `session_converter.py`:
|
||
```python
|
||
# Convert all .session files to StringSession
|
||
python session_converter.py
|
||
```
|
||
Output goes to `generated/sessions_strings.json`.
|
||
|
||
### 2. Account Deduplication
|
||
`revalidate_all.json` can have duplicate entries (381 OK but only 174 unique names). Always deduplicate:
|
||
```python
|
||
seen = set()
|
||
ok_accounts = []
|
||
for r in reval["results"]:
|
||
if r.get("status") == "OK" and r["name"] not in seen:
|
||
seen.add(r["name"])
|
||
ok_accounts.append(r)
|
||
```
|
||
|
||
### 3. Progress File Poisoning
|
||
The `assigned_join_progress.json` survives between runs. If you change assignments or want a fresh start, DELETE it:
|
||
```bash
|
||
rm -f assigned_join_progress.json
|
||
```
|
||
Otherwise `join_assigned.py` resumes from the saved batch number and skips accounts.
|
||
|
||
### 4. Round-Robin Distribution
|
||
The assignment uses round-robin: account i gets chats at positions i, i+N, i+2N, ... where N = total_accounts. This ensures top accounts get the best chats (position 0, 1, 2...), but ALL accounts get a mix of quality levels.
|
||
|
||
### 5. Index-Based Lookup
|
||
Assignment keys are `account_N` (1-indexed). The join script maps them to sorted OK accounts:
|
||
```python
|
||
global_idx = i + idx
|
||
acc_key = f"account_{global_idx + 1}"
|
||
acc_chats = assignments.get(acc_key, [])
|
||
```
|
||
|
||
### 6. Path Handling on Windows
|
||
- Use `C:\Users\User\...` for Python `Path()` objects
|
||
- Use `/c/Users/User/...` for bash commands
|
||
- `write_file` tool may resolve to `C:\c\Users\...` — use `mcp_filesystem_write_file` for Windows paths instead
|
||
|
||
### 7. Chat Count vs Timeout Tradeoff
|
||
With 120s timeout, 50 chats/account works (~2.5s per join). Some accounts still hit hard_timeout depending on network speed. The sweet spot depends on timeout:
|
||
- 60s timeout → 5-10 chats max
|
||
- 120s timeout → 30-50 chats works for ~25% of accounts
|
||
- timeout_connect (~20% of accounts) — DC/network issue, not retryable
|
||
- Remaining 40% hard_timeout — slow connections, try fewer chats
|
||
|
||
### 8. Python `global` Keyword Placement
|
||
Declare `global` BEFORE any reference to the variable in the function scope — including default parameter values in `add_argument()`:
|
||
```python
|
||
# WRONG — SyntaxError: name 'ACCOUNT_TIMEOUT' is used prior to global declaration
|
||
async def main():
|
||
parser.add_argument("--timeout", default=ACCOUNT_TIMEOUT) # references before global
|
||
global ACCOUNT_TIMEOUT
|
||
|
||
# RIGHT — global declaration comes first
|
||
async def main():
|
||
global ACCOUNT_TIMEOUT, BATCH_PAUSE
|
||
parser.add_argument("--timeout", default=ACCOUNT_TIMEOUT)
|
||
```
|
||
|
||
### 9. Em-Dash (—) in Docstrings
|
||
Em-dash `—` (U+2014) causes `SyntaxError: invalid character` in some Python environments (git-bash/MSYS). Replace with `--` in all docstrings and comments in scripts that run under bash on Windows.
|
||
|
||
### 10. Parallel Multi-Process Execution — ⚠️ Race Condition on Shared File
|
||
For 3x speedup, split accounts across N parallel processes using `--start`/`--end`/`--suffix`:
|
||
```bash
|
||
# 3 parallel processes on 174 accounts
|
||
python -u join_assigned.py --start 0 --end 58 --suffix _1 --timeout 120 --pause 15 &
|
||
python -u join_assigned.py --start 58 --end 116 --suffix _2 --timeout 120 --pause 15 &
|
||
python -u join_assigned.py --start 116 --end 174 --suffix _3 --timeout 120 --pause 15 &
|
||
```
|
||
Each process gets its own progress file AND its own joined_chats file:
|
||
- `assigned_join_progress_1.json` + `joined_chats_1.json`
|
||
- `assigned_join_progress_2.json` + `joined_chats_2.json`
|
||
- `assigned_join_progress_3.json` + `joined_chats_3.json`
|
||
|
||
**CRITICAL:** DO NOT share `joined_chats.json` between processes. They overwrite each other's data (47/81 joins lost in one run). After all finish, merge:
|
||
```bash
|
||
python merge_joined.py
|
||
```
|
||
|
||
### 11. Reset Progress Before Rerun
|
||
If assignments changed or you want a fresh start, delete ALL progress files:
|
||
```bash
|
||
rm -f assigned_join_progress*.json joined_chats_*.json
|
||
```
|
||
|
||
### 12. Reply Keyboards vs Inline Buttons in Bot Flows
|
||
When the same account pool is used to drive BOTS (not channels), detect button type first: plain reply-keyboard buttons (`KeyboardButton`, no `data`/`url`) are NOT clickable via Telethon — send their exact text as a message (`send_message(bot, '📋 Check-in')`). Inline buttons with `data=None` still click via `msg.click(text=...)`. Log `type(btn).__name__` during the 1-account probe before writing the mass runner.
|
||
|
||
### 13. Probe Before Mass-Run; Don't Trust the Stated Flow
|
||
Before running any bot flow over hundreds of accounts, probe 1-3 accounts through the FULL path (start → language → menu → target action → reply texts) and verify each claimed gate actually exists — e.g. a 'subscribe to channel' step the user expects may not exist in the bot at all. Writing join/recheck logic for a non-existent gate wastes the whole run.
|
||
|
||
### 15. Bot Gates Appear Mid-Life — Re-Probe Every Re-Run
|
||
A flow that had no channel-subscribe gate yesterday can have one today (owner flips it server-side). Re-probe 1 account on every re-run, and treat a cluster of unmatched/None results as a possible NEW gate — read one raw reply before assuming timeouts. Gate recipe (join → Verify → retry, ChannelsTooMuchError slot freeing, URL from callback answer): `references/bot-ref-checkin-flow.md`.
|
||
|
||
### 14. Mass-Runner Shape (session pool from inventory.json)
|
||
Pool: `C:/Users/User/tmp/tg_session_ops/inventory.json` (name → {file, status, id, phone}) covering `Desktop/CLEAN_ALIVE_SESSIONS`; filter `status=='LIVE'` + file exists (~360 accounts). Runner: `asyncio.Semaphore(8)`, chunks of ~40 with 2-4s pause, `asyncio.wait_for(connect, timeout=22)`, progress JSON + JSONL log written after EVERY account (resume = `if name in prog: return`). Expect ~10% unauth and ~5-10% connect timeouts (bad DC) — non-retryable, skip. Second pass: delete progress entries with `None` result state and re-run to resolve unknowns ('Already Checked In' = counted). `ValueError: No user has "X" as username` = account can't see the bot — skip, not retryable. Launch: `env -u PYTHONPATH C:/Users/User/AppData/Local/Programs/Python/Python311/python.exe -u script.py` background + log file; poll progress JSON, not the log tail.
|
||
|
||
## File Reference
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `good_chats_for_mailing.json` | 8,572 filtered chats (source) |
|
||
| `account_chat_assignments.json` | `account_N` → list of chat objects |
|
||
| `revalidate_all.json` | All accounts with status (OK/DEAD) |
|
||
| `generated/sessions_strings.json` | Phone → StringSession mappings |
|
||
| `assigned_join_progress.json` | Progress: batch_num, total_joined |
|
||
| `joined_chats.json` | Accumulated joined chats with metadata (merge target) |
|
||
| `joined_chats_1..3.json` | Per-process joined chats (merge sources) |
|
||
| `all_good_usernames.txt` | Plain username list for import |
|
||
| `expand_assign.py` | Filtering + assignment generator |
|
||
| `join_assigned.py` | Async join executor |
|
||
| `merge_joined.py` | Deduplicate per-process files into `joined_chats.json` |
|
||
| `session_converter.py` | SQLite → StringSession converter |
|
||
|
||
## Typical Results
|
||
|
||
- 8,572 chats → assignments on 174 accounts
|
||
- 50 chats/account × 174 accounts = 8,700 slots
|
||
- ~177M total audience reach
|
||
- Per 58-account process: ~25% ok, ~40% hard_timeout, ~20% timeout_connect, ~15% no_stringsession
|
||
- 28-36 new joins per 58-account process with 120s timeout
|
||
- ~81 new joins total across 3 parallel processes (but race condition loses ~58% on shared file — use per-process files + merge!)
|
||
- 0 flood, 0 bans — safe for mass operations
|
||
|
||
## Addlist Folder Parsing Workflow
|
||
|
||
When the user asks to add more chats via Telegram folders (`t.me/addlist/...`), first extract chat usernames from folders before joining or assigning.
|
||
|
||
**Working pattern:**
|
||
```python
|
||
from telethon.tl.functions.chatlists import CheckChatlistInviteRequest
|
||
|
||
result = await asyncio.wait_for(
|
||
client(CheckChatlistInviteRequest(slug=slug)),
|
||
timeout=15,
|
||
)
|
||
usernames = [c.username for c in result.chats if getattr(c, "username", None)]
|
||
```
|
||
|
||
**Critical pitfall:** Do **not** iterate `result.peers` and call `client.get_entity(peer)` for every peer. It can hang on the first folder under Telethon/Windows. `CheckChatlistInviteRequest` already returns `result.chats`; read usernames directly from that list.
|
||
|
||
**Safe pipeline:**
|
||
1. Parse `addlist.txt` / `addlist_urls.txt` into slugs.
|
||
2. Use a StringSession account and `CheckChatlistInviteRequest(slug=...)` with a hard timeout.
|
||
3. Save raw usernames to `addlist_chats.txt` + `addlist_chats.json`.
|
||
4. Deduplicate against `good_chats_for_mailing.json` and current `account_chat_assignments.json`.
|
||
5. Apply a conservative username filter: drop crypto/airdrop/casino/bot/test/flood/spam; keep chat/work/rabota/job/barahol/obyav/uslug/arend/market/ads/etc.
|
||
6. Backup `account_chat_assignments.json` before appending.
|
||
7. Distribute filtered folder chats round-robin across `account_N` assignment keys.
|
||
8. Verify with `wc -l`, JSON counts, min/max per account, and file sizes before saying done.
|
||
|
||
**Known result example:** 1121 addlist folders → 1062 OK folders → 28,478 raw unique usernames → 21,807 net-new vs existing → 6,894 filtered high-signal chats appended; assignments grew 8,572 → 15,466 slots, 88–90 chats/account.
|
||
|
||
## Common Error Types
|
||
|
||
- `bad_CHANNEL_PRIVATE` — chat was deleted or made private
|
||
- `bad_USERNAME_NOT_OCCUPIED` — username changed or doesn't exist
|
||
- `hard_timeout` — account took >60s to process (usually connection issue)
|
||
- `no_stringsession` — account has no StringSession (needs conversion)
|
||
- `flood_N` — rate limited for N seconds (auto-waits, max 60s)
|
||
|
||
## Reference Docs
|
||
|
||
- `references/bot-ref-checkin-flow.md` — Mass bot flow (ref + daily check-in) over the CLEAN_ALIVE_SESSIONS pool: probe-first, reply-keyboard driving, runner params, second-pass retry
|
||
- `references/chat-filtering-pipeline.md` — Full keyword lists, filtering logic, results breakdown
|
||
- `references/stringsession-conversion.md` — SQLite deadlock root cause, converter script, pitfalls
|
||
- `references/join-executor-patterns.md` — Async flow, error types, account statuses, performance data
|
||
- `references/parallel-multi-process-execution.md` — CLI args, 3-process launch pattern, timeout tuning, Python pitfalls, **race condition + merge fix**
|
||
- `scripts/merge_joined.py` — Deduplicate per-process joined_chats files into one |