# Parallel Multi-Process Join Execution ## Motivation Single `join_assigned.py` on 174 accounts × 50 chats with 60s timeout = ~50 minutes. Most of that time is waiting for Telegram responses. Running 3 parallel processes on different account ranges cuts it to ~17 minutes. ## CLI Arguments Added to `join_assigned.py` ``` --start N First account index (0-based, default 0) --end N Last account index (exclusive, default all) --timeout S Account hard timeout in seconds (default 120) --pause S Inter-batch pause in seconds (default 15) --suffix STR Progress file suffix (e.g. "_1", "_2", "_3") ``` ## Launch Pattern 174 accounts → 3 processes of 58 each: ```bash python -u join_assigned.py --start 0 --end 58 --suffix _1 --timeout 120 --pause 15 & python -u join_assigned.py --start 58 --end 116 --suffix _2 --timeout 120 --pause 15 & python -u join_assigned.py --start 116 --end 174 --suffix _3 --timeout 120 --pause 15 & ``` ## Why 120s Timeout for 50 Chats Each chat join takes ~2-3 seconds (get_entity + JoinChannelRequest + random delay). 50 chats × 2.5s avg = 125s. 120s timeout covers most accounts while still catching true hangs. With 60s timeout and 50 chats → almost all accounts hit hard_timeout. 120s is the sweet spot. ## Progress File Isolation Each process uses its own progress file: - `assigned_join_progress_1.json` - `assigned_join_progress_2.json` - `assigned_join_progress_3.json` Each writes to its own `joined_chats{suffix}.json` (NOT shared — race condition otherwise). After all finish: ```bash python merge_joined.py ``` `merge_joined.py` loads all `joined_chats*.json`, deduplicates by username/id, writes to `joined_chats.json`. ### ⚠️ Race Condition on Shared Progress File When 3+ processes read/write the same `joined_chats.json`, they overwrite each other's data. Last save wins → ~58% data loss (47 of 81 new joins lost in one run). **Each process MUST use its own suffix throughout:** | File | Process _1 | Process _2 | Process _3 | |------|-----------|-----------|-----------| | Joined chats | `joined_chats_1.json` | `joined_chats_2.json` | `joined_chats_3.json` | | Progress | `assigned_join_progress_1.json` | `assigned_join_progress_2.json` | `assigned_join_progress_3.json` | Merge step is mandatory after parallel runs. ## Python Pitfalls Encountered ### 1. `global` before any reference ```python # CRASH: SyntaxError: name 'ACCOUNT_TIMEOUT' is used prior to global declaration async def main(): parser.add_argument("--timeout", default=ACCOUNT_TIMEOUT) global ACCOUNT_TIMEOUT # too late # FIX: global first async def main(): global ACCOUNT_TIMEOUT, BATCH_PAUSE # must be FIRST parser.add_argument("--timeout", default=ACCOUNT_TIMEOUT) ``` ### 2. Em-dash in docstrings ```python # CRASH: SyntaxError: invalid character '—' (U+2014) """Join chats — fast and targeted.""" # FIX: use ASCII double dash """Join chats -- fast and targeted.""" ``` ## Expected Results - 3× 58 accounts = 174 total - 3× 12 batches = 36 total batches - ~17 minutes total runtime - 500-1500 new joins (from 50 chats/account, most error out)