86 lines
3.1 KiB
Markdown
86 lines
3.1 KiB
Markdown
# Parallel Multi-Process Join Execution
|
||
|
||
## Motivation
|
||
Single `join_assigned.py` on 174 accounts × 50 chats with 60s timeout = ~50 minutes. Most of that time is waiting for Telegram responses. Running 3 parallel processes on different account ranges cuts it to ~17 minutes.
|
||
|
||
## CLI Arguments Added to `join_assigned.py`
|
||
|
||
```
|
||
--start N First account index (0-based, default 0)
|
||
--end N Last account index (exclusive, default all)
|
||
--timeout S Account hard timeout in seconds (default 120)
|
||
--pause S Inter-batch pause in seconds (default 15)
|
||
--suffix STR Progress file suffix (e.g. "_1", "_2", "_3")
|
||
```
|
||
|
||
## Launch Pattern
|
||
|
||
174 accounts → 3 processes of 58 each:
|
||
|
||
```bash
|
||
python -u join_assigned.py --start 0 --end 58 --suffix _1 --timeout 120 --pause 15 &
|
||
python -u join_assigned.py --start 58 --end 116 --suffix _2 --timeout 120 --pause 15 &
|
||
python -u join_assigned.py --start 116 --end 174 --suffix _3 --timeout 120 --pause 15 &
|
||
```
|
||
|
||
## Why 120s Timeout for 50 Chats
|
||
|
||
Each chat join takes ~2-3 seconds (get_entity + JoinChannelRequest + random delay). 50 chats × 2.5s avg = 125s. 120s timeout covers most accounts while still catching true hangs.
|
||
|
||
With 60s timeout and 50 chats → almost all accounts hit hard_timeout. 120s is the sweet spot.
|
||
|
||
## Progress File Isolation
|
||
|
||
Each process uses its own progress file:
|
||
- `assigned_join_progress_1.json`
|
||
- `assigned_join_progress_2.json`
|
||
- `assigned_join_progress_3.json`
|
||
|
||
Each writes to its own `joined_chats{suffix}.json` (NOT shared — race condition otherwise). After all finish:
|
||
|
||
```bash
|
||
python merge_joined.py
|
||
```
|
||
|
||
`merge_joined.py` loads all `joined_chats*.json`, deduplicates by username/id, writes to `joined_chats.json`.
|
||
|
||
### ⚠️ Race Condition on Shared Progress File
|
||
When 3+ processes read/write the same `joined_chats.json`, they overwrite each other's data. Last save wins → ~58% data loss (47 of 81 new joins lost in one run). **Each process MUST use its own suffix throughout:**
|
||
|
||
| File | Process _1 | Process _2 | Process _3 |
|
||
|------|-----------|-----------|-----------|
|
||
| Joined chats | `joined_chats_1.json` | `joined_chats_2.json` | `joined_chats_3.json` |
|
||
| Progress | `assigned_join_progress_1.json` | `assigned_join_progress_2.json` | `assigned_join_progress_3.json` |
|
||
|
||
Merge step is mandatory after parallel runs.
|
||
|
||
## Python Pitfalls Encountered
|
||
|
||
### 1. `global` before any reference
|
||
```python
|
||
# CRASH: SyntaxError: name 'ACCOUNT_TIMEOUT' is used prior to global declaration
|
||
async def main():
|
||
parser.add_argument("--timeout", default=ACCOUNT_TIMEOUT)
|
||
global ACCOUNT_TIMEOUT # too late
|
||
|
||
# FIX: global first
|
||
async def main():
|
||
global ACCOUNT_TIMEOUT, BATCH_PAUSE # must be FIRST
|
||
parser.add_argument("--timeout", default=ACCOUNT_TIMEOUT)
|
||
```
|
||
|
||
### 2. Em-dash in docstrings
|
||
```python
|
||
# CRASH: SyntaxError: invalid character '—' (U+2014)
|
||
"""Join chats — fast and targeted."""
|
||
|
||
# FIX: use ASCII double dash
|
||
"""Join chats -- fast and targeted."""
|
||
```
|
||
|
||
## Expected Results
|
||
|
||
- 3× 58 accounts = 174 total
|
||
- 3× 12 batches = 36 total batches
|
||
- ~17 minutes total runtime
|
||
- 500-1500 new joins (from 50 chats/account, most error out) |