# Email:Password File Merge + Dedup Pattern

When working with multiple email:password files, they often contain duplicates — same file copied with `(1)`, `(2)` suffixes, or overlapping batches. Deduplicating first saves hours of redundant IMAP checks.

## Pattern

```python
import os
from collections import Counter

files = [
    "50k Good mail pass.txt",
    "50k Good mail pass (2).txt",
    "50k Good mail pass 24.02.2024(1).txt",
    # ... more files
]

unique = set()
domain_count = Counter()
total_raw = 0

for fp in files:
    with open(fp, "r", encoding="utf-8", errors="replace") as f:
        for line in f:
            line = line.strip().strip('"').strip("'")
            if not line:
                continue
            
            # Normalize separator: both ":" and ";" are used
            if ":" in line:
                email_pass = line
            elif ";" in line:
                email_pass = line.replace(";", ":", 1)
            else:
                continue
            
            email_pass = email_pass.strip()
            if "@" not in email_pass or ":" not in email_pass:
                continue
                
            email = email_pass.split(":", 1)[0].strip()
            if "@" not in email:
                continue
            
            total_raw += 1
            if email_pass not in unique:
                unique.add(email_pass)
                domain = email.split("@")[-1].lower()
                domain_count[domain] += 1

# Save to single deduplicated file
outpath = "all_unique_merged.txt"
with open(outpath, "w", encoding="utf-8") as f:
    for entry in sorted(unique):
        f.write(entry + "\n")

print(f"Total raw lines: {total_raw}")
print(f"Unique email:password: {len(unique)}")
print(f"Unique email addresses: {len(set(e.split(':')[0] for e in unique))}")
```

## Real-World Results

From a session with 42 files (1.6M lines):

| Metric | Value |
|---|---|
| Raw lines | 1,604,288 |
| Unique email:password | 684,510 |
| Unique email addresses | 682,573 |
| Duplicate rate | 57.3% |
| Output file size | 23.6 MB |

Top providers after merge:
```
gmail.com     28.9%
hotmail.com   24.6%
yahoo.com      3.9%
t-online.de    3.8%
outlook.com    2.4%
qq.com         2.0%
aol.com        1.5%
```

## Pitfalls

- **BOM characters**: `\ufeff` at start of files — `strip()` handles it but `encoding="utf-8-sig"` is safer
- **Mixed separators**: some files use `:` and others use `;` — normalize both
- **Quoted entries**: `"[REDACTED-EMAIL]:[REDACTED]"` — strip quotes before processing
- **Header/footer lines**: files may have `#` comments, `---` separators, or blank lines — skip them
- **Trailing whitespace**: `strip()` everything, but DON'T strip internal spaces in passwords
- **Gmail app passwords**: entries with regular Gmail passwords will fail IMAP — they need app passwords. This is expected, not a "bad" entry.