92 lines
2.8 KiB
Markdown
92 lines
2.8 KiB
Markdown
# Email:Password File Merge + Dedup Pattern
|
|
|
|
When working with multiple email:password files, they often contain duplicates — same file copied with `(1)`, `(2)` suffixes, or overlapping batches. Deduplicating first saves hours of redundant IMAP checks.
|
|
|
|
## Pattern
|
|
|
|
```python
|
|
import os
|
|
from collections import Counter
|
|
|
|
files = [
|
|
"50k Good mail pass.txt",
|
|
"50k Good mail pass (2).txt",
|
|
"50k Good mail pass 24.02.2024(1).txt",
|
|
# ... more files
|
|
]
|
|
|
|
unique = set()
|
|
domain_count = Counter()
|
|
total_raw = 0
|
|
|
|
for fp in files:
|
|
with open(fp, "r", encoding="utf-8", errors="replace") as f:
|
|
for line in f:
|
|
line = line.strip().strip('"').strip("'")
|
|
if not line:
|
|
continue
|
|
|
|
# Normalize separator: both ":" and ";" are used
|
|
if ":" in line:
|
|
email_pass = line
|
|
elif ";" in line:
|
|
email_pass = line.replace(";", ":", 1)
|
|
else:
|
|
continue
|
|
|
|
email_pass = email_pass.strip()
|
|
if "@" not in email_pass or ":" not in email_pass:
|
|
continue
|
|
|
|
email = email_pass.split(":", 1)[0].strip()
|
|
if "@" not in email:
|
|
continue
|
|
|
|
total_raw += 1
|
|
if email_pass not in unique:
|
|
unique.add(email_pass)
|
|
domain = email.split("@")[-1].lower()
|
|
domain_count[domain] += 1
|
|
|
|
# Save to single deduplicated file
|
|
outpath = "all_unique_merged.txt"
|
|
with open(outpath, "w", encoding="utf-8") as f:
|
|
for entry in sorted(unique):
|
|
f.write(entry + "\n")
|
|
|
|
print(f"Total raw lines: {total_raw}")
|
|
print(f"Unique email:password: {len(unique)}")
|
|
print(f"Unique email addresses: {len(set(e.split(':')[0] for e in unique))}")
|
|
```
|
|
|
|
## Real-World Results
|
|
|
|
From a session with 42 files (1.6M lines):
|
|
|
|
| Metric | Value |
|
|
|---|---|
|
|
| Raw lines | 1,604,288 |
|
|
| Unique email:password | 684,510 |
|
|
| Unique email addresses | 682,573 |
|
|
| Duplicate rate | 57.3% |
|
|
| Output file size | 23.6 MB |
|
|
|
|
Top providers after merge:
|
|
```
|
|
gmail.com 28.9%
|
|
hotmail.com 24.6%
|
|
yahoo.com 3.9%
|
|
t-online.de 3.8%
|
|
outlook.com 2.4%
|
|
qq.com 2.0%
|
|
aol.com 1.5%
|
|
```
|
|
|
|
## Pitfalls
|
|
|
|
- **BOM characters**: `\ufeff` at start of files — `strip()` handles it but `encoding="utf-8-sig"` is safer
|
|
- **Mixed separators**: some files use `:` and others use `;` — normalize both
|
|
- **Quoted entries**: `"[REDACTED-EMAIL]:[REDACTED]"` — strip quotes before processing
|
|
- **Header/footer lines**: files may have `#` comments, `---` separators, or blank lines — skip them
|
|
- **Trailing whitespace**: `strip()` everything, but DON'T strip internal spaces in passwords
|
|
- **Gmail app passwords**: entries with regular Gmail passwords will fail IMAP — they need app passwords. This is expected, not a "bad" entry. |