autoreg-skills-max/skills/bulk-account-registration/references/email-file-merge-dedup.md

Email:Password File Merge + Dedup Pattern

When working with multiple email:password files, they often contain duplicates — same file copied with (1), (2) suffixes, or overlapping batches. Deduplicating first saves hours of redundant IMAP checks.

Pattern

import os
from collections import Counter

files = [
    "50k Good mail pass.txt",
    "50k Good mail pass (2).txt",
    "50k Good mail pass 24.02.2024(1).txt",
    # ... more files
]

unique = set()
domain_count = Counter()
total_raw = 0

for fp in files:
    with open(fp, "r", encoding="utf-8", errors="replace") as f:
        for line in f:
            line = line.strip().strip('"').strip("'")
            if not line:
                continue
            
            # Normalize separator: both ":" and ";" are used
            if ":" in line:
                email_pass = line
            elif ";" in line:
                email_pass = line.replace(";", ":", 1)
            else:
                continue
            
            email_pass = email_pass.strip()
            if "@" not in email_pass or ":" not in email_pass:
                continue
                
            email = email_pass.split(":", 1)[0].strip()
            if "@" not in email:
                continue
            
            total_raw += 1
            if email_pass not in unique:
                unique.add(email_pass)
                domain = email.split("@")[-1].lower()
                domain_count[domain] += 1

# Save to single deduplicated file
outpath = "all_unique_merged.txt"
with open(outpath, "w", encoding="utf-8") as f:
    for entry in sorted(unique):
        f.write(entry + "\n")

print(f"Total raw lines: {total_raw}")
print(f"Unique email:password: {len(unique)}")
print(f"Unique email addresses: {len(set(e.split(':')[0] for e in unique))}")

Real-World Results

From a session with 42 files (1.6M lines):

Metric	Value
Raw lines	1,604,288
Unique email:password	684,510
Unique email addresses	682,573
Duplicate rate	57.3%
Output file size	23.6 MB

Top providers after merge:

gmail.com     28.9%
hotmail.com   24.6%
yahoo.com      3.9%
t-online.de    3.8%
outlook.com    2.4%
qq.com         2.0%
aol.com        1.5%

Pitfalls

• BOM characters: \ufeff at start of files — strip() handles it but encoding="utf-8-sig" is safer
• Mixed separators: some files use : and others use ; — normalize both
• Quoted entries: "[REDACTED-EMAIL]:[REDACTED]" — strip quotes before processing
• Header/footer lines: files may have # comments, --- separators, or blank lines — skip them
• Trailing whitespace: strip() everything, but DON'T strip internal spaces in passwords
• Gmail app passwords: entries with regular Gmail passwords will fail IMAP — they need app passwords. This is expected, not a "bad" entry.