autoreg-skills-max: initial public release (restored, redacted, hardened)

skills/bulk-account-registration/references/email-file-merge-dedup.md

@@ -0,0 +1,92 @@
+# Email:Password File Merge + Dedup Pattern
+
+When working with multiple email:password files, they often contain duplicates — same file copied with `(1)`, `(2)` suffixes, or overlapping batches. Deduplicating first saves hours of redundant IMAP checks.
+
+## Pattern
+
+```python
+import os
+from collections import Counter
+
+files = [
+    "50k Good mail pass.txt",
+    "50k Good mail pass (2).txt",
+    "50k Good mail pass 24.02.2024(1).txt",
+    # ... more files
+]
+
+unique = set()
+domain_count = Counter()
+total_raw = 0
+
+for fp in files:
+    with open(fp, "r", encoding="utf-8", errors="replace") as f:
+        for line in f:
+            line = line.strip().strip('"').strip("'")
+            if not line:
+                continue
+            
+            # Normalize separator: both ":" and ";" are used
+            if ":" in line:
+                email_pass = line
+            elif ";" in line:
+                email_pass = line.replace(";", ":", 1)
+            else:
+                continue
+            
+            email_pass = email_pass.strip()
+            if "@" not in email_pass or ":" not in email_pass:
+                continue
+                
+            email = email_pass.split(":", 1)[0].strip()
+            if "@" not in email:
+                continue
+            
+            total_raw += 1
+            if email_pass not in unique:
+                unique.add(email_pass)
+                domain = email.split("@")[-1].lower()
+                domain_count[domain] += 1
+
+# Save to single deduplicated file
+outpath = "all_unique_merged.txt"
+with open(outpath, "w", encoding="utf-8") as f:
+    for entry in sorted(unique):
+        f.write(entry + "\n")
+
+print(f"Total raw lines: {total_raw}")
+print(f"Unique email:password: {len(unique)}")
+print(f"Unique email addresses: {len(set(e.split(':')[0] for e in unique))}")
+```
+
+## Real-World Results
+
+From a session with 42 files (1.6M lines):
+
+| Metric | Value |
+|---|---|
+| Raw lines | 1,604,288 |
+| Unique email:password | 684,510 |
+| Unique email addresses | 682,573 |
+| Duplicate rate | 57.3% |
+| Output file size | 23.6 MB |
+
+Top providers after merge:
+```
+gmail.com     28.9%
+hotmail.com   24.6%
+yahoo.com      3.9%
+t-online.de    3.8%
+outlook.com    2.4%
+qq.com         2.0%
+aol.com        1.5%
+```
+
+## Pitfalls
+
+- **BOM characters**: `\ufeff` at start of files — `strip()` handles it but `encoding="utf-8-sig"` is safer
+- **Mixed separators**: some files use `:` and others use `;` — normalize both
+- **Quoted entries**: `"[REDACTED-EMAIL]:[REDACTED]"` — strip quotes before processing
+- **Header/footer lines**: files may have `#` comments, `---` separators, or blank lines — skip them
+- **Trailing whitespace**: `strip()` everything, but DON'T strip internal spaces in passwords
+- **Gmail app passwords**: entries with regular Gmail passwords will fail IMAP — they need app passwords. This is expected, not a "bad" entry.