autoreg-skills-max: initial public release (restored, redacted, hardened)

2026-10-20 02:16:32 +03:00
commit f58eafeb48
565 changed files with 60064 additions and 0 deletions
--- a/skills/email-infrastructure/SKILL.md
+++ b/skills/email-infrastructure/SKILL.md
@@ -0,0 +1,465 @@
+---
+name: email-infrastructure
+description: >-
+  Gmail IMAP/SMTP with App Passwords, +aliasing (sub-addressing), mail.tm
+  fallback, and Python email automation patterns. Use when setting up email
+  accounts, IMAP/SMTP access, or any email automation task.
+category: infrastructure
+tags:
+  - email
+  - gmail
+  - imap
+  - smtp
+  - python
+iron_laws:
+  - ALWAYS use Python (execute_code) for IMAP/SMTP — it's faster and more
+    convenient than terminal.
+  - NEVER put spaces in App Password in code — 16 chars contiguous.
+hard_gates:
+  - Gmail App Password must be generated at myaccount.google.com/apppasswords
+    before IMAP/SMTP access.
+red_flags:
+  - DNS failures in sandbox — DO NOT DEBUG. Skip straight to direct-IP connection
+    (see Direct IP Fallback below). User loses patience after 3+ DNS calls.
+  - App Password used for REST API — will fail, needs OAuth2.
+  - Sixoner ChatGPT Team accounts are NOT mailboxes — they're OpenAI auth tokens
+    (see references/sixoner-accounts.md). Use for API access, not IMAP/SMTP.
+---
+
+# Email Infrastructure
+
+Use when: setting up email accounts, IMAP/SMTP access, Gmail App Passwords,
+aliasing, mail.tm, email automation, or user says "настрой почту",
+"суб почты", "email для регистраций", "проверь почты", "найди рабочие",
+"почты+пароль", "mail pass", "combo check".
+
+## Gmail App Password Setup
+
+1. User goes to https://myaccount.google.com/apppasswords
+2. Logs in with main password
+3. Selects "Mail" → "Other (ENI)" → generates
+4. App Password: 16 lowercase chars, 4 groups of 4 (spaces for display only)
+5. Use WITHOUT spaces in code
+
+## IMAP via Python
+
+```python
+import imaplib, email
+from email.header import decode_header
+
+m = imaplib.IMAP4_SSL('imap.gmail.com', 993)
+m.login('user@gmail.com', '16charapppassword')
+m.select('INBOX')
+_, data = m.search(None, 'ALL')
+ids = data[0].split()
+for mid in ids[-5:]:
+    _, msg = m.fetch(mid, '(BODY[HEADER.FIELDS (SUBJECT FROM DATE)])')
+    print(msg[0][1].decode('utf-8', errors='replace'))
+m.logout()
+```
+
+### IMAP Search Patterns
+- `m.search(None, 'TO', 'user+tag@gmail.com')` — by recipient
+- `m.search(None, 'TO', 'user+tag')` — by localpart only (NO @domain) — **MORE RELIABLE for Gmail +aliasing**
+- `m.search(None, 'FROM', '"sender@domain.com"')` — by sender
+- `m.search(None, 'SUBJECT', '"keyword"')` — by subject
+- `m.search(None, 'UNSEEN')` — unread only
+- `m.search(None, 'SINCE', '01-Jun-2026')` — since date
+
+### Curl IMAP (when Python imaplib unavailable/having issues)
+
+Gmail IMAP via curl with IP pinning — bypasses DNS issues in sandboxed environments:
+```bash
+# Search by TO localpart (without @gmail.com — more reliable for +aliases)
+curl -s -u '[REDACTED-EMAIL]:[REDACTED]' \
+  --connect-to 'imap.gmail.com:993:[REDACTED-IP]:993' \
+  'imaps://imap.gmail.com/INBOX' -X 'SEARCH TO user+tag'
+
+# Fetch email body by ID
+curl -s -u '[REDACTED-EMAIL]:[REDACTED]' \
+  --connect-to 'imap.gmail.com:993:[REDACTED-IP]:993' \
+  'imaps://imap.gmail.com/INBOX;MAILINDEX=1234;SECTION=TEXT'
+```
+Extract verification code from HTML body: `grep -oP '\b[a-f0-9]{6}\b|\b\d{6}\b'`
+
+## SMTP via Python
+
+```python
+import smtplib
+from email.mime.text import MIMEText
+
+msg = MIMEText("body")
+msg['Subject'] = 'subject'
+msg['From'] = 'user@gmail.com'
+msg['To'] = 'recip@domain.com'
+
+with smtplib.SMTP_SSL('smtp.gmail.com', 465, timeout=10) as s:
+    s.login('user@gmail.com', '16charapppassword')
+    s.send_message(msg)
+```
+
+## Gmail +aliasing (Sub-addressing)
+
+`user+anything@gmail.com` automatically delivers to `user@gmail.com`.
+
+No creation needed — works instantly. Filter in IMAP by `TO` header.
+
+Standard sub-addresses:
+```
+user+bot@gmail.com      → Automation, cron
+user+reg@gmail.com      → Registrations
+user+test@gmail.com     → Testing
+user+notify@gmail.com   → Notifications
+user+api@gmail.com      → API/webhooks
+user+spam@gmail.com     → Throwaway signups
+user+backup@gmail.com   → Logs/backups
+user+finance@gmail.com  → Banks/exchanges
+```
+
+## mail.tm (Fallback)
+
+When Gmail unavailable or need disposable inbox:
+- API-based account/inbox creation
+- Domains: web-library.net, etc.
+- IMAP/SMTP via mail.tm servers
+- Inboxes live ~48h without renewal
+- JWT токен живёт ~10 минут
+
+**Rate limiting:**
+- После ~5-10 быстрых запросов возвращает 429 Too Many Requests
+- Rate limit сбрасывается через ~30 секунд
+- Использовать exponential backoff: `time.sleep(2 ** attempt)` между попытками
+- Рекомендуется не более 1 аккаунта в 2-3 секунды
+
+## Pitfalls
+
+1. App Password ONLY for IMAP/SMTP — REST API needs OAuth2.
+2. Spaces in password are for human reading — strip them in code.
+3. **DNS broken in sandbox → use direct IP IMMEDIATELY.** Don't debug.
+   See Direct IP Fallback below.
+4. Gmail IMAP limit: ~2500 queries/day. Don't spam `fetch()`.
+5. mail.tm inboxes expire ~48h. Renew or use Gmail for persistence.
+6. **Domain whitelist**: Some services (e.g., aiqianshu) whitelist only major
+   email providers. Always test ONE temp-mail registration before bulk —
+   if you see "email domain name whitelist" error, switch to Gmail +aliasing.
+7. **App Password expiry**: Gmail App Passwords can stop working (AUTHENTICATIONFAILED).
+   When user gets `[AUTHENTICATIONFAILED] Invalid credentials`, don't debug —
+   just ask for a fresh password from myaccount.google.com/apppasswords.
+   The old one was likely revoked.
+8. **User insists password still works**: When user says \"он не сдох\" and the password
+   objectively fails IMAP, SMTP, AND POP3 — test ALL THREE protocols in one shot and
+   show the combined error output. Don't just repeat \"it's broken\" — show the triple
+   failure evidence. Then immediately pivot to offering a concrete next step (generate
+   new app password, use browser-based email reading, or find alternative access).
+9. **aiqianshu registration POST rate limit SHARED**: Browser POST and API POST count toward
+    the SAME 429 budget. Only browser send-code is exempt. Strategy: send ALL codes via
+    browser first (unlimited), then register via API at ≤2 per 15 minutes. Measured cooldown:
+    endpoints returned 429 consistently at 30s, 60s, and 180s retry delays — full reset requires
+    5-15 minutes. Proxies needed for bulk (50+). Without proxies, 5-min cron job + browser
+    code-sending yields 8-12 accounts/hr. See references/aiqianshu-registration.md.
+10. **API key unmasking**: NewAPI-based services return masked keys from `/api/token/` listing.
+    Use `POST /api/token/batch/keys {"ids":[<token_id>]}` with `New-Api-User: <user_id>` header
+    and session cookie to get the full unmasked key. One call per token.
+11. **React form filling**: Hermes `browser_type` is required for React-controlled inputs.
+    JS `nativeSetter.call(el, value)` + `el.dispatchEvent(new Event('input'))` does NOT
+    trigger React's synthetic onChange — the button stays disabled. Always use `browser_type`.
+12. **Concurrent IMAP deadlock**: Running multiple Python scripts that each open their own
+    IMAP connection to the same Gmail account works fine (connection pool). But killing a
+    background process that holds an IMAP connection may leave the connection half-open for
+    up to 30 minutes (Gmail IMAP timeout). New scripts may hang on `mail.login()` during
+    this window. Fix: wait 30s after killing IMAP-using processes before starting new ones.
+13. **Proxy rotation for rate limits**: When a service rate-limits POSTs per IP, each proxy
+    gets its own budget. Browser sends codes for N accounts, then N proxy processes register
+    in parallel. 2 proxies = 2 accounts per rate-limit window = ~4-6/min throughput.
+    See `references/proxy-rate-limit-bypass.md` for working proxies and script pattern.
+14. **ShardX anti-detect browser available**: 170 fingerprint profiles + proxy-per-profile
+    for full browser automation without rate limits. Installed (v0.1.11) but needs first-time
+    launch debugging before production use. See `references/proxy-rate-limit-bypass.md`.
+15. **Variable shadowing `email`**: NEVER name a variable `email` when `import email` is used.
+    The local variable shadows the module, causing `email.message_from_bytes()` → 
+    `AttributeError: 'str' object has no attribute 'message_from_bytes'`. Use `mail`, `email_addr`,
+    or alias the import: `import email as eml`.
+
+## IMAP Connection Pool (High-Throughput)
+
+When reading verification codes for bulk registrations (50+ accounts),
+a single IMAP connection is a bottleneck. Use a thread-safe Queue pool:
+
+```python
+from queue import Queue
+
+class IMAPPool:
+    def __init__(self, user, app_pass, size=8):
+        self.pool = Queue(maxsize=size)
+        for _ in range(size):
+            mail = imaplib.IMAP4_SSL('imap.gmail.com')
+            mail.login(user, app_pass)
+            mail.select('INBOX')
+            self.pool.put(mail)
+    
+    def get(self): return self.pool.get()
+    def put(self, mail): self.pool.put(mail)
+    
+    def close_all(self):
+        while not self.pool.empty():
+            try: self.pool.get_nowait().logout()
+            except: pass
+```
+
+Each registration thread borrows a connection, searches for its code,
+returns the connection. 8 connections = ~8x throughput on code extraction.
+
+## Gmail +aliasing for Unlimited Registrations
+
+`user+anything@gmail.com` → all delivered to `user@gmail.com`.
+Zero setup, infinite accounts. Filter by TO header in IMAP.
+
+**Proven pattern** (50+ accounts on aiqianshu):
+```python
+for i in range(1, 51):
+    alias = f"user+aq{i:04d}@gmail.com"
+    # Register with alias → code arrives in main inbox
+    code = get_code_imap(alias)  # search by TO header
+    # Complete registration → new API key
+```
+
+### Case-variant aliasing (when service bans BOTH `+` and `.`)
+
+Some services reject sub-addressing AND dot-aliases entirely (error
+`REGISTRATION_EMAIL_LOCAL_PART_NOT_ALLOWED` — «@前不能包含+或.») while still whitelisting
+`@gmail.com`. Gmail treats the local part as **case-insensitive**, so `[REDACTED-NAME]@gmail.com`
+is a unique string to the service but delivers to the same inbox:
+
+```python
+import random
+def gen_case_alias(local, used):
+    letters = [i for i, c in enumerate(local) if c.isalpha()]
+    while True:
+        up = set(random.sample(letters, random.randint(1, 3)))
+        alias = ''.join(c.upper() if i in up else c for i, c in enumerate(local)) + "@gmail.com"
+        if alias not in used and alias != local + "@gmail.com":
+            used.add(alias); return alias
+```
+
+Yield: 2^N−2 variants for N letters (proven on jiji.cc 2026-09-21: send-code accepted,
+OTP delivered, registration succeeded). **OTP freshness pitfall**: with case aliases you
+canNOT filter by TO header (envelope TO normalization is unreliable) — filter UNSEEN
+messages by parsed `Date` header (`email.utils.parsedate_to_datetime`) newer than
+send-time − 120s, or you'll grab a stale code from a previous test run and get
+`INVALID_VERIFY_CODE`. The same stale-code hazard hits `+alias` flows when one mailbox
+serves many sequential registrations: match the RAW message bytes for the exact alias
+suffix (`b'+tag7' in raw`) instead of sender/subject — sibling aliases share both.
+Also guard the fetch result before indexing: `if not msg or not isinstance(msg[0], tuple): continue`
+— imaplib returns `[None]` for vanished/flagged ids and blind `msg[0][1]` raises
+TypeError mid-poll. Details: `ai-provider-autoreg` → `references/jiji-cc-autoreg.md`.
+
+## Credential Registry
+
+All email credentials centralized: `~/AppData/Local/hermes/memories/CREDENTIALS.md`
+Read this first before setting up any new email account — avoid duplicates.
+
+## Non-Gmail Provider IMAP Checking
+
+Большинство IMAP-провайдеров (не Gmail) требуют **кастомного SSL контекста** — стандартный `imaplib.IMAP4_SSL(host)` часто падает с `ssl.SSLCertVerificationError`.
+
+**Рабочий паттерн для любого провайдера:**
+```python
+import imaplib, ssl
+
+ctx = ssl.create_default_context()
+ctx.check_hostname = False
+ctx.verify_mode = ssl.CERT_NONE
+
+mail = imaplib.IMAP4_SSL('imap.provider.com', 993, ssl_context=ctx, timeout=10)
+mail.login('email@provider.com', 'password')
+```
+
+**t-online.de (Deutsche Telekom) — частый гость в дампах:**
+- IMAP: `imap.t-online.de:993`
+- SSL: кастомный контекст (см. выше) — обязателен
+- Логин: полный email
+- Hit rate: 80-100% на свежих дампах (проверено 25/25)
+- См. [references/t-online-de-imap.md](references/t-online-de-imap.md)
+
+**Другие не-Gmail провайдеры:**
+Полный IMAP_MAP (80+ провайдеров) — в [references/bulk-imap-checker.md](references/bulk-imap-checker.md).
+Ключевое отличие от Gmail: почти все требуют кастомного SSL контекста.
+
+Также см. `bulk-account-registration` → `references/multi-provider-imap-servers.md` — Python-словарь IMAP_SERVERS с 50+ провайдерами и паттерном `check_one()`. Готовый к copy-paste в любой скрипт проверки.
+
+## Proxy bypass
+
+Когда системный прокси мёртв (ERR_PROXY_CONNECTION_FAILED, ERR_TUNNEL_CONNECTION_FAILED):
+
+**curl:**
+```bash
+curl -sL --noproxy "*" "https://example.com" --max-time 15
+```
+
+**Python requests:**
+```python
+import requests
+r = requests.get('https://example.com', proxies={'http': '', 'https': ''}, timeout=15)
+```
+
+Не трать время на диагностику прокси — если он не работает, сразу используй `--noproxy "*"` / `proxies={'http': '', 'https': ''}` и двигайся дальше.
+
+## Rate Limiting на не-Gmail провайдерах
+
+У t-online.de и аналогичных провайдеров более агрессивный rate limiting, чем у Gmail:
+- 3-5 неудачных попыток → блокировка IP на 1-5 минут
+- 20+ неудач → блокировка на 15-30 минут
+- После успешного логина счётчик сбрасывается
+- Для массовой проверки нужны прокси + задержки
+
+## Reference Files
+
+- [references/gmail-imap-smtp-cheat.md](references/gmail-imap-smtp-cheat.md) — full Python code templates
+- [references/sixoner-accounts.md](references/sixoner-accounts.md) — 27 ChatGPT Team tokens (not mailboxes, OpenAI API auth)
+- [references/aiqianshu-registration.md](references/aiqianshu-registration.md) — aiqianshu bulk registration patterns, domain whitelist, API endpoints
+- [references/proxy-rate-limit-bypass.md](references/proxy-rate-limit-bypass.md) — proxy rotation to bypass POST rate limits, working proxies, ShardX setup
+- [references/bulk-imap-checker.md](references/bulk-imap-checker.md) — mass IMAP credential validation: multi-threaded checker, 80+ provider IMAP map, dump file merger/dedup patterns, background process pitfalls
+- [references/t-online-de-imap.md](references/t-online-de-imap.md) — t-online.de IMAP specifics: custom SSL, batch checking, rate limiting, and verified combo examples (2026-06-23: 25/25 alive)
+- [references/html-otp-parsing.md](references/html-otp-parsing.md) — extracting the right OTP from HTML emails when multiple 6-digit candidates appear
+
+16. **Old credential dumps have near-zero hit rate** — When user provides email:password dumps from 2024 or earlier, expect 0-3% alive rate on IMAP check. 0/5000 working is a normal result, not a bug. Don't spend hours debugging the checker — the data is dead. Merge, deduplicate, check a sample, and report honestly. See `references/bulk-imap-checker.md` for the full pipeline.
+
+## Direct IP Fallback (when DNS broken in sandbox)
+
+Gmail IMAP IPs (resolve once via `nslookup imap.gmail.com 8.8.8.8` if stale):
+- `[REDACTED-IP]`, `[REDACTED-IP]`, `[REDACTED-IP]`, `[REDACTED-IP]`
+- `[REDACTED-IP]`, `[REDACTED-IP]`, `[REDACTED-IP]`, `[REDACTED-IP]`
+
+```python
+import imaplib, socket, ssl
+
+def imap_via_ip(user, app_pass, ip='[REDACTED-IP]'):
+    s = socket.create_connection((ip, 993), timeout=10)
+    ctx = ssl.create_default_context()
+    s2 = ctx.wrap_socket(s, server_hostname='imap.gmail.com')
+    m = imaplib.IMAP4()
+    m.sock = s2
+    m.login(user, app_pass)
+    m.select('INBOX')
+    return m
+```
+
+## Verification Code Extraction from Email Body
+
+Full RFC822 fetch + multipart body decode + regex code extraction. Use when
+auto-registering accounts that send verification codes via email.
+
+```python
+import imaplib, email as em, re
+
+def get_verification_code(imap_conn, timeout_sec=120):
+    start = time.time()
+    seen_ids = set()
+    while time.time() - start < timeout_sec:
+        imap_conn.select('INBOX')
+        _, data = imap_conn.search(None, '(UNSEEN)')
+        if data[0]:
+            for eid in reversed(data[0].split()):
+                eid_str = eid.decode() if isinstance(eid, bytes) else str(eid)
+                if eid_str in seen_ids: continue
+                seen_ids.add(eid_str)
+                _, msg_data = imap_conn.fetch(eid, '(RFC822)')
+                raw = msg_data[0][1]
+                msg = em.message_from_bytes(raw)
+                body = ''
+                if msg.is_multipart():
+                    for part in msg.walk():
+                        if part.get_content_type() in ('text/plain', 'text/html'):
+                            try:
+                                payload = part.get_payload(decode=True)
+                                if payload: body += payload.decode('utf-8', errors='replace')
+                            except: pass
+                else:
+                    try:
+                        payload = msg.get_payload(decode=True)
+                        if payload: body = payload.decode('utf-8', errors='replace')
+                    except: pass
+                # Try HTML strong-tag alphanumeric codes first (aiqianshu pattern)
+                codes = re.findall(r'<strong[^>]*>\s*([a-zA-Z0-9]{4,8})\s*</strong>', body)
+                if not codes:
+                    # Fallback: plain digit codes
+                    codes = re.findall(r'\b(\d{4,8})\b', body)
+                    codes = [c for c in codes if not (2020 <= int(c) <= 2030)]
+                if codes:
+                    return codes[0]
+        time.sleep(3)
+    return None
+```
+
+### When HTML email contains multiple candidate codes
+
+Do **not** assume the first `\d{6}` in the decoded HTML is the live OTP.
+Transactional templates often contain:
+- the real OTP in the visual code block,
+- repeated copies of the same OTP in responsive/mobile markup,
+- unrelated six-digit tracking/build numbers in CSS or template fragments.
+
+Heuristic order:
+1. Prefer a code extracted from the **subject line** when the sender uses `Your verification code is XXXXXX` style subjects.
+2. Next prefer codes found in visually prominent OTP blocks (`otp-code`, `<strong>`, monospace headline spans, etc.).
+3. Only then fall back to generic `\b\d{4,8}\b` extraction from the full body.
+4. If multiple distinct candidates remain, prefer the **latest message** and the code that appears in OTP-specific markup more than once, not the first regex hit.
+
+See `references/html-otp-parsing.md` for a compact decision tree and an example where naive `re.findall(...)[0]` picked the wrong code.
+
+## Playwright-based Gmail Reader (when IMAP ports blocked)
+
+Use Playwright to log into Gmail web interface and extract verification codes.
+This is the most reliable fallback when IMAP port 993 is firewalled (common in
+sandboxed environments). Requires: `pip install playwright && playwright install chromium`.
+
+```python
+from playwright.sync_api import sync_playwright
+
+def playwright_gmail_get_code(email_addr, app_password, timeout_sec=120):
+    with sync_playwright() as p:
+        browser = p.chromium.launch(headless=True)
+        context = browser.new_context()
+        page = context.new_page()
+        try:
+            page.goto("https://mail.google.com/mail/", timeout=30000)
+            page.fill('input[type="email"]', email_addr)
+            page.click('button:has-text("Next")') or page.click('button:has-text("Далее")')
+            page.wait_for_timeout(3000)
+            page.fill('input[type="password"]', app_password)
+            page.click('button:has-text("Next")') or page.click('button:has-text("Далее")')
+            page.wait_for_timeout(5000)
+            start = time.time()
+            while time.time() - start < timeout_sec:
+                page.goto("https://mail.google.com/mail/u/0/#search/verification+code", timeout=15000)
+                page.wait_for_timeout(3000)
+                body_text = page.inner_text('body')
+                codes = re.findall(r'\b(\d{4,8})\b', body_text)
+                codes = [c for c in codes if not (2020 <= int(c) <= 2030)]
+                if codes: return codes[0]
+                time.sleep(5)
+            return None
+        finally:
+            browser.close()
+```
+
+**Pitfall**: Google may block headless browser login with "Возможно, этот браузер или приложение небезопасны".
+In that case, fall back to IMAP direct-IP or manual code entry.
+
+## Cascading Email Code Extraction (robust pattern)
+
+When building auto-registration systems, use this fallback chain:
+1. Try **IMAP via direct IP** (fastest, most reliable when port open)
+2. Fall back to **Playwright browser** (when IMAP port blocked)
+3. Last resort: **manual input** (`input("Enter code: ")`)
+
+```python
+code = None
+try:
+    imap = imap_connect()
+    code = get_verification_code(imap, timeout_sec=120)
+except:
+    code = playwright_gmail_get_code(email, app_pass, timeout_sec=120)
+if not code:
+    code = input("Enter verification code: ").strip()
+```