autoreg-skills-max: initial public release (restored, redacted, hardened)

skills/ai-provider-autoreg/references/tavily-onboarding-bypass-and-interceptor.md

@@ -0,0 +1,67 @@
+# Tavily Onboarding Bypass + Network Interceptor (Oct 2026)
+
+As of Oct 2026, tavily.com has a **browser-killing onboarding flow** that blocks all `page.content()` calls and prevents key extraction via page scanning. The solution is a combination of native playwright locators (not JS evaluate clicks) + network response interception.
+
+## Onboarding structure (2 steps)
+
+1. **Cookie consent**: "Accept All Cookies" / "Reject All" / "Cookies Settings" — appears first, blocks interaction with onboarding
+2. **Step 1**: "How will you use Tavily?" — 6 purpose cards (People Research, News & Monitoring, Products & Pricing, Company Research, Coding & Development, Science & Academia) + **"Continue →"** button + "A different way?" link
+3. **Step 2**: (prompts after Continue) — additional preferences + Continue/Get Started/Done button
+
+## Working bypass
+
+```python
+# DISMISS COOKIES — use native locator, not JS evaluate
+await page.locator("button:has-text('Accept All Cookies')").first.click(timeout=4000)
+await page.wait_for_timeout(2000)
+
+# DETECT onboarding by page TEXT (not URL — /home redirects to onboarding without URL change)
+h2 = await page.evaluate("() => (document.body.innerText || '')")
+if 'How will you use' in h2 or 'People Research' in h2:
+    # CLICK purpose card — native locator
+    await page.locator("text=Coding & Development").first.click(timeout=3000)
+    await page.wait_for_timeout(2000)
+    # CLICK Continue
+    await page.locator("button:has-text('Continue')").first.click(timeout=5000)
+    await page.wait_for_timeout(6000)
+    # Repeat for step 2 (up to 3 cycles)
+```
+
+## CRITICAL: page.content() DEADLOCKS on Tavily's React SPA
+
+`await page.content()` hangs indefinitely on Tavily's React pages — the serialization triggers a synchronous re-render that never completes. **Use `page.evaluate("() => document.documentElement.outerHTML")` instead** — returns the same DOM string without deadlocking. This applies to ALL React SPAs where `page.content()` shows >5s latency or hangs.
+
+## Network interceptor (catches keys even when page scan fails)
+
+Tavily's API keys appear in network responses (GET `/api/keys?include_deleted=true`) before they appear in page HTML. Intercept them:
+
+```python
+CAPTURED = []
+async def on_resp(resp):
+    ct = resp.headers.get('content-type', '')
+    if 'json' in ct or 'text/html' in ct:
+        body = await resp.text()
+        if 'tvly-' in body:
+            CAPTURED.append(body)
+            print(f'[INTERCEPT] tvly- at {resp.url[:100]}', flush=True)
+page.on('response', on_resp)
+```
+
+Key format: `tvly-[A-Za-z0-9_-]{20,}` (same regex works).
+
+## Complete working find_key pattern
+
+1. Navigate to `/home` (redirects to onboarding)
+2. Dismiss cookies + walk onboarding (native locators, up to 3 cycles)
+3. Navigate to `/api-keys`
+4. Scan intercepted responses + page outerHTML for `tvly-`
+5. If not found, click "Create API key" / "Create" / "Generate"
+6. Wait 4s, re-scan
+
+## Pitfalls
+
+- **JS evaluate clicks DON'T WORK on React components** — `el.dispatchEvent(new MouseEvent('click', ...))` is silently ignored. Use native playwright `page.locator(...).click()`.
+- **Onboarding URL check is unreliable**: `/home` redirects to `/onboarding/intent` but the URL doesn't always reflect it. Detect by page TEXT content, not URL.
+- **Accept All Cookies button may not be in DOM** on first load — use `.first.click(timeout=4000)` with try/except, not `.count()` + `.click()`.
+- **page.content() hangs** on Tavily's React SPA — use `page.evaluate("() => document.documentElement.outerHTML")`. This is a general React SPA pitfall, not just Tavily.
+- **wait_until='domcontentloaded' vs 'load'**: 'domcontentloaded' may complete before React hydrates. Use 'load' for key-scan pages.