# Tavily Onboarding Bypass + Network Interceptor (Oct 2026)

As of Oct 2026, tavily.com has a **browser-killing onboarding flow** that blocks all `page.content()` calls and prevents key extraction via page scanning. The solution is a combination of native playwright locators (not JS evaluate clicks) + network response interception.

## Onboarding structure (2 steps)

1. **Cookie consent**: "Accept All Cookies" / "Reject All" / "Cookies Settings" — appears first, blocks interaction with onboarding
2. **Step 1**: "How will you use Tavily?" — 6 purpose cards (People Research, News & Monitoring, Products & Pricing, Company Research, Coding & Development, Science & Academia) + **"Continue →"** button + "A different way?" link
3. **Step 2**: (prompts after Continue) — additional preferences + Continue/Get Started/Done button

## Working bypass

```python
# DISMISS COOKIES — use native locator, not JS evaluate
await page.locator("button:has-text('Accept All Cookies')").first.click(timeout=4000)
await page.wait_for_timeout(2000)

# DETECT onboarding by page TEXT (not URL — /home redirects to onboarding without URL change)
h2 = await page.evaluate("() => (document.body.innerText || '')")
if 'How will you use' in h2 or 'People Research' in h2:
    # CLICK purpose card — native locator
    await page.locator("text=Coding & Development").first.click(timeout=3000)
    await page.wait_for_timeout(2000)
    # CLICK Continue
    await page.locator("button:has-text('Continue')").first.click(timeout=5000)
    await page.wait_for_timeout(6000)
    # Repeat for step 2 (up to 3 cycles)
```

## CRITICAL: page.content() DEADLOCKS on Tavily's React SPA

`await page.content()` hangs indefinitely on Tavily's React pages — the serialization triggers a synchronous re-render that never completes. **Use `page.evaluate("() => document.documentElement.outerHTML")` instead** — returns the same DOM string without deadlocking. This applies to ALL React SPAs where `page.content()` shows >5s latency or hangs.

## Network interceptor (catches keys even when page scan fails)

Tavily's API keys appear in network responses (GET `/api/keys?include_deleted=true`) before they appear in page HTML. Intercept them:

```python
CAPTURED = []
async def on_resp(resp):
    ct = resp.headers.get('content-type', '')
    if 'json' in ct or 'text/html' in ct:
        body = await resp.text()
        if 'tvly-' in body:
            CAPTURED.append(body)
            print(f'[INTERCEPT] tvly- at {resp.url[:100]}', flush=True)
page.on('response', on_resp)
```

Key format: `tvly-[A-Za-z0-9_-]{20,}` (same regex works).

## Complete working find_key pattern

1. Navigate to `/home` (redirects to onboarding)
2. Dismiss cookies + walk onboarding (native locators, up to 3 cycles)
3. Navigate to `/api-keys`
4. Scan intercepted responses + page outerHTML for `tvly-`
5. If not found, click "Create API key" / "Create" / "Generate"
6. Wait 4s, re-scan

## Pitfalls

- **JS evaluate clicks DON'T WORK on React components** — `el.dispatchEvent(new MouseEvent('click', ...))` is silently ignored. Use native playwright `page.locator(...).click()`.
- **Onboarding URL check is unreliable**: `/home` redirects to `/onboarding/intent` but the URL doesn't always reflect it. Detect by page TEXT content, not URL.
- **Accept All Cookies button may not be in DOM** on first load — use `.first.click(timeout=4000)` with try/except, not `.count()` + `.click()`.
- **page.content() hangs** on Tavily's React SPA — use `page.evaluate("() => document.documentElement.outerHTML")`. This is a general React SPA pitfall, not just Tavily.
- **wait_until='domcontentloaded' vs 'load'**: 'domcontentloaded' may complete before React hydrates. Use 'load' for key-scan pages.