Tavily Onboarding Bypass + Network Interceptor (Oct 2026)

As of Oct 2026, tavily.com has a browser-killing onboarding flow that blocks all page.content() calls and prevents key extraction via page scanning. The solution is a combination of native playwright locators (not JS evaluate clicks) + network response interception.

Onboarding structure (2 steps)

Cookie consent: "Accept All Cookies" / "Reject All" / "Cookies Settings" — appears first, blocks interaction with onboarding
Step 1: "How will you use Tavily?" — 6 purpose cards (People Research, News & Monitoring, Products & Pricing, Company Research, Coding & Development, Science & Academia) + "Continue →" button + "A different way?" link
Step 2: (prompts after Continue) — additional preferences + Continue/Get Started/Done button

Working bypass

# DISMISS COOKIES — use native locator, not JS evaluate
await page.locator("button:has-text('Accept All Cookies')").first.click(timeout=4000)
await page.wait_for_timeout(2000)

# DETECT onboarding by page TEXT (not URL — /home redirects to onboarding without URL change)
h2 = await page.evaluate("() => (document.body.innerText || '')")
if 'How will you use' in h2 or 'People Research' in h2:
    # CLICK purpose card — native locator
    await page.locator("text=Coding & Development").first.click(timeout=3000)
    await page.wait_for_timeout(2000)
    # CLICK Continue
    await page.locator("button:has-text('Continue')").first.click(timeout=5000)
    await page.wait_for_timeout(6000)
    # Repeat for step 2 (up to 3 cycles)

CRITICAL: page.content() DEADLOCKS on Tavily's React SPA

await page.content() hangs indefinitely on Tavily's React pages — the serialization triggers a synchronous re-render that never completes. Use page.evaluate("() => document.documentElement.outerHTML") instead — returns the same DOM string without deadlocking. This applies to ALL React SPAs where page.content() shows >5s latency or hangs.

Network interceptor (catches keys even when page scan fails)

Tavily's API keys appear in network responses (GET /api/keys?include_deleted=true) before they appear in page HTML. Intercept them:

CAPTURED = []
async def on_resp(resp):
    ct = resp.headers.get('content-type', '')
    if 'json' in ct or 'text/html' in ct:
        body = await resp.text()
        if 'tvly-' in body:
            CAPTURED.append(body)
            print(f'[INTERCEPT] tvly- at {resp.url[:100]}', flush=True)
page.on('response', on_resp)

Key format: tvly-[A-Za-z0-9_-]{20,} (same regex works).

Complete working find_key pattern

Navigate to /home (redirects to onboarding)
Dismiss cookies + walk onboarding (native locators, up to 3 cycles)
Navigate to /api-keys
Scan intercepted responses + page outerHTML for tvly-
If not found, click "Create API key" / "Create" / "Generate"
Wait 4s, re-scan

Pitfalls

JS evaluate clicks DON'T WORK on React components — el.dispatchEvent(new MouseEvent('click', ...)) is silently ignored. Use native playwright page.locator(...).click().
Onboarding URL check is unreliable: /home redirects to /onboarding/intent but the URL doesn't always reflect it. Detect by page TEXT content, not URL.
Accept All Cookies button may not be in DOM on first load — use .first.click(timeout=4000) with try/except, not .count() + .click().
page.content() hangs on Tavily's React SPA — use page.evaluate("() => document.documentElement.outerHTML"). This is a general React SPA pitfall, not just Tavily.
wait_until='domcontentloaded' vs 'load': 'domcontentloaded' may complete before React hydrates. Use 'load' for key-scan pages.

3.7 KiB Raw Blame History