68 lines
3.7 KiB
Markdown
68 lines
3.7 KiB
Markdown
# Tavily Onboarding Bypass + Network Interceptor (Oct 2026)
|
|
|
|
As of Oct 2026, tavily.com has a **browser-killing onboarding flow** that blocks all `page.content()` calls and prevents key extraction via page scanning. The solution is a combination of native playwright locators (not JS evaluate clicks) + network response interception.
|
|
|
|
## Onboarding structure (2 steps)
|
|
|
|
1. **Cookie consent**: "Accept All Cookies" / "Reject All" / "Cookies Settings" — appears first, blocks interaction with onboarding
|
|
2. **Step 1**: "How will you use Tavily?" — 6 purpose cards (People Research, News & Monitoring, Products & Pricing, Company Research, Coding & Development, Science & Academia) + **"Continue →"** button + "A different way?" link
|
|
3. **Step 2**: (prompts after Continue) — additional preferences + Continue/Get Started/Done button
|
|
|
|
## Working bypass
|
|
|
|
```python
|
|
# DISMISS COOKIES — use native locator, not JS evaluate
|
|
await page.locator("button:has-text('Accept All Cookies')").first.click(timeout=4000)
|
|
await page.wait_for_timeout(2000)
|
|
|
|
# DETECT onboarding by page TEXT (not URL — /home redirects to onboarding without URL change)
|
|
h2 = await page.evaluate("() => (document.body.innerText || '')")
|
|
if 'How will you use' in h2 or 'People Research' in h2:
|
|
# CLICK purpose card — native locator
|
|
await page.locator("text=Coding & Development").first.click(timeout=3000)
|
|
await page.wait_for_timeout(2000)
|
|
# CLICK Continue
|
|
await page.locator("button:has-text('Continue')").first.click(timeout=5000)
|
|
await page.wait_for_timeout(6000)
|
|
# Repeat for step 2 (up to 3 cycles)
|
|
```
|
|
|
|
## CRITICAL: page.content() DEADLOCKS on Tavily's React SPA
|
|
|
|
`await page.content()` hangs indefinitely on Tavily's React pages — the serialization triggers a synchronous re-render that never completes. **Use `page.evaluate("() => document.documentElement.outerHTML")` instead** — returns the same DOM string without deadlocking. This applies to ALL React SPAs where `page.content()` shows >5s latency or hangs.
|
|
|
|
## Network interceptor (catches keys even when page scan fails)
|
|
|
|
Tavily's API keys appear in network responses (GET `/api/keys?include_deleted=true`) before they appear in page HTML. Intercept them:
|
|
|
|
```python
|
|
CAPTURED = []
|
|
async def on_resp(resp):
|
|
ct = resp.headers.get('content-type', '')
|
|
if 'json' in ct or 'text/html' in ct:
|
|
body = await resp.text()
|
|
if 'tvly-' in body:
|
|
CAPTURED.append(body)
|
|
print(f'[INTERCEPT] tvly- at {resp.url[:100]}', flush=True)
|
|
page.on('response', on_resp)
|
|
```
|
|
|
|
Key format: `tvly-[A-Za-z0-9_-]{20,}` (same regex works).
|
|
|
|
## Complete working find_key pattern
|
|
|
|
1. Navigate to `/home` (redirects to onboarding)
|
|
2. Dismiss cookies + walk onboarding (native locators, up to 3 cycles)
|
|
3. Navigate to `/api-keys`
|
|
4. Scan intercepted responses + page outerHTML for `tvly-`
|
|
5. If not found, click "Create API key" / "Create" / "Generate"
|
|
6. Wait 4s, re-scan
|
|
|
|
## Pitfalls
|
|
|
|
- **JS evaluate clicks DON'T WORK on React components** — `el.dispatchEvent(new MouseEvent('click', ...))` is silently ignored. Use native playwright `page.locator(...).click()`.
|
|
- **Onboarding URL check is unreliable**: `/home` redirects to `/onboarding/intent` but the URL doesn't always reflect it. Detect by page TEXT content, not URL.
|
|
- **Accept All Cookies button may not be in DOM** on first load — use `.first.click(timeout=4000)` with try/except, not `.count()` + `.click()`.
|
|
- **page.content() hangs** on Tavily's React SPA — use `page.evaluate("() => document.documentElement.outerHTML")`. This is a general React SPA pitfall, not just Tavily.
|
|
- **wait_until='domcontentloaded' vs 'load'**: 'domcontentloaded' may complete before React hydrates. Use 'load' for key-scan pages.
|