--- name: windows-browser-automation description: Control Comet/Edge/Chrome on Windows via CDP (Playwright/browser-use) or Win32 SendKeys fallback. Covers Comet cold-start CDP, real-profile attach, address-bar JS injection, window focus, SendKeys escaping, and cookie extraction. triggers: - "Comet browser" - "SendKeys" - "Win32 browser control" - "no CDP port" - "browser cookie extraction" - "автоматизация браузера" - "comet.exe" - "Comet CDP" - "comet agent" - "comet-browser-control" - "remote-debugging-port" - "remote-allow-origins" - "browser-use real profile" - 'Playwright CDP Comet' - 'открой браузер' - 'включи браузер' - 'я регнусь' - 'ручная регистрация' - 'start chrome' - 'visible browser' - 'manual browser' --- # Windows Browser Automation via Win32 API ## When to use - **Comet browser** (Perplexity build) — CDP is available BUT only at cold start. An already-running Comet ignores `--remote-debugging-port`. To get CDP on the real profile: kill all `comet.exe` processes, then launch with `comet.exe --remote-debugging-port=9233 --remote-allow-origins=* --user-data-dir=C:\Users\User\AppData\Local\Perplexity\Comet\User Data`. The `--remote-allow-origins=*` flag is REQUIRED for WebSocket CDP connections (Playwright/Puppeteer/browser-use) — without it you get HTTP 403 on WebSocket handshake. - Edge/Chrome CDP port crashed or unreachable - MCP Playwright server is dead but the browser IS running with the target page open - Need to inject JavaScript, copy page content, or navigate in a running browser - **Comet Agent project** lives at `C:\Users\User\Desktop\_PROJECTS\comet-browser-control\` — use `launcher_start.py` for one-click cold-start Comet + CDP + hidden agent server on `:8765` ## Core pattern ```python import win32com.client, win32gui, win32clipboard, time shell = win32com.client.Dispatch("WScript.Shell") # 1. Find the window HWND # Use: python -c "import win32gui; ..." or tasklist + window title search hwnd = 6425040 # example: "Dashboard | TECH HAUS Workshop – Comet" # 2. Activate it win32gui.SetForegroundWindow(hwnd) time.sleep(0.3) # Always wait between actions # 3. Send keystrokes shell.SendKeys("^l") # Ctrl+L = address bar shell.SendKeys("{ESC}") # Close popups shell.SendKeys("^a") # Select all shell.SendKeys("^c") # Copy # 4. Read clipboard win32clipboard.OpenClipboard() data = win32clipboard.GetClipboardData() win32clipboard.CloseClipboard() ``` ## SendKeys escaping rules - **Parentheses** `(` `)` MUST be escaped: `{(}` and `{)}` — they're special keys - **Curly braces** `{` `}` wrap key names like `{ENTER}`, `{TAB}`, `{F12}` - **Special chars** `+` `^` `%` `~` are modifiers (Shift/Ctrl/Alt) — escape with braces: `{+}` - **Period** `.` is safe in most contexts ## JavaScript injection via address bar ```python shell.SendKeys("%d") # Alt+D = focus address bar (more reliable than Ctrl+L) shell.SendKeys("javascript:void{(}{(}document.body.innerText=document.cookie{)}{)}") shell.SendKeys("{ENTER}") time.sleep(1.5) # Then Ctrl+A, Ctrl+C to copy the result ``` **Pitfall:** `document.cookie` only returns non-httpOnly cookies. Supabase auth cookies are httpOnly=false so this works. `th_chunk_gate` may be httpOnly=true and invisible to JS. ## Comet-specific facts - Path: `C:\Program Files\Perplexity\Comet\Application\comet.exe` - 48+ processes normally — this is architectural, not a leak - Real user profile: `C:\Users\User\AppData\Local\Perplexity\Comet\User Data\` (profile name "Nikita", Default directory) - **CDP IS available on Comet** but only at cold start. Chromium ignores `--remote-debugging-port` if an instance is already running (it just forwards the URL and exits). To get CDP on the real profile: 1. Kill all `comet.exe` processes 2. Launch: `comet.exe --remote-debugging-port=9233 --remote-allow-origins=* --user-data-dir=C:\Users\User\AppData\Local\Perplexity\Comet\User Data --no-first-run --no-default-browser-check --disable-features=Translate` 3. `--remote-allow-origins=*` is REQUIRED for WebSocket CDP connections — without it, Playwright/Puppeteer/browser-use get HTTP 403 on WebSocket handshake 4. Verified: `/json/version` returns `Chrome/149.0.7827.1095`, `/json/list` shows all real tabs - **Comet Agent project** at `C:\Users\User\Desktop\_PROJECTS\comet-browser-control\` — `launcher_start.py` handles the cold-start kill+restart flow automatically. Also includes `comet_gui_control.py` (UIA fallback), `cdp_scan.py`, `playwright_attach.py`, `browser_use_cdp_check.py`, `clone_to_chrome_cdp.py`. - Cookies DB: `User Data\Default\Network\Cookies` — SQLite, **locked while browser runs** - While running: only part of cookies may be visible on disk; auth cookies can be in memory - To extract cookies: (a) F12 → Application → Cookies manually, or (b) close Comet → DB unlocks → read with sqlite3 ## CDP Auth Cookie Injection Pattern (2026-07-16) **When you have auth cookies from API sign-in but can't set them in the browser UI** — use Playwright `connect_over_cdp` + `context.add_cookies`. This bridges API-authenticated sessions into a real browser: ```python import asyncio, json from playwright.async_api import async_playwright async def inject_and_navigate(api_cookies: dict, target_url: str): async with async_playwright() as p: browser = await p.chromium.connect_over_cdp("http://127.0.0.1:9222") context = browser.contexts[0] page = context.pages[0] if context.pages else await context.new_page() await context.add_cookies([ {"name": k, "value": v, "domain": ".target.com", "path": "/", "httpOnly": False, "secure": True, "sameSite": "Lax"} for k, v in api_cookies.items() ]) await page.goto(target_url, wait_until="load") # Now authenticated — buttons like "Claim Offer" are visible ``` **Starting Chrome with CDP for cookie injection:** ```python import subprocess, time, requests subprocess.run("taskkill /F /IM chrome.exe 2>nul", shell=True) subprocess.Popen([ r"C:\Program Files\Google\Chrome\Application\chrome.exe", "--remote-debugging-port=9222", "--remote-allow-origins=*", # REQUIRED for WebSocket CDP r"--user-data-dir=C:\Users\User\AppData\Local\hermes\cache\chrome-profile", "--no-first-run", "--no-default-browser-check", "about:blank" ], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) time.sleep(3) # Verify r = requests.get("http://127.0.0.1:9222/json/version") print(f"Chrome: {r.json().get('Browser')}") ``` **Pitfalls:** - `--remote-allow-origins=*` is REQUIRED — without it Playwright WebSocket handshake returns 403 - Use `wait_until="load"` (not `"networkidle"`) for Cloudflare-protected sites — `networkidle` times out on CF-challenged pages - Clerk `__client` cookie carries the session; losing it = "Signed out" - Fresh `--user-data-dir` avoids profile lock conflicts with existing Chrome instances - `taskkill` kills all Chrome processes — warn user if they have important tabs open ## Find all Comet/Chromium windows ```python import win32gui def enum_callback(hwnd, results): if win32gui.IsWindowVisible(hwnd): title = win32gui.GetWindowText(hwnd) if title: results.append((hwnd, title, win32gui.GetWindowRect(hwnd))) results = [] win32gui.EnumWindows(enum_callback, results) for hwnd, title, rect in results: if 'Comet' in title or 'premium' in title.lower(): print(f"HWND={hwnd} | \"{title[:80]}\" | rect={rect}") ``` ## Pitfalls 1. **Popup interception:** If Ctrl+A copies dropdown content instead of page, close popups with `{ESC}` first 2. **SendKeys timing:** Always `time.sleep(0.3)` between actions — too fast and keystrokes merge 3. **Window focus:** `SetForegroundWindow` can fail silently if another app steals focus — verify with `GetForegroundWindow()` 4. **javascript: URLs:** Some browsers block these from address bar for security — fall back to DevTools Console 5. **Clipboard races:** Other apps may overwrite clipboard between SendKeys and read — use `win32clipboard` immediately 6. **Playwright CDP + real Comet profile:** do NOT reuse `ctx.pages[0]` for checks; real profiles often have YouTube/X/Stripe/file tabs and title/navigation can hang. For `--goto`, open a disposable `ctx.new_page()`, wrap title/goto/close in timeouts, then close the disposable page. 7. **browser-use 0.13.x + real profile:** its SecurityWatchdog treats existing `file://` tabs as disallowed and may close them on attach. Before `BrowserSession.start()` monkeypatch `SecurityWatchdog._is_url_allowed` to allow `file://`, `chrome://`, `devtools://` (or use a dedicated clean Comet Agent profile). 8. **Comet CDP WebSocket 403:** if Playwright/Puppeteer/browser-use fail with `WebSocketBadStatusException: Handshake status 403`, Comet was launched without `--remote-allow-origins=*`. Add it to the launch args and restart Comet. Also pass `suppress_origin=True` to `websocket.create_connection()` when talking CDP directly (plain websocket-client) — Chrome 152+ rejects connections with an `http://127.0.0.1:PORT` Origin header. 9. **OAuth consent opens in a NEW TAB:** Google OAuth flows (and most SSO) spawn a separate tab for the consent screen (`accounts.google.com/signin/oauth/id`). Your existing websocket stays bound to the old tab — `document.querySelector` finds nothing. Fix: re-fetch `http://127.0.0.1:9222/json`, pick the tab by URL substring, open a fresh websocket to its `webSocketDebuggerUrl`. Managed (Workspace/Cloud Identity) accounts additionally spawn `chrome://managed-user-profile-notice/` — filter tabs by `type=="page"` and target URL, never `tabs[0]`. 10. **React controlled inputs:** native-setter + `dispatchEvent('input')` works for text fields, but form submits sometimes silently no-op or lose focus. More reliable: `element.focus()` via Runtime.evaluate, then CDP `Input.insertText` (real keystroke path), then click submit. Verify `input.value` after typing before clicking. ## Pre-requisites ```bash pip install pywin32 ``` `win32com`, `win32gui`, `win32clipboard` are all in `pywin32`. ## Reference files - `references/comet-browser-control-project.md` — full project structure, all scripts, config, verified dependencies - `references/browser-use-security-watchdog-patch.md` — monkeypatch to prevent browser-use from closing `file://` tabs on real Comet profiles - `references/comet-cookie-extraction.md` — cookie extraction methods - `references/unjail-premium-accessor.md` — premium.unjail.ai access patterns - `references/groq-auth-flow.md` — Groq console auth flow: working URL, magic-link pattern, pyautogui field positions - `references/cloudflare-turnstile-limitations.md` — Cloudflare Turnstile automation barriers - `references/spa-auth-form-debugging.md` — headless SPA auth/OTP form failures: API-response gating vs "Loading..." buttons, false hasAccount pre-checks, screenshot font-hangs, noscript-challenge detection, stalled post-OTP redirect fix, diagnostic script skeleton - `references/spa-trial-autoreg-verified-flow.md` — VERIFIED end-to-end autoreg pattern (2026-09-10, ProxyScrape): headed Chrome CDP + Turnstile auto-solve + JS-click submit + IMAP hex-code extraction pitfalls + placeholder-credential trap ## Form filling via Tab navigation (captcha bypass) When CDP/browser automation is blocked by reCAPTCHA Enterprise v3, SendKeys is the LAST RESORT that works because it simulates real human keystrokes on the desktop. The browser sees real keyboard events, not injected JavaScript. ```python import win32gui, win32com.client, win32clipboard, time, secrets, random shell = win32com.client.Dispatch('WScript.Shell') # Find the browser window def enum_cb(hwnd, results): if win32gui.IsWindowVisible(hwnd): title = win32gui.GetWindowText(hwnd) if 'Chrome' in title or 'Signup' in title: results.append((hwnd, title[:80])) results = [] win32gui.EnumWindows(enum_cb, results) hwnd = results[0][0] # Focus and fill form via Tab navigation win32gui.SetForegroundWindow(hwnd) time.sleep(0.5) alias = secrets.token_hex(4)[:8] email = f'user+cu_{alias}@gmail.com' pwd = '[REDACTED]' user = f'user_{alias}' # Tab through fields: Full Name -> Email -> Password -> Submit shell.SendKeys('{TAB}') # Focus first field time.sleep(0.4) shell.SendKeys(user) # Type username time.sleep(0.4) shell.SendKeys('{TAB}') # Next field time.sleep(0.4) shell.SendKeys(email) # Type email time.sleep(0.4) shell.SendKeys('{TAB}') # Next field time.sleep(0.4) shell.SendKeys(pwd) # Type password time.sleep(0.5) shell.SendKeys('{TAB}') # Focus submit button time.sleep(0.5) shell.SendKeys('{ENTER}') # Submit time.sleep(8) # Get resulting URL shell.SendKeys('{F6}') # Focus address bar (more reliable than ^l) time.sleep(0.3) shell.SendKeys('^c') # Copy time.sleep(0.3) win32clipboard.OpenClipboard() url = win32clipboard.GetClipboardData() win32clipboard.CloseClipboard() print(f'URL: {url[:120]}') print('SUCCESS' if '/login' in url.lower() else 'FAIL') ``` ## Pitfalls specific to form filling: - **Tab order varies** — some forms have hidden fields; test the Tab sequence manually first - **Clipboard unreliable for URL** — other apps may overwrite between `^c` and read. Use `{F6}` instead of `^l` for address bar focus (more reliable) - **SendKeys sends instantly** — reCAPTCHA v3 may still detect speed. Add `time.sleep(0.05)` between individual characters for human-like typing if needed - **Window title is the fallback indicator** — if clipboard fails, check `win32gui.GetWindowText(hwnd)` for the page title ("Login" means success, "Signup" means failed) ## pyautogui — coordinate-based GUI automation (SendKeys alternative) When SendKeys Tab-navigation is unreliable (multi-step React SPAs where Tab order is unknown or elements appear/destroy dynamically), use pyautogui for direct coordinate-based clicks and typing. **Use case:** Multi-step signup forms (email → Continue → password → captcha → submit) where each step shows/hides elements, making Tab order unpredictable. ### Dependencies ```bash pip install pyautogui pygetwindow pywin32 ``` ### Core pattern: Chrome launch + pyautogui fill + result capture ```python import subprocess, time, json import pyautogui import pygetwindow as gw import ctypes pyautogui.FAILSAFE = False # Disable corner safety check # 1. Launch Chrome _without_ CDP (no --remote-debugging-port) chrome = r'C:\Program Files\Google\Chrome\Application\chrome.exe' user_data = os.path.expanduser('~/lovable_temp') os.makedirs(user_data, exist_ok=True) proc = subprocess.Popen([ chrome, f'--user-data-dir={user_data}', '--start-maximized', '--no-first-run', '--disable-default-apps', '--new-window', 'https://target.com/signup', ], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) time.sleep(8) # Wait for Chrome to start and page to load # 2. Find and activate the Chrome window windows = gw.getWindowsWithTitle('') chrome_wins = [w for w in windows if w.title and 'chrome' in w.title.lower()] if chrome_wins: win = chrome_wins[-1] # SwitchToThisWindow bypasses UIPI restrictions (SetForegroundWindow fails from bg processes) user32 = ctypes.windll.user32 user32.SwitchToThisWindow(win._hWnd, True) time.sleep(1) # Also click title bar to ensure focus pyautogui.click(max(0, win.left + win.width // 2), max(0, win.top + 15)) # 3. Fill form via coordinates (adjust for screen resolution) # For 1920x1080: cx=960, cy=540 | For 2560x1440: cx=1280, cy=720 sw, sh = pyautogui.size() cx, cy = sw // 2, sh // 2 def click(x, y, delay=0.5): pyautogui.moveTo(x, y, duration=0.15) time.sleep(0.1) pyautogui.click(x, y) time.sleep(delay) def type_text(text, interval=0.03): pyautogui.typewrite(text, interval=interval) time.sleep(0.3) # Accept cookie banner (bottom-right) click(sw - 200, sh - 100) # Fill email field (form center-left on most SPAs) click(cx - 240, cy - 260) # Adjust y-offset based on form position type_text('user@email.com') # Click Continue click(cx - 240, cy - 190) time.sleep(4) # Wait for next step to render # Password click(cx - 240, cy - 260) type_text('StrongPass1!') # Click captcha checkbox area (left of form center) click(cx - 290, cy - 140) time.sleep(4) click(cx - 290, cy - 140) # Submit click(cx - 240, cy - 80) time.sleep(2) pyautogui.press('enter') # 4. Capture result time.sleep(30) # Wait for redirect pyautogui.screenshot('result.png') ``` ### Coordinate calibration - **2024×768 / 1920×1080:** form center ≈ (cx-240, cy-260), continue ≈ (cx-240, cy-190), captcha ≈ (cx-290, cy-140), submit ≈ (cx-240, cy-80) - **2560×1440:** same x-offsets, y-offsets shift up by ~100px (form starts higher proportionally) - **Always verify:** run once without `FAILSAFE`, take screenshot `autopilot_result.png`, analyze with PIL to confirm clicks landed on target ### Screenshot analysis (PIL fallback when vision API is unavailable) ```python from PIL import Image import numpy as np img = Image.open('result.png') arr = np.array(img) # Check if form area is dark (signup page dark theme) or light (app/page after redirect) form = arr[400:900, arr.shape[1]//2-300:arr.shape[1]//2+300] bright = (form.mean(axis=2) > 150).sum() total = form.shape[0] * form.shape[1] print(f'Bright pixels: {bright}/{total} ({100*bright/total:.1f}%)') print(f'Avg color: {form.mean():.0f}') # If avg is dark (<100), page is likely still on signup. If bright (>200), redirect happened. ``` ### Pitfalls 1. **`SetForegroundWindow` fails from background processes** — Windows UIPI blocks cross-process window activation. Use `ctypes.windll.user32.SwitchToThisWindow(hwnd, True)` instead. 2. **`pygetwindow.Window.activate()`** wraps `SetForegroundWindow` and will fail with "Error code 0" which is actually SUCCESS but raised as exception. Catch the exception and use ctypes fallback. 3. **Chrome must NOT have CDP** (`--remote-debugging-port`) — Cloudflare Turnstile detects CDP at protocol level and enters `auto_timeout` mode even in real Chrome with `navigator.webdriver=undefined`. No stealth technique bypasses this. 4. **pyautogui.screenshot captures the primary monitor** — if Chrome opened on a secondary monitor, the screenshot will be black/dark. Minimize then restore, or use `pyautogui.size()` to verify screen dimensions. 5. **`FAILSAFE = True` (default)** — moving mouse to screen corner aborts the script. Set `pyautogui.FAILSAFE = False` before unattended automation. 6. **pyautogui.typewrite triggers TypedChar event** — Cloudflare Turnstile may detect superhuman typing speed. Add `interval=0.03-0.05` for human-like speed. 7. **pyautogui might miss after OS sleep/resume** — the screen resolution or DPI may change. Re-calibrate coordinates after system resume. 8. **Multi-monitor** — pyautogui works on the primary monitor only (by default). Use `pyautogui.moveTo` with absolute screen coordinates for secondary monitors when using `pygetwindow` to find window positions. 9. **Window focus loss mid-automation** — terminal windows, VPN overlays (CyberGhost), Google Translate popups, or system notifications can steal focus from the browser. After every `time.sleep()`, verify Chrome is still foreground: check `gw.getActiveWindow().title`. If focus lost, `SwitchToThisWindow` again before continuing. ALWAYS verify with screenshot after typing — if placeholder text still shows, the click or typing didn't register. 10. **pyautogui clicks miss target on centered forms** — the form may be offset from screen center due to header bars, VPN panels, or browser chrome. Start with center-screen clicks, then verify with screenshot. If placeholder text remains, adjust y-offset ±50px and retry. ## Opening Chrome visibly for manual user interaction When the user says "открой", "включи браузер", "я регнусь", "давай я сам", or wants to interact with a page manually — CDP-managed Chrome (via `mcp__chrome_devtools__`) is NOT reliably visible on the user's desktop. The CDP window can be hidden, minimized, off-screen, or on a virtual desktop the user doesn't see. **Correct pattern — launch Chrome via shell for visible interaction:** ```bash taskkill /F /IM chrome.exe 2>nul sleep 2 cmd /c start "" "https://target.com/signup" ``` **What happens:** - `taskkill` removes ALL Chrome processes (including CDP-managed instances) - `cmd /c start "" "URL"` opens Chrome as a normal desktop window the user sees - The new Chrome is NOT attached to CDP — the user gets a clean, undetectable browser for manual typing **After opening for user, also navigate CDP Chrome to same URL via `mcp__chrome_devtools__navigate_page(url="...")`** — this makes the page observable via agent tools while the user interacts. **Verification:** After launching, the user should see a normal Chrome window on their desktop with the target page. If they say "не открыл", the Chrome instance with CDP is not the right one — re-launch via `start ""`. **Pitfalls:** - `start "" "https://..."` must be in a real cmd/terminal — does NOT work in every context - `explorer "https://..."` does NOT open URLs reliably on Windows 10/11 - CDP Chrome (`remote-debugging-port` mode) and visible Chrome are SEPARATE processes — the user sees the non-CDP one - After killing all Chrome, CDP will auto-restart and connect to a new instance — this is fine ## Cloudflare Turnstile — automation barriers (verified 2026-08-04) Turnstile in `auto` mode is a hard block for ALL automation approaches: | Approach | Result | |----------|--------| | Playwright headless | iframe empty (`auto_timeout`) | | Playwright headed + stealth | iframe empty | | Playwright + real Chrome + init script (`webdriver=undefined`) | iframe rendered but `auto_timeout` / `failure_retry` | | Chrome DevTools MCP (CDP to real user browser) | checkbox visible but click → `failure_retry` | | `window.turnstile.execute()` programmatic | returns success, token stays empty | | 2captcha | `ERROR_CAPTCHA_UNSOLVABLE` (workers can't access the page) | | rebrowser_playwright | iframe empty | | pyautogui (physical click, no automation protocol) | NOT tested — only remaining path | **Why CDP is detected:** Cloudflare detects the DevTools Protocol connection itself, not just `navigator.webdriver`. Even connecting to a real user's Chrome via `--remote-debugging-port` triggers this. No amount of stealth patches hides CDP. **UPDATE 2026-09-10 (verified):** the table above does NOT hold for every Turnstile deployment. On dashboard.rainproxy.io and dashboard.proxyscrape.com, Turnstile in `auto` mode **self-solved** inside a real (headed) Chrome launched with `--remote-debugging-port=9333` + a FRESH `--user-data-dir`, driven via Playwright `connect_over_cdp`. The hidden field `[name=cf-turnstile-response]` filled itself (~773 chars) within seconds; poll `document.querySelector('[name=cf-turnstile-response]')?.value?.length > 10` before clicking submit. Headless browsers were still blocked (SPA bundle ERR_CONNECTION_RESET + empty iframe), and firecrawl stealth also worked for form-fill. So: try headed-Chrome-CDP FIRST for Turnstile sites — manual pyautogui is the fallback, not the default. **Overlay-intercepted clicks (Intercom/Trustpilot/etc.):** when Playwright `locator.click()` times out with "