vlad/autoreg-skills-max
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
209e142d443042a2d86342c948d06a76c09eb696
autoreg-skills-max/skills/windows-browser-automation/SKILL.md

24 KiB
Raw Blame History

name, description, triggers
name description triggers
windows-browser-automation Control Comet/Edge/Chrome on Windows via CDP (Playwright/browser-use) or Win32 SendKeys fallback. Covers Comet cold-start CDP, real-profile attach, address-bar JS injection, window focus, SendKeys escaping, and cookie extraction.
Comet browser
SendKeys
Win32 browser control
no CDP port
browser cookie extraction
автоматизация браузера
comet.exe
Comet CDP
comet agent
comet-browser-control
remote-debugging-port
remote-allow-origins
browser-use real profile
Playwright CDP Comet
открой браузер
включи браузер
я регнусь
ручная регистрация
start chrome
visible browser
manual browser

Windows Browser Automation via Win32 API

When to use

  • Comet browser (Perplexity build) — CDP is available BUT only at cold start. An already-running Comet ignores --remote-debugging-port. To get CDP on the real profile: kill all comet.exe processes, then launch with comet.exe --remote-debugging-port=9233 --remote-allow-origins=* --user-data-dir=C:\Users\User\AppData\Local\Perplexity\Comet\User Data. The --remote-allow-origins=* flag is REQUIRED for WebSocket CDP connections (Playwright/Puppeteer/browser-use) — without it you get HTTP 403 on WebSocket handshake.
  • Edge/Chrome CDP port crashed or unreachable
  • MCP Playwright server is dead but the browser IS running with the target page open
  • Need to inject JavaScript, copy page content, or navigate in a running browser
  • Comet Agent project lives at C:\Users\User\Desktop\_PROJECTS\comet-browser-control\ — use launcher_start.py for one-click cold-start Comet + CDP + hidden agent server on :8765

Core pattern

import win32com.client, win32gui, win32clipboard, time

shell = win32com.client.Dispatch("WScript.Shell")

# 1. Find the window HWND
# Use: python -c "import win32gui; ..." or tasklist + window title search
hwnd = 6425040  # example: "Dashboard | TECH HAUS Workshop – Comet"

# 2. Activate it
win32gui.SetForegroundWindow(hwnd)
time.sleep(0.3)  # Always wait between actions

# 3. Send keystrokes
shell.SendKeys("^l")        # Ctrl+L = address bar
shell.SendKeys("{ESC}")     # Close popups
shell.SendKeys("^a")        # Select all
shell.SendKeys("^c")        # Copy

# 4. Read clipboard
win32clipboard.OpenClipboard()
data = win32clipboard.GetClipboardData()
win32clipboard.CloseClipboard()

SendKeys escaping rules

  • Parentheses ( ) MUST be escaped: {(} and {)} — they're special keys
  • Curly braces { } wrap key names like {ENTER}, {TAB}, {F12}
  • Special chars + ^ % ~ are modifiers (Shift/Ctrl/Alt) — escape with braces: {+}
  • Period . is safe in most contexts

JavaScript injection via address bar

shell.SendKeys("%d")  # Alt+D = focus address bar (more reliable than Ctrl+L)
shell.SendKeys("javascript:void{(}{(}document.body.innerText=document.cookie{)}{)}")
shell.SendKeys("{ENTER}")
time.sleep(1.5)
# Then Ctrl+A, Ctrl+C to copy the result

Pitfall: document.cookie only returns non-httpOnly cookies. Supabase auth cookies are httpOnly=false so this works. th_chunk_gate may be httpOnly=true and invisible to JS.

Comet-specific facts

  • Path: C:\Program Files\Perplexity\Comet\Application\comet.exe
  • 48+ processes normally — this is architectural, not a leak
  • Real user profile: C:\Users\User\AppData\Local\Perplexity\Comet\User Data\ (profile name "Nikita", Default directory)
  • CDP IS available on Comet but only at cold start. Chromium ignores --remote-debugging-port if an instance is already running (it just forwards the URL and exits). To get CDP on the real profile:
    1. Kill all comet.exe processes
    2. Launch: comet.exe --remote-debugging-port=9233 --remote-allow-origins=* --user-data-dir=C:\Users\User\AppData\Local\Perplexity\Comet\User Data --no-first-run --no-default-browser-check --disable-features=Translate
    3. --remote-allow-origins=* is REQUIRED for WebSocket CDP connections — without it, Playwright/Puppeteer/browser-use get HTTP 403 on WebSocket handshake
    4. Verified: /json/version returns Chrome/149.0.7827.1095, /json/list shows all real tabs
  • Comet Agent project at C:\Users\User\Desktop\_PROJECTS\comet-browser-control\ — launcher_start.py handles the cold-start kill+restart flow automatically. Also includes comet_gui_control.py (UIA fallback), cdp_scan.py, playwright_attach.py, browser_use_cdp_check.py, clone_to_chrome_cdp.py.
  • Cookies DB: User Data\Default\Network\Cookies — SQLite, locked while browser runs
  • While running: only part of cookies may be visible on disk; auth cookies can be in memory
  • To extract cookies: (a) F12 → Application → Cookies manually, or (b) close Comet → DB unlocks → read with sqlite3

CDP Auth Cookie Injection Pattern (2026-07-16)

When you have auth cookies from API sign-in but can't set them in the browser UI — use Playwright connect_over_cdp + context.add_cookies. This bridges API-authenticated sessions into a real browser:

import asyncio, json
from playwright.async_api import async_playwright

async def inject_and_navigate(api_cookies: dict, target_url: str):
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp("http://127.0.0.1:9222")
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else await context.new_page()
        
        await context.add_cookies([
            {"name": k, "value": v, "domain": ".target.com",
             "path": "/", "httpOnly": False, "secure": True, "sameSite": "Lax"}
            for k, v in api_cookies.items()
        ])
        
        await page.goto(target_url, wait_until="load")
        # Now authenticated — buttons like "Claim Offer" are visible

Starting Chrome with CDP for cookie injection:

import subprocess, time, requests

subprocess.run("taskkill /F /IM chrome.exe 2>nul", shell=True)
subprocess.Popen([
    r"C:\Program Files\Google\Chrome\Application\chrome.exe",
    "--remote-debugging-port=9222",
    "--remote-allow-origins=*",  # REQUIRED for WebSocket CDP
    r"--user-data-dir=C:\Users\User\AppData\Local\hermes\cache\chrome-profile",
    "--no-first-run", "--no-default-browser-check",
    "about:blank"
], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
time.sleep(3)

# Verify
r = requests.get("http://127.0.0.1:9222/json/version")
print(f"Chrome: {r.json().get('Browser')}")

Pitfalls:

  • --remote-allow-origins=* is REQUIRED — without it Playwright WebSocket handshake returns 403
  • Use wait_until="load" (not "networkidle") for Cloudflare-protected sites — networkidle times out on CF-challenged pages
  • Clerk __client cookie carries the session; losing it = "Signed out"
  • Fresh --user-data-dir avoids profile lock conflicts with existing Chrome instances
  • taskkill kills all Chrome processes — warn user if they have important tabs open

Find all Comet/Chromium windows

import win32gui

def enum_callback(hwnd, results):
    if win32gui.IsWindowVisible(hwnd):
        title = win32gui.GetWindowText(hwnd)
        if title:
            results.append((hwnd, title, win32gui.GetWindowRect(hwnd)))

results = []
win32gui.EnumWindows(enum_callback, results)
for hwnd, title, rect in results:
    if 'Comet' in title or 'premium' in title.lower():
        print(f"HWND={hwnd} | \"{title[:80]}\" | rect={rect}")

Pitfalls

  1. Popup interception: If Ctrl+A copies dropdown content instead of page, close popups with {ESC} first
  2. SendKeys timing: Always time.sleep(0.3) between actions — too fast and keystrokes merge
  3. Window focus: SetForegroundWindow can fail silently if another app steals focus — verify with GetForegroundWindow()
  4. javascript: URLs: Some browsers block these from address bar for security — fall back to DevTools Console
  5. Clipboard races: Other apps may overwrite clipboard between SendKeys and read — use win32clipboard immediately
  6. Playwright CDP + real Comet profile: do NOT reuse ctx.pages[0] for checks; real profiles often have YouTube/X/Stripe/file tabs and title/navigation can hang. For --goto, open a disposable ctx.new_page(), wrap title/goto/close in timeouts, then close the disposable page.
  7. browser-use 0.13.x + real profile: its SecurityWatchdog treats existing file:// tabs as disallowed and may close them on attach. Before BrowserSession.start() monkeypatch SecurityWatchdog._is_url_allowed to allow file://, chrome://, devtools:// (or use a dedicated clean Comet Agent profile).
  8. Comet CDP WebSocket 403: if Playwright/Puppeteer/browser-use fail with WebSocketBadStatusException: Handshake status 403, Comet was launched without --remote-allow-origins=*. Add it to the launch args and restart Comet. Also pass suppress_origin=True to websocket.create_connection() when talking CDP directly (plain websocket-client) — Chrome 152+ rejects connections with an http://127.0.0.1:PORT Origin header.
  9. OAuth consent opens in a NEW TAB: Google OAuth flows (and most SSO) spawn a separate tab for the consent screen (accounts.google.com/signin/oauth/id). Your existing websocket stays bound to the old tab — document.querySelector finds nothing. Fix: re-fetch http://127.0.0.1:9222/json, pick the tab by URL substring, open a fresh websocket to its webSocketDebuggerUrl. Managed (Workspace/Cloud Identity) accounts additionally spawn chrome://managed-user-profile-notice/ — filter tabs by type=="page" and target URL, never tabs[0].
  10. React controlled inputs: native-setter + dispatchEvent('input') works for text fields, but form submits sometimes silently no-op or lose focus. More reliable: element.focus() via Runtime.evaluate, then CDP Input.insertText (real keystroke path), then click submit. Verify input.value after typing before clicking.

Pre-requisites

pip install pywin32

win32com, win32gui, win32clipboard are all in pywin32.

Reference files

  • references/comet-browser-control-project.md — full project structure, all scripts, config, verified dependencies
  • references/browser-use-security-watchdog-patch.md — monkeypatch to prevent browser-use from closing file:// tabs on real Comet profiles
  • references/comet-cookie-extraction.md — cookie extraction methods
  • references/unjail-premium-accessor.md — premium.unjail.ai access patterns
  • references/groq-auth-flow.md — Groq console auth flow: working URL, magic-link pattern, pyautogui field positions
  • references/cloudflare-turnstile-limitations.md — Cloudflare Turnstile automation barriers
  • references/spa-auth-form-debugging.md — headless SPA auth/OTP form failures: API-response gating vs "Loading..." buttons, false hasAccount pre-checks, screenshot font-hangs, noscript-challenge detection, stalled post-OTP redirect fix, diagnostic script skeleton
  • references/spa-trial-autoreg-verified-flow.md — VERIFIED end-to-end autoreg pattern (2026-09-10, ProxyScrape): headed Chrome CDP + Turnstile auto-solve + JS-click submit + IMAP hex-code extraction pitfalls + placeholder-credential trap

Form filling via Tab navigation (captcha bypass)

When CDP/browser automation is blocked by reCAPTCHA Enterprise v3, SendKeys is the LAST RESORT that works because it simulates real human keystrokes on the desktop. The browser sees real keyboard events, not injected JavaScript.

import win32gui, win32com.client, win32clipboard, time, secrets, random

shell = win32com.client.Dispatch('WScript.Shell')

# Find the browser window
def enum_cb(hwnd, results):
    if win32gui.IsWindowVisible(hwnd):
        title = win32gui.GetWindowText(hwnd)
        if 'Chrome' in title or 'Signup' in title:
            results.append((hwnd, title[:80]))
results = []
win32gui.EnumWindows(enum_cb, results)
hwnd = results[0][0]

# Focus and fill form via Tab navigation
win32gui.SetForegroundWindow(hwnd)
time.sleep(0.5)

alias = secrets.token_hex(4)[:8]
email = f'user+cu_{alias}@gmail.com'
pwd = '[REDACTED]'
user = f'user_{alias}'

# Tab through fields: Full Name -> Email -> Password -> Submit
shell.SendKeys('{TAB}')       # Focus first field
time.sleep(0.4)
shell.SendKeys(user)           # Type username
time.sleep(0.4)
shell.SendKeys('{TAB}')        # Next field
time.sleep(0.4)
shell.SendKeys(email)          # Type email
time.sleep(0.4)
shell.SendKeys('{TAB}')        # Next field
time.sleep(0.4)
shell.SendKeys(pwd)            # Type password
time.sleep(0.5)
shell.SendKeys('{TAB}')        # Focus submit button
time.sleep(0.5)
shell.SendKeys('{ENTER}')      # Submit
time.sleep(8)

# Get resulting URL
shell.SendKeys('{F6}')         # Focus address bar (more reliable than ^l)
time.sleep(0.3)
shell.SendKeys('^c')           # Copy
time.sleep(0.3)
win32clipboard.OpenClipboard()
url = win32clipboard.GetClipboardData()
win32clipboard.CloseClipboard()
print(f'URL: {url[:120]}')
print('SUCCESS' if '/login' in url.lower() else 'FAIL')

Pitfalls specific to form filling:

  • Tab order varies — some forms have hidden fields; test the Tab sequence manually first
  • Clipboard unreliable for URL — other apps may overwrite between ^c and read. Use {F6} instead of ^l for address bar focus (more reliable)
  • SendKeys sends instantly — reCAPTCHA v3 may still detect speed. Add time.sleep(0.05) between individual characters for human-like typing if needed
  • Window title is the fallback indicator — if clipboard fails, check win32gui.GetWindowText(hwnd) for the page title ("Login" means success, "Signup" means failed)

pyautogui — coordinate-based GUI automation (SendKeys alternative)

When SendKeys Tab-navigation is unreliable (multi-step React SPAs where Tab order is unknown or elements appear/destroy dynamically), use pyautogui for direct coordinate-based clicks and typing.

Use case: Multi-step signup forms (email → Continue → password → captcha → submit) where each step shows/hides elements, making Tab order unpredictable.

Dependencies

pip install pyautogui pygetwindow pywin32

Core pattern: Chrome launch + pyautogui fill + result capture

import subprocess, time, json
import pyautogui
import pygetwindow as gw
import ctypes

pyautogui.FAILSAFE = False  # Disable corner safety check

# 1. Launch Chrome _without_ CDP (no --remote-debugging-port)
chrome = r'C:\Program Files\Google\Chrome\Application\chrome.exe'
user_data = os.path.expanduser('~/lovable_temp')
os.makedirs(user_data, exist_ok=True)

proc = subprocess.Popen([
    chrome, f'--user-data-dir={user_data}',
    '--start-maximized', '--no-first-run',
    '--disable-default-apps', '--new-window',
    'https://target.com/signup',
], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

time.sleep(8)  # Wait for Chrome to start and page to load

# 2. Find and activate the Chrome window
windows = gw.getWindowsWithTitle('')
chrome_wins = [w for w in windows if w.title and 'chrome' in w.title.lower()]
if chrome_wins:
    win = chrome_wins[-1]
    # SwitchToThisWindow bypasses UIPI restrictions (SetForegroundWindow fails from bg processes)
    user32 = ctypes.windll.user32
    user32.SwitchToThisWindow(win._hWnd, True)
    time.sleep(1)
    # Also click title bar to ensure focus
    pyautogui.click(max(0, win.left + win.width // 2), max(0, win.top + 15))

# 3. Fill form via coordinates (adjust for screen resolution)
# For 1920x1080: cx=960, cy=540 | For 2560x1440: cx=1280, cy=720
sw, sh = pyautogui.size()
cx, cy = sw // 2, sh // 2

def click(x, y, delay=0.5):
    pyautogui.moveTo(x, y, duration=0.15)
    time.sleep(0.1)
    pyautogui.click(x, y)
    time.sleep(delay)

def type_text(text, interval=0.03):
    pyautogui.typewrite(text, interval=interval)
    time.sleep(0.3)

# Accept cookie banner (bottom-right)
click(sw - 200, sh - 100)

# Fill email field (form center-left on most SPAs)
click(cx - 240, cy - 260)  # Adjust y-offset based on form position
type_text('user@email.com')

# Click Continue
click(cx - 240, cy - 190)
time.sleep(4)  # Wait for next step to render

# Password
click(cx - 240, cy - 260)
type_text('StrongPass1!')

# Click captcha checkbox area (left of form center)
click(cx - 290, cy - 140)
time.sleep(4)
click(cx - 290, cy - 140)

# Submit
click(cx - 240, cy - 80)
time.sleep(2)
pyautogui.press('enter')

# 4. Capture result
time.sleep(30)  # Wait for redirect
pyautogui.screenshot('result.png')

Coordinate calibration

  • 2024×768 / 1920×1080: form center ≈ (cx-240, cy-260), continue ≈ (cx-240, cy-190), captcha ≈ (cx-290, cy-140), submit ≈ (cx-240, cy-80)
  • 2560×1440: same x-offsets, y-offsets shift up by ~100px (form starts higher proportionally)
  • Always verify: run once without FAILSAFE, take screenshot autopilot_result.png, analyze with PIL to confirm clicks landed on target

Screenshot analysis (PIL fallback when vision API is unavailable)

from PIL import Image
import numpy as np

img = Image.open('result.png')
arr = np.array(img)

# Check if form area is dark (signup page dark theme) or light (app/page after redirect)
form = arr[400:900, arr.shape[1]//2-300:arr.shape[1]//2+300]
bright = (form.mean(axis=2) > 150).sum()
total = form.shape[0] * form.shape[1]
print(f'Bright pixels: {bright}/{total} ({100*bright/total:.1f}%)')
print(f'Avg color: {form.mean():.0f}')
# If avg is dark (<100), page is likely still on signup. If bright (>200), redirect happened.

Pitfalls

  1. SetForegroundWindow fails from background processes — Windows UIPI blocks cross-process window activation. Use ctypes.windll.user32.SwitchToThisWindow(hwnd, True) instead.
  2. pygetwindow.Window.activate() wraps SetForegroundWindow and will fail with "Error code 0" which is actually SUCCESS but raised as exception. Catch the exception and use ctypes fallback.
  3. Chrome must NOT have CDP (--remote-debugging-port) — Cloudflare Turnstile detects CDP at protocol level and enters auto_timeout mode even in real Chrome with navigator.webdriver=undefined. No stealth technique bypasses this.
  4. pyautogui.screenshot captures the primary monitor — if Chrome opened on a secondary monitor, the screenshot will be black/dark. Minimize then restore, or use pyautogui.size() to verify screen dimensions.
  5. FAILSAFE = True (default) — moving mouse to screen corner aborts the script. Set pyautogui.FAILSAFE = False before unattended automation.
  6. pyautogui.typewrite triggers TypedChar event — Cloudflare Turnstile may detect superhuman typing speed. Add interval=0.03-0.05 for human-like speed.
  7. pyautogui might miss after OS sleep/resume — the screen resolution or DPI may change. Re-calibrate coordinates after system resume.
  8. Multi-monitor — pyautogui works on the primary monitor only (by default). Use pyautogui.moveTo with absolute screen coordinates for secondary monitors when using pygetwindow to find window positions.
  9. Window focus loss mid-automation — terminal windows, VPN overlays (CyberGhost), Google Translate popups, or system notifications can steal focus from the browser. After every time.sleep(), verify Chrome is still foreground: check gw.getActiveWindow().title. If focus lost, SwitchToThisWindow again before continuing. ALWAYS verify with screenshot after typing — if placeholder text still shows, the click or typing didn't register.
  10. pyautogui clicks miss target on centered forms — the form may be offset from screen center due to header bars, VPN panels, or browser chrome. Start with center-screen clicks, then verify with screenshot. If placeholder text remains, adjust y-offset ±50px and retry.

Opening Chrome visibly for manual user interaction

When the user says "открой", "включи браузер", "я регнусь", "давай я сам", or wants to interact with a page manually — CDP-managed Chrome (via mcp__chrome_devtools__) is NOT reliably visible on the user's desktop. The CDP window can be hidden, minimized, off-screen, or on a virtual desktop the user doesn't see.

Correct pattern — launch Chrome via shell for visible interaction:

taskkill /F /IM chrome.exe 2>nul
sleep 2
cmd /c start "" "https://target.com/signup"

What happens:

  • taskkill removes ALL Chrome processes (including CDP-managed instances)
  • cmd /c start "" "URL" opens Chrome as a normal desktop window the user sees
  • The new Chrome is NOT attached to CDP — the user gets a clean, undetectable browser for manual typing

After opening for user, also navigate CDP Chrome to same URL via mcp__chrome_devtools__navigate_page(url="...") — this makes the page observable via agent tools while the user interacts.

Verification: After launching, the user should see a normal Chrome window on their desktop with the target page. If they say "не открыл", the Chrome instance with CDP is not the right one — re-launch via start "".

Pitfalls:

  • start "" "https://..." must be in a real cmd/terminal — does NOT work in every context
  • explorer "https://..." does NOT open URLs reliably on Windows 10/11
  • CDP Chrome (remote-debugging-port mode) and visible Chrome are SEPARATE processes — the user sees the non-CDP one
  • After killing all Chrome, CDP will auto-restart and connect to a new instance — this is fine

Cloudflare Turnstile — automation barriers (verified 2026-08-04)

Turnstile in auto mode is a hard block for ALL automation approaches:

Approach Result
Playwright headless iframe empty (auto_timeout)
Playwright headed + stealth iframe empty
Playwright + real Chrome + init script (webdriver=undefined) iframe rendered but auto_timeout / failure_retry
Chrome DevTools MCP (CDP to real user browser) checkbox visible but click → failure_retry
window.turnstile.execute() programmatic returns success, token stays empty
2captcha ERROR_CAPTCHA_UNSOLVABLE (workers can't access the page)
rebrowser_playwright iframe empty
pyautogui (physical click, no automation protocol) NOT tested — only remaining path

Why CDP is detected: Cloudflare detects the DevTools Protocol connection itself, not just navigator.webdriver. Even connecting to a real user's Chrome via --remote-debugging-port triggers this. No amount of stealth patches hides CDP.

UPDATE 2026-09-10 (verified): the table above does NOT hold for every Turnstile deployment. On dashboard.rainproxy.io and dashboard.proxyscrape.com, Turnstile in auto mode self-solved inside a real (headed) Chrome launched with --remote-debugging-port=9333 + a FRESH --user-data-dir, driven via Playwright connect_over_cdp. The hidden field [name=cf-turnstile-response] filled itself (~773 chars) within seconds; poll document.querySelector('[name=cf-turnstile-response]')?.value?.length > 10 before clicking submit. Headless browsers were still blocked (SPA bundle ERR_CONNECTION_RESET + empty iframe), and firecrawl stealth also worked for form-fill. So: try headed-Chrome-CDP FIRST for Turnstile sites — manual pyautogui is the fallback, not the default.

Overlay-intercepted clicks (Intercom/Trustpilot/etc.): when Playwright locator.click() times out with "<iframe ...> subtree intercepts pointer events" (chat widgets parked over the submit button), do NOT fight coordinates — click via page.evaluate("document.querySelector('button[type=submit]').click()"). Verified working on proxyscrape (Intercom) and webshare forms. Caveat: JS .click() bypasses the overlay but does NOT satisfy reCAPTCHA-gated submits (webshare silently no-ops); it DOES work when the only gate is Turnstile auto-mode.

What DOES work: pyautogui (physical mouse/keyboard via Windows SendInput API) — Cloudflare sees human input events indistinguishable from a real user. But requires the browser to be launched without CDP (--remote-debugging-port must be absent).

Reference: references/cloudflare-turnstile-limitations.md