autoreg-skills-max/skills/bulk-account-registration/references/form-automation-advanced.md

# Advanced Form Automation Techniques

> Techniques discovered during TJC (Dynamic Forms / NGWebSolutions) account registration automation. General patterns applicable to any web form with stubborn client-side validation or invisible reCAPTCHA.

## 1. Password Field Bypass via `setNativeValue` + `dispatchEvent`

### The Problem

Some forms use custom JavaScript validation that checks for specific input events before accepting a password field. Standard Playwright methods fail:

| Method | Result |
|--------|--------|
| `page.fill(selector, password)` | Field visually filled, but validation rejects ("The Password field is required") |
| `page.type(selector, password)` | Same result — validation rejects |
| `page.pressSequentially(selector, password)` | Same result |
| `page.locator(selector).fill(password)` | Same result |

The root cause: the form's JavaScript validation checks for `input` events dispatched by the native `EventTarget.prototype.dispatchEvent` API, but Playwright's `fill()` and `type()` dispatch synthetic events that the validation doesn't recognize.

### The Fix

Use `page.evaluate()` to set the value natively and dispatch the event manually:

```javascript
// In page.evaluate():
const input = document.querySelector('input#Password');
const nativeInputValueSetter = Object.getOwnPropertyDescriptor(
  window.HTMLInputElement.prototype, 'value'
).set;
nativeInputValueSetter.call(input, 'YourPasswordHere123@');
input.dispatchEvent(new Event('input', { bubbles: true }));
```

### Why it works

- `Object.getOwnPropertyDescriptor(..., 'value').set` — accesses the native property setter defined by the browser, bypassing any framework-overridden setters (React, Vue, Angular, etc.)
- `.call(input, value)` — invokes the setter directly on the element, setting the internal value without triggering the framework's synthetic event system
- `dispatchEvent(new Event('input', { bubbles: true }))` — fires a browser-native input event that the form's validation listener recognizes

### Pattern (Python + Playwright)

```python
async def set_value_native(page, selector, value):
    """Set form field value bypassing JS framework validation"""
    await page.evaluate(f"""
        (() => {{
            const input = document.querySelector('{selector}');
            const setter = Object.getOwnPropertyDescriptor(
                window.HTMLInputElement.prototype, 'value'
            ).set;
            setter.call(input, '{value}');
            input.dispatchEvent(new Event('input', {{ bubbles: true }}));
        }})()
    """)
```

**⚠️ Pitfall: escaping special characters.** If the password contains single quotes, backslashes, or other JS-significant characters, use `JSON.stringify(value)` or pass the value as a function argument:

```python
await page.evaluate("""
    ([selector, value]) => {
        const input = document.querySelector(selector);
        const setter = Object.getOwnPropertyDescriptor(
            window.HTMLInputElement.prototype, 'value'
        ).set;
        setter.call(input, value);
        input.dispatchEvent(new Event('input', { bubbles: true }));
    }
""", [selector, password])
```

### When to use this

- Playwright `fill()` / `type()` / `pressSequentially()` all fail
- The form shows "This field is required" even though the field visually has content
- The form uses custom JavaScript validation (common in ASP.NET MVC, Angular, React forms)
- Specifically for password fields — text fields rarely have this issue

### Verification

After calling `setValueNative`, verify the field actually has the value:

```javascript
document.querySelector('input#Password').value.length  // Should be > 0
```

## 2. Invisible reCAPTCHA with `g-recaptcha` Class

### Identifying invisible reCAPTCHA

The form button has these attributes:

```html
<button class="btn btn-default g-recaptcha"
        data-sitekey="6Lc1DN8ZAAAAAJ24AatUEzMn95wjUkjmHt65BfQi"
        data-callback="onSubmit"
        data-action="CreateAccount">
  Create Account
</button>
```

Key signals:
- **`class="g-recaptcha"`** — Google's CSS class for automatic binding
- **`data-callback="onSubmit"`** — function called when reCAPTCHA challenge succeeds
- **`data-sitekey="..."`** — the reCAPTCHA site key
- **No `type="submit"`** — the button is NOT a submit button; reCAPTCHA handles submission

### How it works

1. Google's reCAPTCHA JS scans the page for elements with `class="g-recaptcha"`
2. When the button is clicked, reCAPTCHA intercepts the click
3. Runs the invisible challenge (no user interaction required)
4. On success, calls `onSubmit(token)` — which sets the token in a hidden field and submits the form

### The `onSubmit` callback pattern

Typically looks like:

```javascript
function onSubmit(token) {
    $("#RecaptchaToken").val(token);
    document.getElementById("createAccountForm").submit();
}
```

### Hidden field for token

The form has a hidden input that stores the reCAPTCHA token:

[REDACTED]
<input id="RecaptchaToken" name="RecaptchaToken" type="hidden" value="" />
```

This is DIFFERENT from the standard `g-recaptcha-response` hidden field. Some custom implementations use their own field name.

### Approach 1: Solve via 2captcha + manual injection

```python
import requests, time

CAPTCHA_KEY = "your_2captcha_key"
SITEKEY = "6Lc1DN8ZAAAAAJ24AatUEzMn95wjUkjmHt65BfQi"
PAGEURL = "https://target.com/register"

# 1. Submit to 2captcha
resp = requests.get("https://2captcha.com/in.php", params={
    "key": CAPTCHA_KEY, "method": "userrecaptcha",
    "googlekey": SITEKEY, "pageurl": PAGEURL, "json": 1
}, timeout=30)
captcha_id = resp.json()["request"]

# 2. Poll for result
for i in range(20):
    time.sleep(5)
    resp = requests.get("https://2captcha.com/res.php", params={
        "key": CAPTCHA_KEY, "action": "get", "id": captcha_id, "json": 1
    }, timeout=15)
    data = resp.json()
    if data.get("status") == 1:
        token = data["request"]
        break

# 3. Inject into browser (via page.evaluate or browser_console)
await page.evaluate("""
    (token) => {
        document.getElementById('RecaptchaToken').value = token;
    }
""", token)

# 4. Trigger onSubmit
await page.evaluate("""
    onSubmit(document.getElementById('RecaptchaToken').value);
""")
```

### Approach 2: Browser-native `grecaptcha.execute()`

If the reCAPTCHA script is loaded and the widget is rendered:

```javascript
// In browser_console or page.evaluate:
grecaptcha.execute();
```

This returns a Promise that resolves to a fresh token. However, calling `execute()` alone does NOT trigger the `onSubmit` callback — it just generates a new token. You still need to either:
- click the `g-recaptcha` button (which internally calls execute + callback)
- manually call `onSubmit(token)` after getting the token

### Approach 3: Click the button (preferred)

If the password field is filled correctly (via `setNativeValue`), simply clicking the `g-recaptcha` button triggers the full flow:

```javascript
document.querySelector('.g-recaptcha').click();
```

The reCAPTCHA library intercepts the click, runs the challenge, and calls `onSubmit` on success.

### Known issues

- **Migration warning**: Some reCAPTCHA site keys show "Google reCAPTCHA migration" warnings in the console. This happens when the site key was created under an older Google account/API project. The key may still work for 2captcha solving but the server may reject the token.
- **Token expiry**: reCAPTCHA tokens expire after ~2 minutes. If you solve via 2captcha and the form submission takes too long, get a fresh token.
- **Server-side rejection**: Even with a valid token, the server may silently reject the form submission (returning the same page with 200, no error message). This usually means the server-side validation failed on something other than the captcha (e.g., duplicate username, email format, anti-forgery token mismatch).

## 3. ASP.NET Anti-Forgery Token Handling

ASP.NET MVC forms use `__RequestVerificationToken` as a hidden field AND a cookie (`.__AspNetCore.Antiforgery.*`). Both must match.

### In the browser

The browser automatically handles this — the cookie is set on page load, and the hidden field value is embedded in the form. `form.submit()` sends both correctly.

### Via curl/requests

1. GET the form page to receive the cookie
2. Extract the `__RequestVerificationToken` from the hidden field in the HTML
3. POST with both the cookie and the token value

```bash
# Step 1: GET form (stores cookie)
curl -c cookies.txt -b cookies.txt -L 'https://target.com/Account/CreateAccount'

# Step 2: Extract token from HTML
TOKEN=$(grep -oP '__RequestVerificationToken" value="\K[^"]+' cookies.txt || \
        grep -oP '__RequestVerificationToken" value="\K[^"]+' response.html)

# Step 3: POST with form data
curl -c cookies.txt -b cookies.txt \
  -X POST 'https://target.com/Account/CreateAccount' \
  -d "__RequestVerificationToken=$TOKEN" \
  -d "Username=testuser" \
  -d "Password=[REDACTED]"
```

**Without the correct token**: server returns HTTP 400 Bad Request.

## 4. Chromium SOCKS5 Proxy (unauthenticated)

Playwright/Chromium does NOT support SOCKS5 with authentication. Workaround: set up an SSH tunnel without auth:

```bash
# Create tunnel (no auth needed)
ssh -D 1081 -N root@your-server.com

# In Playwright
browser = await playwright.chromium.launch(
    args=["--proxy-server=socks5://localhost:1081"]
)
```

The tunnel must be running before launching the browser. The `socks5h://` prefix (DNS resolution through proxy) is not needed when using `--proxy-server` — Playwright handles DNS through the proxy automatically.

## 5. Diagnostic Flow for Form Submission Failures

When a form submits but returns the same page (no error, no redirect):

1. **Check password validation** — inspect `document.getElementById('Password').value.length` → if 0, the password was cleared by the page reload. Use `setNativeValue` technique.
2. **Check reCAPTCHA token** — inspect `document.getElementById('RecaptchaToken').value` → if empty or stale, re-solve.
3. **Check anti-forgery token** — inspect `document.querySelector('input[name=__RequestVerificationToken]').value` → if empty, the page state reset.
4. **Check for validation errors** — `document.querySelector('.validation-summary-errors, .text-danger')?` → if present, read the error text.
5. **Check browser console** — `console.clear()` then `console.log('ready')` before action, check for JS errors after.
6. **Check server response directly** — intercept via curl with the same cookies and form data to see the raw server response.
7. **Try a fresh page load** — if the form state is corrupted, navigate to the URL fresh and re-fill everything.