autoreg-skills-max: initial public release (restored, redacted, hardened)
This commit is contained in:
baradok60915d52e8425
@@ -0,0 +1,263 @@
|
||||
# Advanced Form Automation Techniques
|
||||
|
||||
> Techniques discovered during TJC (Dynamic Forms / NGWebSolutions) account registration automation. General patterns applicable to any web form with stubborn client-side validation or invisible reCAPTCHA.
|
||||
|
||||
## 1. Password Field Bypass via `setNativeValue` + `dispatchEvent`
|
||||
|
||||
### The Problem
|
||||
|
||||
Some forms use custom JavaScript validation that checks for specific input events before accepting a password field. Standard Playwright methods fail:
|
||||
|
||||
| Method | Result |
|
||||
|--------|--------|
|
||||
| `page.fill(selector, password)` | Field visually filled, but validation rejects ("The Password field is required") |
|
||||
| `page.type(selector, password)` | Same result — validation rejects |
|
||||
| `page.pressSequentially(selector, password)` | Same result |
|
||||
| `page.locator(selector).fill(password)` | Same result |
|
||||
|
||||
The root cause: the form's JavaScript validation checks for `input` events dispatched by the native `EventTarget.prototype.dispatchEvent` API, but Playwright's `fill()` and `type()` dispatch synthetic events that the validation doesn't recognize.
|
||||
|
||||
### The Fix
|
||||
|
||||
Use `page.evaluate()` to set the value natively and dispatch the event manually:
|
||||
|
||||
```javascript
|
||||
// In page.evaluate():
|
||||
const input = document.querySelector('input#Password');
|
||||
const nativeInputValueSetter = Object.getOwnPropertyDescriptor(
|
||||
window.HTMLInputElement.prototype, 'value'
|
||||
).set;
|
||||
nativeInputValueSetter.call(input, 'YourPasswordHere123@');
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
```
|
||||
|
||||
### Why it works
|
||||
|
||||
- `Object.getOwnPropertyDescriptor(..., 'value').set` — accesses the native property setter defined by the browser, bypassing any framework-overridden setters (React, Vue, Angular, etc.)
|
||||
- `.call(input, value)` — invokes the setter directly on the element, setting the internal value without triggering the framework's synthetic event system
|
||||
- `dispatchEvent(new Event('input', { bubbles: true }))` — fires a browser-native input event that the form's validation listener recognizes
|
||||
|
||||
### Pattern (Python + Playwright)
|
||||
|
||||
```python
|
||||
async def set_value_native(page, selector, value):
|
||||
"""Set form field value bypassing JS framework validation"""
|
||||
await page.evaluate(f"""
|
||||
(() => {{
|
||||
const input = document.querySelector('{selector}');
|
||||
const setter = Object.getOwnPropertyDescriptor(
|
||||
window.HTMLInputElement.prototype, 'value'
|
||||
).set;
|
||||
setter.call(input, '{value}');
|
||||
input.dispatchEvent(new Event('input', {{ bubbles: true }}));
|
||||
}})()
|
||||
""")
|
||||
```
|
||||
|
||||
**⚠️ Pitfall: escaping special characters.** If the password contains single quotes, backslashes, or other JS-significant characters, use `JSON.stringify(value)` or pass the value as a function argument:
|
||||
|
||||
```python
|
||||
await page.evaluate("""
|
||||
([selector, value]) => {
|
||||
const input = document.querySelector(selector);
|
||||
const setter = Object.getOwnPropertyDescriptor(
|
||||
window.HTMLInputElement.prototype, 'value'
|
||||
).set;
|
||||
setter.call(input, value);
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
}
|
||||
""", [selector, password])
|
||||
```
|
||||
|
||||
### When to use this
|
||||
|
||||
- Playwright `fill()` / `type()` / `pressSequentially()` all fail
|
||||
- The form shows "This field is required" even though the field visually has content
|
||||
- The form uses custom JavaScript validation (common in ASP.NET MVC, Angular, React forms)
|
||||
- Specifically for password fields — text fields rarely have this issue
|
||||
|
||||
### Verification
|
||||
|
||||
After calling `setValueNative`, verify the field actually has the value:
|
||||
|
||||
```javascript
|
||||
document.querySelector('input#Password').value.length // Should be > 0
|
||||
```
|
||||
|
||||
## 2. Invisible reCAPTCHA with `g-recaptcha` Class
|
||||
|
||||
### Identifying invisible reCAPTCHA
|
||||
|
||||
The form button has these attributes:
|
||||
|
||||
```html
|
||||
<button class="btn btn-default g-recaptcha"
|
||||
data-sitekey="6Lc1DN8ZAAAAAJ24AatUEzMn95wjUkjmHt65BfQi"
|
||||
data-callback="onSubmit"
|
||||
data-action="CreateAccount">
|
||||
Create Account
|
||||
</button>
|
||||
```
|
||||
|
||||
Key signals:
|
||||
- **`class="g-recaptcha"`** — Google's CSS class for automatic binding
|
||||
- **`data-callback="onSubmit"`** — function called when reCAPTCHA challenge succeeds
|
||||
- **`data-sitekey="..."`** — the reCAPTCHA site key
|
||||
- **No `type="submit"`** — the button is NOT a submit button; reCAPTCHA handles submission
|
||||
|
||||
### How it works
|
||||
|
||||
1. Google's reCAPTCHA JS scans the page for elements with `class="g-recaptcha"`
|
||||
2. When the button is clicked, reCAPTCHA intercepts the click
|
||||
3. Runs the invisible challenge (no user interaction required)
|
||||
4. On success, calls `onSubmit(token)` — which sets the token in a hidden field and submits the form
|
||||
|
||||
### The `onSubmit` callback pattern
|
||||
|
||||
Typically looks like:
|
||||
|
||||
```javascript
|
||||
function onSubmit(token) {
|
||||
$("#RecaptchaToken").val(token);
|
||||
document.getElementById("createAccountForm").submit();
|
||||
}
|
||||
```
|
||||
|
||||
### Hidden field for token
|
||||
|
||||
The form has a hidden input that stores the reCAPTCHA token:
|
||||
|
||||
[REDACTED]
|
||||
<input id="RecaptchaToken" name="RecaptchaToken" type="hidden" value="" />
|
||||
```
|
||||
|
||||
This is DIFFERENT from the standard `g-recaptcha-response` hidden field. Some custom implementations use their own field name.
|
||||
|
||||
### Approach 1: Solve via 2captcha + manual injection
|
||||
|
||||
```python
|
||||
import requests, time
|
||||
|
||||
CAPTCHA_KEY = "your_2captcha_key"
|
||||
SITEKEY = "6Lc1DN8ZAAAAAJ24AatUEzMn95wjUkjmHt65BfQi"
|
||||
PAGEURL = "https://target.com/register"
|
||||
|
||||
# 1. Submit to 2captcha
|
||||
resp = requests.get("https://2captcha.com/in.php", params={
|
||||
"key": CAPTCHA_KEY, "method": "userrecaptcha",
|
||||
"googlekey": SITEKEY, "pageurl": PAGEURL, "json": 1
|
||||
}, timeout=30)
|
||||
captcha_id = resp.json()["request"]
|
||||
|
||||
# 2. Poll for result
|
||||
for i in range(20):
|
||||
time.sleep(5)
|
||||
resp = requests.get("https://2captcha.com/res.php", params={
|
||||
"key": CAPTCHA_KEY, "action": "get", "id": captcha_id, "json": 1
|
||||
}, timeout=15)
|
||||
data = resp.json()
|
||||
if data.get("status") == 1:
|
||||
token = data["request"]
|
||||
break
|
||||
|
||||
# 3. Inject into browser (via page.evaluate or browser_console)
|
||||
await page.evaluate("""
|
||||
(token) => {
|
||||
document.getElementById('RecaptchaToken').value = token;
|
||||
}
|
||||
""", token)
|
||||
|
||||
# 4. Trigger onSubmit
|
||||
await page.evaluate("""
|
||||
onSubmit(document.getElementById('RecaptchaToken').value);
|
||||
""")
|
||||
```
|
||||
|
||||
### Approach 2: Browser-native `grecaptcha.execute()`
|
||||
|
||||
If the reCAPTCHA script is loaded and the widget is rendered:
|
||||
|
||||
```javascript
|
||||
// In browser_console or page.evaluate:
|
||||
grecaptcha.execute();
|
||||
```
|
||||
|
||||
This returns a Promise that resolves to a fresh token. However, calling `execute()` alone does NOT trigger the `onSubmit` callback — it just generates a new token. You still need to either:
|
||||
- click the `g-recaptcha` button (which internally calls execute + callback)
|
||||
- manually call `onSubmit(token)` after getting the token
|
||||
|
||||
### Approach 3: Click the button (preferred)
|
||||
|
||||
If the password field is filled correctly (via `setNativeValue`), simply clicking the `g-recaptcha` button triggers the full flow:
|
||||
|
||||
```javascript
|
||||
document.querySelector('.g-recaptcha').click();
|
||||
```
|
||||
|
||||
The reCAPTCHA library intercepts the click, runs the challenge, and calls `onSubmit` on success.
|
||||
|
||||
### Known issues
|
||||
|
||||
- **Migration warning**: Some reCAPTCHA site keys show "Google reCAPTCHA migration" warnings in the console. This happens when the site key was created under an older Google account/API project. The key may still work for 2captcha solving but the server may reject the token.
|
||||
- **Token expiry**: reCAPTCHA tokens expire after ~2 minutes. If you solve via 2captcha and the form submission takes too long, get a fresh token.
|
||||
- **Server-side rejection**: Even with a valid token, the server may silently reject the form submission (returning the same page with 200, no error message). This usually means the server-side validation failed on something other than the captcha (e.g., duplicate username, email format, anti-forgery token mismatch).
|
||||
|
||||
## 3. ASP.NET Anti-Forgery Token Handling
|
||||
|
||||
ASP.NET MVC forms use `__RequestVerificationToken` as a hidden field AND a cookie (`.__AspNetCore.Antiforgery.*`). Both must match.
|
||||
|
||||
### In the browser
|
||||
|
||||
The browser automatically handles this — the cookie is set on page load, and the hidden field value is embedded in the form. `form.submit()` sends both correctly.
|
||||
|
||||
### Via curl/requests
|
||||
|
||||
1. GET the form page to receive the cookie
|
||||
2. Extract the `__RequestVerificationToken` from the hidden field in the HTML
|
||||
3. POST with both the cookie and the token value
|
||||
|
||||
```bash
|
||||
# Step 1: GET form (stores cookie)
|
||||
curl -c cookies.txt -b cookies.txt -L 'https://target.com/Account/CreateAccount'
|
||||
|
||||
# Step 2: Extract token from HTML
|
||||
TOKEN=$(grep -oP '__RequestVerificationToken" value="\K[^"]+' cookies.txt || \
|
||||
grep -oP '__RequestVerificationToken" value="\K[^"]+' response.html)
|
||||
|
||||
# Step 3: POST with form data
|
||||
curl -c cookies.txt -b cookies.txt \
|
||||
-X POST 'https://target.com/Account/CreateAccount' \
|
||||
-d "__RequestVerificationToken=$TOKEN" \
|
||||
-d "Username=testuser" \
|
||||
-d "Password=[REDACTED]"
|
||||
```
|
||||
|
||||
**Without the correct token**: server returns HTTP 400 Bad Request.
|
||||
|
||||
## 4. Chromium SOCKS5 Proxy (unauthenticated)
|
||||
|
||||
Playwright/Chromium does NOT support SOCKS5 with authentication. Workaround: set up an SSH tunnel without auth:
|
||||
|
||||
```bash
|
||||
# Create tunnel (no auth needed)
|
||||
ssh -D 1081 -N root@your-server.com
|
||||
|
||||
# In Playwright
|
||||
browser = await playwright.chromium.launch(
|
||||
args=["--proxy-server=socks5://localhost:1081"]
|
||||
)
|
||||
```
|
||||
|
||||
The tunnel must be running before launching the browser. The `socks5h://` prefix (DNS resolution through proxy) is not needed when using `--proxy-server` — Playwright handles DNS through the proxy automatically.
|
||||
|
||||
## 5. Diagnostic Flow for Form Submission Failures
|
||||
|
||||
When a form submits but returns the same page (no error, no redirect):
|
||||
|
||||
1. **Check password validation** — inspect `document.getElementById('Password').value.length` → if 0, the password was cleared by the page reload. Use `setNativeValue` technique.
|
||||
2. **Check reCAPTCHA token** — inspect `document.getElementById('RecaptchaToken').value` → if empty or stale, re-solve.
|
||||
3. **Check anti-forgery token** — inspect `document.querySelector('input[name=__RequestVerificationToken]').value` → if empty, the page state reset.
|
||||
4. **Check for validation errors** — `document.querySelector('.validation-summary-errors, .text-danger')?` → if present, read the error text.
|
||||
5. **Check browser console** — `console.clear()` then `console.log('ready')` before action, check for JS errors after.
|
||||
6. **Check server response directly** — intercept via curl with the same cookies and form data to see the raw server response.
|
||||
7. **Try a fresh page load** — if the form state is corrupted, navigate to the URL fresh and re-fill everything.
|
||||
Reference in New Issue
Block a user