51 lines
1.7 KiB
Markdown
51 lines
1.7 KiB
Markdown
# httpx Concurrent Client Cookie Corruption
|
|
|
|
## Symptom
|
|
All 30 parallel requests return HTTP 401, but individual tests succeed.
|
|
|
|
## Root Cause
|
|
Sharing ONE `httpx.AsyncClient` instance across multiple concurrent `asyncio` tasks corrupts cookie handling. httpx reuses connection pools and cookie jars that are not thread/async-safe when multiple tasks set different cookies.
|
|
|
|
## Wrong Pattern
|
|
```python
|
|
# ❌ Creates one shared client — cookies from different accounts mix up
|
|
async def main():
|
|
async with httpx.AsyncClient(timeout=httpx.Timeout(30)) as shared_client:
|
|
# All 30 concurrent tasks use the SAME client
|
|
tasks = [test_account(shared_client, acc) for acc in accounts]
|
|
await asyncio.gather(*tasks)
|
|
|
|
async def test_account(client, acc):
|
|
# Setting cookies on the shared client
|
|
client.cookies = {name: value} # Gets overwritten by other tasks!
|
|
r = await client.post(...)
|
|
```
|
|
|
|
## Correct Pattern
|
|
```python
|
|
# ✅ Each concurrent task creates its OWN client
|
|
async def test_account(acc):
|
|
cookies = load_cookies(acc['cookies_path'])
|
|
async with httpx.AsyncClient(cookies=cookies, timeout=httpx.Timeout(30)) as client:
|
|
r = await client.post(...)
|
|
|
|
async def main():
|
|
tasks = [test_account(acc) for acc in accounts]
|
|
await asyncio.gather(*tasks)
|
|
```
|
|
|
|
## Concurrency Control
|
|
Use `asyncio.Semaphore(N)` to limit concurrent connections:
|
|
```python
|
|
sem = asyncio.Semaphore(3)
|
|
|
|
async def bounded(acc):
|
|
async with sem:
|
|
return await test_account(acc)
|
|
|
|
results = await asyncio.gather(*[bounded(a) for a in accounts])
|
|
```
|
|
|
|
## Detection
|
|
If you see HTTP 401 on all requests in parallel but HTTP 201 on individual sequential requests, this is almost certainly the shared client issue.
|