66 lines
2.2 KiB
Markdown
66 lines
2.2 KiB
Markdown
# V2EX `/go/free` Public Topic Parser
|
|
|
|
Session: V2EX account validation + freebie harvesting.
|
|
|
|
## What it does
|
|
Parses the public `/go/free` node (and any other `/go/<node>`) without authentication. Extracts topic ID, title, author, reply count, and full URL.
|
|
|
|
## Why not the API?
|
|
V2EX does not expose a public API for node topic lists. The site serves HTML with `div.cell` rows.
|
|
|
|
## Working selectors (current as of 2026-07-02)
|
|
|
|
```python
|
|
import requests
|
|
from bs4 import BeautifulSoup
|
|
|
|
def parse_v2ex_node(node='free', max_pages=5):
|
|
session = requests.Session()
|
|
session.headers.update({'User-Agent': 'Mozilla/5.0'})
|
|
topics = []
|
|
|
|
for page in range(1, max_pages + 1):
|
|
url = f'https://www.v2ex.com/go/{node}?p={page}'
|
|
r = session.get(url, timeout=15)
|
|
r.raise_for_status()
|
|
soup = BeautifulSoup(r.text, 'html.parser')
|
|
|
|
for cell in soup.select('div.cell'):
|
|
title_link = cell.select_one('span.item_title a')
|
|
if not title_link:
|
|
continue
|
|
topics.append({
|
|
'id': title_link['href'].split('/')[-1],
|
|
'title': title_link.get_text(strip=True),
|
|
'url': 'https://www.v2ex.com' + title_link['href'],
|
|
'author': (cell.select_one('a[href^="/member/"]') or {}).get_text(strip=True),
|
|
'replies': (cell.select_one('a.count_livid') or {}).get_text(strip=True) or '0',
|
|
})
|
|
|
|
return topics
|
|
```
|
|
|
|
## HTML structure
|
|
|
|
```html
|
|
<div class="cell">
|
|
<span class="item_title">
|
|
<a href="/t/1234567">Title text</a>
|
|
</span>
|
|
<a href="/member/author">author</a>
|
|
<a class="count_livid" href="/t/1234567#reply123">42</a>
|
|
</div>
|
|
```
|
|
|
|
## Common mistakes
|
|
- Searching for `div.cell.item` — the `item` class is absent in current markup.
|
|
- Using browser tools for bulk public scraping — slower and prone to connection drops.
|
|
|
|
## Rate limits
|
|
- Same 600 req/hr/IP as the rest of V2EX.
|
|
- No auth required, so no token to rotate; use proxies if you need high volume.
|
|
|
|
## Next steps
|
|
- Cross-reference authors with validated account dumps (e.g. `v2ex_valid_with_passwords.json`).
|
|
- For replies containing gift codes, fetch each topic page and search for keywords (Netflix, Spotify, `key`, `token`, `兑换`, `邀请码`).
|