keyhunter-parser/LANE_CONTRACT.md

# KeyFarm lane contract (read fully before writing code)

Project: KeyFarm — distributed API-key/token harvest farm. A "lane" is one standalone
Python 3 script that walks one public data source forever, greps content for secrets with
the shared pattern module, and writes hits into one SQLite DB.

## 1. Where files go

- Canonical staging dir: `C:/Users/User/tmp/vps_stage/`
- New lane file name: `<source>_eternal.py` (snake_case, one source family per file).
- Runtime target is **Linux VPS** (Debian, python3.11, `/root/keyhunter`). Code must be
  OS-agnostic: no drive letters, no backslash paths, no Windows-only APIs. Build paths with
  `os.path.join(BASE, ...)`.
- Python deps: **stdlib + `requests` only**. Never add a dependency. If a source needs one,
  drop that source and say so in the report.

## 2. Reference implementation

Read `C:/Users/User/Desktop/keyhunter/clojars_eternal.py` and
`C:/Users/User/Desktop/keyhunter/crates_eternal.py` first — copy their shape exactly
(log(), db_add(), load/save cursor, seen-set, work(), cycle(), main()).
Do not invent a new architecture, do not create shared libs, do not refactor existing lanes.

## 3. Required module skeleton

```python
"""ETERNAL <source>: <one-line what it walks>.

Endpoint(s): <real URLs you probed>
Auth: none | PAT pool (gh_tokens.txt)
"""
import os
BASE = os.path.dirname(os.path.abspath(__file__))

import hashlib, json, re, sqlite3, sys, threading, time
import requests

os.makedirs(os.path.join(BASE, "data"), exist_ok=True)  # else KF_DB smoke path fails on fresh dir
DB = os.environ.get("KF_DB") or os.path.join(BASE, "data", "keys.db")
LOG = os.path.join(BASE, "data", "<lane>_eternal.log")
CURSOR_FILE = os.path.join(BASE, "data", "<lane>_cursor.txt")
SEEN_FILE = os.path.join(BASE, "data", "<lane>_seen.txt")
N_THREADS = 4            # keep modest unless the endpoint is proven to allow more
SLEEP = 90               # seconds between cycles
MAX_ITEMS = 200          # per cycle cap; --once uses MIN(MAX_ITEMS, 5)

from key_patterns import scan_text   # -> list[(prov, key)]
```

DB schema (already exists — do not create tables other than with
`CREATE TABLE IF NOT EXISTS` identical to this):

```sql
CREATE TABLE keys (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  hash TEXT UNIQUE,
  val TEXT, prov TEXT, status TEXT DEFAULT "NEW",
  plan TEXT, price TEXT, remaining TEXT,
  found TEXT, repo TEXT);
```

`db_add(key, prov, repo)` — exact statement used by every lane:

```python
conn.execute(
    "INSERT INTO keys (hash, val, prov, status, found, repo) "
    "VALUES (?,?,?,?,?,?) "
    "ON CONFLICT(hash) DO UPDATE SET status='NEW', "
    "found=excluded.found, repo=excluded.repo "
    "WHERE keys.status IN ('DEAD','ERR')",
    (hashlib.sha1(key.encode()).hexdigest()[:16],
     key, prov, "NEW", "<lane>", repo))
```

`found` must be a short unique lane tag (e.g. `wayback`, `gitlabjobs`, `dockerhub`).
`repo` is the source locator (URL, repo slug, image name) — truncate to 300 chars.

## 4. Network rules (hard)

- `S = requests.Session(); S.trust_env = False` — a local dashscope proxy on
  `127.0.0.1:16432` intercepts `api.github.com` and other hosts and kills requests after
  ~22 min. `trust_env=False` is mandatory in every lane.
- Every request: explicit `timeout=`, a `User-Agent`, and retry with backoff
  (3 tries, 2/4/8s) on connection errors and 5xx. On 429/403 read `Retry-After` and sleep.
- Never print credentials. If a lane uses the PAT pool (`gh_tokens.txt`, one token per
  line, round-robin), mask any URL that embeds a token before logging
  (`re.sub(r'https://[^@]*@', 'https://***@', url)`).
- Politeness: honour rate limits, cap payload sizes (`FILE_CAP` ~500KB text,
  archive cap ~20MB), skip binaries you cannot decode, and `SLEEP` between cycles.
- Cap BEFORE download, never after: `stream=True`, check `Content-Length` against
  the cap and abort without reading the body; accumulate via `iter_content(65536)`
  into a list + `b"".join(...)` (not `buf += chunk` — O(n²) at 20MB caps). For
  archives, guard on `ZipInfo.file_size` / `TarInfo.size` from metadata BEFORE
  `zf.open(name).read(cap+1)` — never `zf.read(name)` then slice.
- `except MemoryError` is NOT a safety net: on Linux the OOM-killer SIGKILLs the
  process (no exception is raised) and lane_keeper respawns it into the same wall.
- Never let one bad item kill the cycle (wrap `work()` in try/except, count as `fail`).

## 5. Junk filter

Copy the `GARBAGE` tuple from `clojars_eternal.py` and keep the same idea: drop hits
containing `xxxx`, `your`, `YOUR`, `EXAMPLE`, `example`, `placeholder`, `CHANGEME`,
`dummy`, `DUMMY`, `abcdefghijklmnopqrst`, `1234567890abcdef`, `test`, `sk-...` doc samples.
Add a small entropy check if the source yields lots of random-looking noise:

```python
def is_junk(v):
    if any(g in v for g in GARBAGE): return True
    if len(set(v)) < 6: return True
    return False
```
**Length floor (verified):** `scan_text` admits `len(val) >= 20` (key_patterns);
farm-wide `db_add_key` (src/eternal_v10.py) has NO length filter. A `<30` drop is
clojars's **lane-local** tuning for a noisy source — do NOT universalize it: AWS
`AKIA…` key IDs are exactly 20 chars and a blanket `<30` rule silently discards
the whole family. Add source-specific floors only with that citation in mind.

**`scan_text` filters GARBAGE internally** — a correct pattern can legitimately return `[]`
on a fixture whose value contains a GARBAGE substring. Use high-entropy realistic fixtures;
never sequential (`abcdef...`) or placeholder-looking strings in assertions.

## 6. CLI + smoke discipline

- `main()` loops `cycle()` + `time.sleep(SLEEP)` forever; `if __name__ == "__main__": main()`.
- Support `--once`: run exactly one bounded cycle with `MAX_ITEMS = min(MAX_ITEMS, 5)`
  and exit 0. Used for smoke tests.
- **Never** touch the production DB in a smoke run: set `KF_DB=data/_smoke_<lane>.db`.
- **Never** deploy anything, never ssh/sftp, never start a long-running scan,
  never mass-crawl. Total smoke budget: **≤ 40 HTTP requests and ≤ 120 seconds**.
- Leave ONE runnable check behind: `test_<lane>.py` with plain `assert`s, **no network**,
  using a fixture string of realistic source content — it must prove
  (a) parsing/extraction of candidates from that content, (b) junk filtering,
  (c) `db_add` writes a row and dedupes on repeat. Run it with
  `python test_<lane>.py` and show output.
- Verify in a **fresh subprocess** (`python test_<lane>.py` / `python -c "import ..."`),
  never in the same process that just wrote the file — a cached module object gives
  false-negative asserts on correct code.

## 7. Endpoint reality rule

Before writing a lane for a source, **probe it with one real request** and confirm:
the endpoint exists, needs no auth (or works with the PAT pool), and returns usable data.
If it 404s / needs a key / is bot-walled — do **not** write speculative code against it.
Drop it and list it in the report under "rejected (with reason)". No invented APIs.

## 8. Style

- ≤ 400 lines per lane file. Flat functions, early returns, no classes unless stateful.
- English identifiers/comments/strings. No emojis. No debug prints left behind.
- Constants at top, no magic numbers inline.
- Immutable-ish: prefer returning new values over mutating shared state; guard the
  sqlite connection + `seen` set with the module `lock`.

## 9. Report format (return this)

```
LANE: <file(s) created>
SOURCE: <what it walks> — endpoint(s) probed, auth
QUOTA: <per-IP vs per-token> + observed rate limit + COLLISION check vs LIVE lanes' hosts: gitlab.com (gitlab_eternal + gitlab_artifacts — real collision), archive.org (wayback), sourceforge.net, sourcegraph.com, marketplace.visualstudio.com, cdn.jsdelivr.net/data.jsdelivr.com, index.commoncrawl.org/data.commoncrawl.org, api.github.com (PAT pool, per-token). Disabled hosts (currently free): hub.docker.com, jenkins seeds.
SMOKE: <command> → <items scanned / keys found / exit code>
TEST: python test_<lane>.py → <pass line>
CAPS: N_THREADS / SLEEP / MAX_ITEMS / FILE_CAP
REJECTED: <sources probed and dropped, with reason>
NOTES: <rate-limit findings, dedupe notes, deploy prerequisites>
```