KeyHunter parser v10 + 18 scanners + tests (synthetic fixtures only, no keys/db/tokens)
This commit is contained in:
165
LANE_CONTRACT.md
Normal file
165
LANE_CONTRACT.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# KeyFarm lane contract (read fully before writing code)
|
||||
|
||||
Project: KeyFarm — distributed API-key/token harvest farm. A "lane" is one standalone
|
||||
Python 3 script that walks one public data source forever, greps content for secrets with
|
||||
the shared pattern module, and writes hits into one SQLite DB.
|
||||
|
||||
## 1. Where files go
|
||||
|
||||
- Canonical staging dir: `C:/Users/User/tmp/vps_stage/`
|
||||
- New lane file name: `<source>_eternal.py` (snake_case, one source family per file).
|
||||
- Runtime target is **Linux VPS** (Debian, python3.11, `/root/keyhunter`). Code must be
|
||||
OS-agnostic: no drive letters, no backslash paths, no Windows-only APIs. Build paths with
|
||||
`os.path.join(BASE, ...)`.
|
||||
- Python deps: **stdlib + `requests` only**. Never add a dependency. If a source needs one,
|
||||
drop that source and say so in the report.
|
||||
|
||||
## 2. Reference implementation
|
||||
|
||||
Read `C:/Users/User/Desktop/keyhunter/clojars_eternal.py` and
|
||||
`C:/Users/User/Desktop/keyhunter/crates_eternal.py` first — copy their shape exactly
|
||||
(log(), db_add(), load/save cursor, seen-set, work(), cycle(), main()).
|
||||
Do not invent a new architecture, do not create shared libs, do not refactor existing lanes.
|
||||
|
||||
## 3. Required module skeleton
|
||||
|
||||
```python
|
||||
"""ETERNAL <source>: <one-line what it walks>.
|
||||
|
||||
Endpoint(s): <real URLs you probed>
|
||||
Auth: none | PAT pool (gh_tokens.txt)
|
||||
"""
|
||||
import os
|
||||
BASE = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
import hashlib, json, re, sqlite3, sys, threading, time
|
||||
import requests
|
||||
|
||||
os.makedirs(os.path.join(BASE, "data"), exist_ok=True) # else KF_DB smoke path fails on fresh dir
|
||||
DB = os.environ.get("KF_DB") or os.path.join(BASE, "data", "keys.db")
|
||||
LOG = os.path.join(BASE, "data", "<lane>_eternal.log")
|
||||
CURSOR_FILE = os.path.join(BASE, "data", "<lane>_cursor.txt")
|
||||
SEEN_FILE = os.path.join(BASE, "data", "<lane>_seen.txt")
|
||||
N_THREADS = 4 # keep modest unless the endpoint is proven to allow more
|
||||
SLEEP = 90 # seconds between cycles
|
||||
MAX_ITEMS = 200 # per cycle cap; --once uses MIN(MAX_ITEMS, 5)
|
||||
|
||||
from key_patterns import scan_text # -> list[(prov, key)]
|
||||
```
|
||||
|
||||
DB schema (already exists — do not create tables other than with
|
||||
`CREATE TABLE IF NOT EXISTS` identical to this):
|
||||
|
||||
```sql
|
||||
CREATE TABLE keys (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
hash TEXT UNIQUE,
|
||||
val TEXT, prov TEXT, status TEXT DEFAULT "NEW",
|
||||
plan TEXT, price TEXT, remaining TEXT,
|
||||
found TEXT, repo TEXT);
|
||||
```
|
||||
|
||||
`db_add(key, prov, repo)` — exact statement used by every lane:
|
||||
|
||||
```python
|
||||
conn.execute(
|
||||
"INSERT INTO keys (hash, val, prov, status, found, repo) "
|
||||
"VALUES (?,?,?,?,?,?) "
|
||||
"ON CONFLICT(hash) DO UPDATE SET status='NEW', "
|
||||
"found=excluded.found, repo=excluded.repo "
|
||||
"WHERE keys.status IN ('DEAD','ERR')",
|
||||
(hashlib.sha1(key.encode()).hexdigest()[:16],
|
||||
key, prov, "NEW", "<lane>", repo))
|
||||
```
|
||||
|
||||
`found` must be a short unique lane tag (e.g. `wayback`, `gitlabjobs`, `dockerhub`).
|
||||
`repo` is the source locator (URL, repo slug, image name) — truncate to 300 chars.
|
||||
|
||||
## 4. Network rules (hard)
|
||||
|
||||
- `S = requests.Session(); S.trust_env = False` — a local dashscope proxy on
|
||||
`127.0.0.1:16432` intercepts `api.github.com` and other hosts and kills requests after
|
||||
~22 min. `trust_env=False` is mandatory in every lane.
|
||||
- Every request: explicit `timeout=`, a `User-Agent`, and retry with backoff
|
||||
(3 tries, 2/4/8s) on connection errors and 5xx. On 429/403 read `Retry-After` and sleep.
|
||||
- Never print credentials. If a lane uses the PAT pool (`gh_tokens.txt`, one token per
|
||||
line, round-robin), mask any URL that embeds a token before logging
|
||||
(`re.sub(r'https://[^@]*@', 'https://***@', url)`).
|
||||
- Politeness: honour rate limits, cap payload sizes (`FILE_CAP` ~500KB text,
|
||||
archive cap ~20MB), skip binaries you cannot decode, and `SLEEP` between cycles.
|
||||
- Cap BEFORE download, never after: `stream=True`, check `Content-Length` against
|
||||
the cap and abort without reading the body; accumulate via `iter_content(65536)`
|
||||
into a list + `b"".join(...)` (not `buf += chunk` — O(n²) at 20MB caps). For
|
||||
archives, guard on `ZipInfo.file_size` / `TarInfo.size` from metadata BEFORE
|
||||
`zf.open(name).read(cap+1)` — never `zf.read(name)` then slice.
|
||||
- `except MemoryError` is NOT a safety net: on Linux the OOM-killer SIGKILLs the
|
||||
process (no exception is raised) and lane_keeper respawns it into the same wall.
|
||||
- Never let one bad item kill the cycle (wrap `work()` in try/except, count as `fail`).
|
||||
|
||||
## 5. Junk filter
|
||||
|
||||
Copy the `GARBAGE` tuple from `clojars_eternal.py` and keep the same idea: drop hits
|
||||
containing `xxxx`, `your`, `YOUR`, `EXAMPLE`, `example`, `placeholder`, `CHANGEME`,
|
||||
`dummy`, `DUMMY`, `abcdefghijklmnopqrst`, `1234567890abcdef`, `test`, `sk-...` doc samples.
|
||||
Add a small entropy check if the source yields lots of random-looking noise:
|
||||
|
||||
```python
|
||||
def is_junk(v):
|
||||
if any(g in v for g in GARBAGE): return True
|
||||
if len(set(v)) < 6: return True
|
||||
return False
|
||||
```
|
||||
**Length floor (verified):** `scan_text` admits `len(val) >= 20` (key_patterns);
|
||||
farm-wide `db_add_key` (src/eternal_v10.py) has NO length filter. A `<30` drop is
|
||||
clojars's **lane-local** tuning for a noisy source — do NOT universalize it: AWS
|
||||
`AKIA…` key IDs are exactly 20 chars and a blanket `<30` rule silently discards
|
||||
the whole family. Add source-specific floors only with that citation in mind.
|
||||
|
||||
**`scan_text` filters GARBAGE internally** — a correct pattern can legitimately return `[]`
|
||||
on a fixture whose value contains a GARBAGE substring. Use high-entropy realistic fixtures;
|
||||
never sequential (`abcdef...`) or placeholder-looking strings in assertions.
|
||||
|
||||
## 6. CLI + smoke discipline
|
||||
|
||||
- `main()` loops `cycle()` + `time.sleep(SLEEP)` forever; `if __name__ == "__main__": main()`.
|
||||
- Support `--once`: run exactly one bounded cycle with `MAX_ITEMS = min(MAX_ITEMS, 5)`
|
||||
and exit 0. Used for smoke tests.
|
||||
- **Never** touch the production DB in a smoke run: set `KF_DB=data/_smoke_<lane>.db`.
|
||||
- **Never** deploy anything, never ssh/sftp, never start a long-running scan,
|
||||
never mass-crawl. Total smoke budget: **≤ 40 HTTP requests and ≤ 120 seconds**.
|
||||
- Leave ONE runnable check behind: `test_<lane>.py` with plain `assert`s, **no network**,
|
||||
using a fixture string of realistic source content — it must prove
|
||||
(a) parsing/extraction of candidates from that content, (b) junk filtering,
|
||||
(c) `db_add` writes a row and dedupes on repeat. Run it with
|
||||
`python test_<lane>.py` and show output.
|
||||
- Verify in a **fresh subprocess** (`python test_<lane>.py` / `python -c "import ..."`),
|
||||
never in the same process that just wrote the file — a cached module object gives
|
||||
false-negative asserts on correct code.
|
||||
|
||||
## 7. Endpoint reality rule
|
||||
|
||||
Before writing a lane for a source, **probe it with one real request** and confirm:
|
||||
the endpoint exists, needs no auth (or works with the PAT pool), and returns usable data.
|
||||
If it 404s / needs a key / is bot-walled — do **not** write speculative code against it.
|
||||
Drop it and list it in the report under "rejected (with reason)". No invented APIs.
|
||||
|
||||
## 8. Style
|
||||
|
||||
- ≤ 400 lines per lane file. Flat functions, early returns, no classes unless stateful.
|
||||
- English identifiers/comments/strings. No emojis. No debug prints left behind.
|
||||
- Constants at top, no magic numbers inline.
|
||||
- Immutable-ish: prefer returning new values over mutating shared state; guard the
|
||||
sqlite connection + `seen` set with the module `lock`.
|
||||
|
||||
## 9. Report format (return this)
|
||||
|
||||
```
|
||||
LANE: <file(s) created>
|
||||
SOURCE: <what it walks> — endpoint(s) probed, auth
|
||||
QUOTA: <per-IP vs per-token> + observed rate limit + COLLISION check vs LIVE lanes' hosts: gitlab.com (gitlab_eternal + gitlab_artifacts — real collision), archive.org (wayback), sourceforge.net, sourcegraph.com, marketplace.visualstudio.com, cdn.jsdelivr.net/data.jsdelivr.com, index.commoncrawl.org/data.commoncrawl.org, api.github.com (PAT pool, per-token). Disabled hosts (currently free): hub.docker.com, jenkins seeds.
|
||||
SMOKE: <command> → <items scanned / keys found / exit code>
|
||||
TEST: python test_<lane>.py → <pass line>
|
||||
CAPS: N_THREADS / SLEEP / MAX_ITEMS / FILE_CAP
|
||||
REJECTED: <sources probed and dropped, with reason>
|
||||
NOTES: <rate-limit findings, dedupe notes, deploy prerequisites>
|
||||
```
|
||||
Reference in New Issue
Block a user