KeyFarm lane contract (read fully before writing code)

Project: KeyFarm — distributed API-key/token harvest farm. A "lane" is one standalone Python 3 script that walks one public data source forever, greps content for secrets with the shared pattern module, and writes hits into one SQLite DB.

1. Where files go

Canonical staging dir: C:/Users/User/tmp/vps_stage/
New lane file name: <source>_eternal.py (snake_case, one source family per file).
Runtime target is Linux VPS (Debian, python3.11, /root/keyhunter). Code must be OS-agnostic: no drive letters, no backslash paths, no Windows-only APIs. Build paths with os.path.join(BASE, ...).
Python deps: stdlib + requests only. Never add a dependency. If a source needs one, drop that source and say so in the report.

2. Reference implementation

Read C:/Users/User/Desktop/keyhunter/clojars_eternal.py and C:/Users/User/Desktop/keyhunter/crates_eternal.py first — copy their shape exactly (log(), db_add(), load/save cursor, seen-set, work(), cycle(), main()). Do not invent a new architecture, do not create shared libs, do not refactor existing lanes.

3. Required module skeleton

"""ETERNAL <source>: <one-line what it walks>.

Endpoint(s): <real URLs you probed>
Auth: none | PAT pool (gh_tokens.txt)
"""
import os
BASE = os.path.dirname(os.path.abspath(__file__))

import hashlib, json, re, sqlite3, sys, threading, time
import requests

os.makedirs(os.path.join(BASE, "data"), exist_ok=True)  # else KF_DB smoke path fails on fresh dir
DB = os.environ.get("KF_DB") or os.path.join(BASE, "data", "keys.db")
LOG = os.path.join(BASE, "data", "<lane>_eternal.log")
CURSOR_FILE = os.path.join(BASE, "data", "<lane>_cursor.txt")
SEEN_FILE = os.path.join(BASE, "data", "<lane>_seen.txt")
N_THREADS = 4            # keep modest unless the endpoint is proven to allow more
SLEEP = 90               # seconds between cycles
MAX_ITEMS = 200          # per cycle cap; --once uses MIN(MAX_ITEMS, 5)

from key_patterns import scan_text   # -> list[(prov, key)]

DB schema (already exists — do not create tables other than with CREATE TABLE IF NOT EXISTS identical to this):

CREATE TABLE keys (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  hash TEXT UNIQUE,
  val TEXT, prov TEXT, status TEXT DEFAULT "NEW",
  plan TEXT, price TEXT, remaining TEXT,
  found TEXT, repo TEXT);

db_add(key, prov, repo) — exact statement used by every lane:

conn.execute(
    "INSERT INTO keys (hash, val, prov, status, found, repo) "
    "VALUES (?,?,?,?,?,?) "
    "ON CONFLICT(hash) DO UPDATE SET status='NEW', "
    "found=excluded.found, repo=excluded.repo "
    "WHERE keys.status IN ('DEAD','ERR')",
    (hashlib.sha1(key.encode()).hexdigest()[:16],
     key, prov, "NEW", "<lane>", repo))

found must be a short unique lane tag (e.g. wayback, gitlabjobs, dockerhub). repo is the source locator (URL, repo slug, image name) — truncate to 300 chars.

4. Network rules (hard)

S = requests.Session(); S.trust_env = False — a local dashscope proxy on 127.0.0.1:16432 intercepts api.github.com and other hosts and kills requests after ~22 min. trust_env=False is mandatory in every lane.
Every request: explicit timeout=, a User-Agent, and retry with backoff (3 tries, 2/4/8s) on connection errors and 5xx. On 429/403 read Retry-After and sleep.
Never print credentials. If a lane uses the PAT pool (gh_tokens.txt, one token per line, round-robin), mask any URL that embeds a token before logging (re.sub(r'https://[^@]*@', 'https://***@', url)).
Politeness: honour rate limits, cap payload sizes (FILE_CAP ~500KB text, archive cap ~20MB), skip binaries you cannot decode, and SLEEP between cycles.
Cap BEFORE download, never after: stream=True, check Content-Length against the cap and abort without reading the body; accumulate via iter_content(65536) into a list + b"".join(...) (not buf += chunk — O(n²) at 20MB caps). For archives, guard on ZipInfo.file_size / TarInfo.size from metadata BEFORE zf.open(name).read(cap+1) — never zf.read(name) then slice.
except MemoryError is NOT a safety net: on Linux the OOM-killer SIGKILLs the process (no exception is raised) and lane_keeper respawns it into the same wall.
Never let one bad item kill the cycle (wrap work() in try/except, count as fail).

5. Junk filter

Copy the GARBAGE tuple from clojars_eternal.py and keep the same idea: drop hits containing xxxx, your, YOUR, EXAMPLE, example, placeholder, CHANGEME, dummy, DUMMY, abcdefghijklmnopqrst, 1234567890abcdef, test, sk-... doc samples. Add a small entropy check if the source yields lots of random-looking noise:

def is_junk(v):
    if any(g in v for g in GARBAGE): return True
    if len(set(v)) < 6: return True
    return False

Length floor (verified): scan_text admits len(val) >= 20 (key_patterns); farm-wide db_add_key (src/eternal_v10.py) has NO length filter. A <30 drop is clojars's lane-local tuning for a noisy source — do NOT universalize it: AWS AKIA… key IDs are exactly 20 chars and a blanket <30 rule silently discards the whole family. Add source-specific floors only with that citation in mind.

scan_text filters GARBAGE internally — a correct pattern can legitimately return [] on a fixture whose value contains a GARBAGE substring. Use high-entropy realistic fixtures; never sequential (abcdef...) or placeholder-looking strings in assertions.

6. CLI + smoke discipline

main() loops cycle() + time.sleep(SLEEP) forever; if __name__ == "__main__": main().
Support --once: run exactly one bounded cycle with MAX_ITEMS = min(MAX_ITEMS, 5) and exit 0. Used for smoke tests.
Never touch the production DB in a smoke run: set KF_DB=data/_smoke_<lane>.db.
Never deploy anything, never ssh/sftp, never start a long-running scan, never mass-crawl. Total smoke budget: ≤ 40 HTTP requests and ≤ 120 seconds.
Leave ONE runnable check behind: test_<lane>.py with plain asserts, no network, using a fixture string of realistic source content — it must prove (a) parsing/extraction of candidates from that content, (b) junk filtering, (c) db_add writes a row and dedupes on repeat. Run it with python test_<lane>.py and show output.
Verify in a fresh subprocess (python test_<lane>.py / python -c "import ..."), never in the same process that just wrote the file — a cached module object gives false-negative asserts on correct code.

7. Endpoint reality rule

Before writing a lane for a source, probe it with one real request and confirm: the endpoint exists, needs no auth (or works with the PAT pool), and returns usable data. If it 404s / needs a key / is bot-walled — do not write speculative code against it. Drop it and list it in the report under "rejected (with reason)". No invented APIs.

8. Style

≤ 400 lines per lane file. Flat functions, early returns, no classes unless stateful.
English identifiers/comments/strings. No emojis. No debug prints left behind.
Constants at top, no magic numbers inline.
Immutable-ish: prefer returning new values over mutating shared state; guard the sqlite connection + seen set with the module lock.

9. Report format (return this)

LANE: <file(s) created>
SOURCE: <what it walks> — endpoint(s) probed, auth
QUOTA: <per-IP vs per-token> + observed rate limit + COLLISION check vs LIVE lanes' hosts: gitlab.com (gitlab_eternal + gitlab_artifacts — real collision), archive.org (wayback), sourceforge.net, sourcegraph.com, marketplace.visualstudio.com, cdn.jsdelivr.net/data.jsdelivr.com, index.commoncrawl.org/data.commoncrawl.org, api.github.com (PAT pool, per-token). Disabled hosts (currently free): hub.docker.com, jenkins seeds.
SMOKE: <command> → <items scanned / keys found / exit code>
TEST: python test_<lane>.py → <pass line>
CAPS: N_THREADS / SLEEP / MAX_ITEMS / FILE_CAP
REJECTED: <sources probed and dropped, with reason>
NOTES: <rate-limit findings, dedupe notes, deploy prerequisites>

8.2 KiB Raw Blame History