# tests/ - the executable half of this repository's evidence

`tests/benchmark.md` is a **record**: what happened when a route was run once, by hand,
against a real sample and a real device. This directory is the other half: checks that run
on every commit with nothing but the standard library and pytest.

The two are not substitutes. A benchmark row can establish that a route works on a hardened
target; it cannot tell you that yesterday's refactor broke the dex width table. A test in
here can tell you that every time, and it can never tell you whether the route achieves the
goal on a real app.

```
python -m pytest -q                    # whole suite
python tests/run_tests.py              # same thing, with a clear message if pytest is missing
python -m pytest -q -m unit            # one marker
python -m pytest -q -m cli             # exit codes, flags, --json
python -m pytest -q -m integration     # no-device flows
python -m pytest -q -ra                # list every skip and xfail reason
```

## What the markers mean

- `unit` - pure functions and parsers of real shipped scripts: the dex width table and branch
  arithmetic, the scheme-less protobuf decoder, the blob rotation search, the leak rules, and
  the established `dex_patch_bytes` regression hash.
- `cli` - one subprocess run per script per invocation shape: `--help`, an unknown flag, no
  arguments, and every input pointed at a file that does not exist.
- `integration` - flows that were made of several parts: adb argv construction under a fake
  runner, logcat signal counting, tombstone parsing, the two-sample launch-health verdict.

## Nothing here needs a device

No test in this suite opens a frida session or touches the network. The device-dependent
behaviour is tested by intercepting the command that *would* have been run and asserting on the
argv list.

The exception is deliberate and enforced: `contract.DEVICE_OR_NETWORK_BOUND` names the 15
scripts that reach a device once their arguments parse, and those are **never executed** with
complete arguments — only `--help` and an unknown flag, which argparse answers before any
device call. That blocklist exists because a development host may well have a phone attached
(the machine this harness was written on did), and an earlier version of the CLI contract test
drove it for real. General safety net: every child runs in a private temporary working
directory, so a script that defaults an output path relative to cwd cannot write into the
shipped tree.

Anything that genuinely cannot be reached this way is not faked into a pass — it is listed as a
coverage gap in `docs/tool-verification/EXTENSION-test-harness.md`.

## Skipped is not passed

Two kinds of non-passing outcome are kept separate on purpose:

- **`skip`** - the check does not apply here. `tools/_work/bench/**` is git-ignored, so the
  `dex_patch_bytes` regression hash self-skips on a fresh checkout and in CI, printing the
  reason. A --json script that needs a sample to reach its JSON path skips too.
- **`xfail`** - the script is present and does **not** meet the repository contract. The
  reason carries the measured value, for example
  `contract says exit 2, measured 1`. These are work items, not flaky tests.

`pytest -q -ra` prints both lists. The xfail list for the CLI layer is the exit-code
unification work order; it is reproduced in `EXTENSION-test-harness.md` with owners.

## Ground rules

- **This suite never edits anything under `skills/`.** A defect found here is registered in
  the harness report. Fixing a script's exit code or quoting strategy is its owner's change,
  and doing it from a test would hide the drift the test exists to expose.
- **The bench tree is evidence and is read-only here.** The regression test copies its input
  into `tmp_path` before patching, so `tools/_work/bench/l1l3/dex/classes.patched.dex` cannot
  be overwritten by a test run.
- **No real target identity is committed.** The leak-scanner decoys are synthesised in
  `tmp_path` at run time, and token-shaped decoys are assembled from fragments so that this
  directory cannot itself become a finding.
- Fixtures in `tests/fixtures/` are small, synthetic sample files. `tests/fixtures/apk/**` is
  owned by a different work item and is not touched from here.