# EXTENSION — executable test harness, CI, and the exit-code work order This file records what the regression harness in `tests/` actually does, what it measured on this machine, and what it does **not** cover. Strength labels follow the repository rule in `references/long-task-discipline.md`: `observed` means an exact command and its output are below, `inferred` means reasoned from a neighbouring measurement, `unverified` means nobody has put a command behind it here. The distinction this file exists to keep honest: `tests/benchmark.md` is a **record** of routes run once by hand against real samples and a real device. `tests/` is a **gate** that runs on every commit with the standard library and pytest and nothing else. Neither replaces the other, and a green suite is not evidence that any route works on a hardened target. --- ## 1. How to run it, and what it printed ``` $ python -m pytest -q 596 passed, 153 skipped, 26 xfailed in 49.67s observed $ python -m pytest -q -m unit 163 passed, 612 deselected in 4.50s observed $ python -m pytest -q -m cli 342 passed, 153 skipped, 255 deselected, 25 xfailed in 38.26s observed $ python -m pytest -q -m integration 65 passed, 709 deselected, 1 xfailed in 1.33s observed $ python tests/run_tests.py -q -m unit 163 passed, 612 deselected in 5.08s observed ``` `tests/run_tests.py` is the one-command entry point. It refuses to fail obscurely when pytest is absent — measured by putting a `pytest.py` that raises `ImportError` first on `PYTHONPATH`: ``` $ python tests/run_tests.py pytest is not installed; this suite cannot run. install: python -m pip install pytest the suite is stdlib-only otherwise (no device, no network, no adb) exit=3 observed ``` Exit 3 is the repository's "environment capability insufficient" code, so a wrapper can tell "no pytest here" apart from "tests failed". The suite is stdlib-only apart from pytest itself. CI targets Python 3.11 and 3.13; this machine measured on 3.14.0 with pytest 8.4.1. --- ## 2. Coverage | Area | What runs | Strength | |---|---|---| | dex format table | `insn_units` asserted opcode by opcode against 29 named traps (payload widths, `0x32`-`0x3D` being two units, `const-string` vs `/jumbo`, the `goto` family at `0x28`-`0x2A`); `OP_UNITS` and the accessor must agree for every listed opcode; unallocated opcodes still terminate | observed | | dex header | magic/`file_size`/table-pointer checks against a synthesised consistent header; the verified offset table asserted by value; signature-before-checksum order, including the failure mode where a checksum taken over a zeroed signature never verifies | observed | | branch arithmetic | `if-test`/`if-testz`/`goto`/`goto16`/`goto32` targets and register extraction, on a synthetic buffer | observed | | the sample regression | `dex_patch_bytes.py` + `l1_root.spec.json` → sha256 `67ae1e58…` and byte-identical to `classes.patched.dex`; the diff window is pinned (see §6) | observed | | scheme-less protobuf | built-in fixture set 26/26, per-fixture expectations, decode→re-encode byte-identical for every canonical fixture, the non-canonical fixture required to *report* its mismatch, both wire-format traps | observed | | blob rotation search | outer encodings round-trip, raw/zlib/gzip identified by name, the rotation and prefix (`skip`) parameters recovered from a payload built with known parameters | observed | | leak scanner | synthesised decoys in a temporary directory are reported with the right rules and strengths; the exemption list (`com.android.*`, `com.google.*`, `org.example.*`, ``, loopback) produces no finding; an RFC 5737 address stays `weak` and keeps the default gate green while `--fail-on any` fails it; JSON report keys and the last-line `RESULT=` token | observed | | CLI contract | every shipped script, four invocation shapes where they apply: `--help`, unknown flag, no arguments, all inputs pointed at a missing file | observed | | adb argv construction | `devsh.py`, `coldstart.Device`, `install_test.adb` build an **argv list**; `\|`, `>`, `$()` and quotes survive as literal text inside one argument; serial injection as `-s`; env-var precedence | observed | | screenshot validation | `coldstart.Device.screencap` judges the bytes on disk: PNG magic accepted, non-PNG rejected, a sub-8-byte body rejected, a timeout reported as a failed capture | observed | | launch-health verdict | `install_test.main` driven with a scripted `pidof`: stable pid + no fatal signature is the only pass; a changed pid, a dead app, or a stable pid with a `FATAL EXCEPTION` line all fail; two samples are actually taken | observed | | log/tombstone parsing | `coldstart.signal_counts` over a logcat fixture (9 patterns, exact counts); `native_crash.parse` signal/code/pid/uptime/frames/registers; `is_system_lib` classification; the `verdict_hint` for an arranged fault | observed | --- ## 3. Not covered — the honest gaps ### 3.1 Device work: excluded by design, because hardware is present This harness must not touch a device, and after one measured incident that is now enforced rather than assumed. It has to be: `adb devices` on the machine this was written on reports an online device, so a naive "run every script with its arguments filled in" test drives real hardware. ``` $ adb devices device observed ``` **What happened.** An early version of the CLI contract test executed `coldstart.py` with its requirements filled in. The script found the single online device, adopted it, started `adb logcat -v threadtime`, wrote a 927,893-byte `logcat.txt`, captured four frames, and left **five orphaned `adb logcat` processes** behind when the 12 s timeout killed the parent. The whole mess landed in `skills/apk-reverse/scripts/`, which is another owner's write scope. **What was cleaned up.** The five orphaned `adb logcat` processes were terminated, and the directory the script created (`skills/apk-reverse/scripts/coldstart/`, containing `logcat.txt` and `shots/f0*.png`) was removed. Verified: `git status` on `skills/apk-reverse/scripts` no longer lists it. **What changed in the design.** - Every child process now runs in a private temporary working directory, so a script that defaults an output path relative to cwd cannot write into the shipped tree. This is the general fix; the incident was one instance of the class. - `contract.DEVICE_OR_NETWORK_BOUND` — 15 scripts — is never executed with complete arguments. `--help` and an unknown flag are still exercised for all of them, because argparse answers both before any device call is reached, and that was verified for every script individually: `--help` returned in well under a second and an unknown flag returned 2 for each. **Residual risk, stated plainly.** This is a blocklist, not a sandbox. A script not on the list that grows device access would be executed. The list is checked against the tree on every run (a script that appears without being classified becomes a new pass or a new xfail rather than a silent gap), but it is maintained by hand. Fifteen scripts are device- or network-bound and their argument-complete behaviour is `unverified` here. What that means concretely: this harness cannot tell you that a repack installs, that an anti-debug patch survives, that a hook fires, or that a tracer reports blocks. Those live in `tests/benchmark.md`, and the `unverified` marks there are unchanged by anything in this directory. ### 3.2 CI has never executed: `unverified` `.github/workflows/ci.yml` exists and is syntactically valid — verified locally: ``` $ python -c "import yaml,io; yaml.safe_load(io.open('.github/workflows/ci.yml',encoding='utf-8'))" ci.yml: valid YAML observed ``` and every command it runs was executed by hand on this machine: ``` $ python check_refs.py exit 0 observed $ python check_budget.py exit 0 ("0 failure(s), 0 warning(s), 3 note(s)") observed $ python check_repo.py exit 0 (with PYTHONUTF8=1; see §5.1) observed $ python -m pytest -q exit 0 ("596 passed, 153 skipped, 26 xfailed") observed ``` It is still `unverified` as a workflow, and the blocker is specific: there is no GitHub Actions runner reachable from this machine, the workflow has not been pushed (this work item is forbidden from committing), and no local runner (`act`) is installed. What cannot be claimed until the first push: that `actions/setup-python` resolves both matrix versions, that the gates behave identically under a Linux UTF-8 locale, and that the suite's timing stays inside any runner limit. The last two matter because the gates behave *differently* on this Windows host — see §5.1 — and because the CLI suite spawns roughly 200 subprocesses. A remote does exist (`origin` → `github.com/newliver666/apk-reverse`), so the workflow will run on the first push; nobody has observed it do so. ### 3.3 Agent behaviour: `unverified`, and deliberately out of scope `skills/apk-reverse/evals/evals.json` is a skeleton: three cases with expected outputs and grading assertions, and an explicit statement **in the file** that nothing in it has ever been executed. No with-skill arm and no without-skill arm has been run. Running them means running the agent twice per case and grading transcripts, which was excluded from this pass by decision (`REBUILD-BRIEF.md` §2, the do-not-do list). The file carries its own run procedure. ### 3.4 The 153 skips, by category | Count | Reason | Reading | |---|---|---| | 60 | device/network bound, not executed | 15 scripts × 4 scenarios. Deliberate; see §3.1. | | 40 | requires no argument, so the contract does not apply | 10 scripts × 4 scenarios. Requiring exit 2 from `doctor.py` or `scan_leaks.py` with no arguments would invent a rule the contract does not state. | | 34 | does not advertise `--json` | the 34 of 46 scripts with no `--json` flag. That is the §1.6 problem, not a harness gap. | | 11 | `--json` advertised but the path needs a device or a sample | includes `capabilities.py`, added during this pass. The shape is not asserted because the run that would reach it needs a target. | | 8 | hand-written usage, argv not derivable | `dexutil.py` and `smtool.py`, 4 scenarios each. Their usage line is prose, so the missing-input argv is not machine-derivable. `devsh.py` and `patch_smali.py` share the trait but are already covered by row 1. | Total 153, which is why the headline is "passed, skipped, xfailed" and not "passed". --- ## 4. The exit-code work order — 26 xfails These are not flaky tests. Each is a script on the tree that does not meet the repository contract in `REBUILD-BRIEF.md` §4 (`0` success, `1` target result failed, `2` bad arguments/input, `3` environment capability missing, `4` internal error). The xfail reason carries the measured value, so `pytest -q -ra` prints the work order at any time. **A. `--help` does not exit 0 — 4 scripts.** A hand-written parser rather than argparse, so `--help` falls through to "unknown argument". None of them prints a traceback, so this is a contract gap and not a crash. `devsh.py`, `dexutil.py`, `patch_smali.py`, `smtool.py` — measured `2`, expected `0`. **B. A missing input file exits 1 instead of 2 — 13 scripts.** The distinction matters to a calling agent: 1 means "your target failed", 2 means "your invocation was wrong". Reporting a typo in a path as a target failure sends the next agent to debug the sample. `apk_diff.py`, `dart_disasm.py`, `dart_pool_strings.py`, `dex_classdiff.py`, `dex_dump_validate.py`, `dex_find_insn.py`, `dex_mem_scan.py`, `dex_strpatch.py`, `elf_plt.py`, `native_crash.py`, `patch_smali.py`, `stalker_report.py`, `svc_scan.py` — measured `1`, expected `2`. **C. An uncaught exception instead of a clean report — 7 scripts.** The user-visible difference: a traceback instead of a sentence. `dart_pool_strings.py`, `dex_classdiff.py`, `dex_find_insn.py`, `dex_mem_scan.py`, `dex_strpatch.py`, `patch_smali.py`, `stalker_report.py` — all `FileNotFoundError` (on Windows, `dex_mem_scan.py` reports `WinError 3`). **D. `--json` without a `status` field — 1 script.** `preflight.py --json` returns top-level keys `rows` and `worst`, with no `status`. Per §4 the document must carry `status` (and ideally `exit_code`, `capability`, `evidence`, `warnings`, `next_action`). For contrast, `doctor.py --json` was measured during this pass and *does* carry `status`, and now exits 3 rather than 0 when a capability is missing — the unified convention is arriving script by script. **E. `su` quoting — 1 item, outside the exit-code list.** `devsh.su` wraps in double quotes and escapes only `"`, so `;`, `$()`, backticks and `|` from a `--pkg`-style value still reach the device shell. `coldstart.Device.shell` uses the POSIX single-quote escape and does neutralise them. Two helpers, two strategies — the `device_shell()` unification item. The contrast is asserted in `tests/integration/test_adb_command_construction.py::TestDevshSuQuotingGap`. Nothing in this list was fixed by this work item: every one of those scripts belongs to another owner, and a test that quietly repaired its subject would stop being a test. --- ## 5. Defects found and registered, not fixed ### 5.1 `check_repo.py` misreports a script as crashing under a non-UTF-8 console (`observed`) ``` $ python check_repo.py - E:\apk-reverse\skills\apk-reverse\scripts\dex_strpatch.py --help crashed: == result: 1 problem(s) == exit=1 ``` with, above it, `UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 63` raised inside `subprocess`'s reader thread. The gate spawns each script with `text=True` and no explicit encoding, so on this host the child's output is decoded with GBK. One script's `--help` text is not GBK-decodable, and the gate attributes the failure to that script. The script itself is fine: ``` $ python skills/apk-reverse/scripts/dex_strpatch.py --help ; echo $? 0 observed ``` Pinning the codec fixes it: ``` $ PYTHONUTF8=1 python check_repo.py == result: 0 problem(s) == exit=0 observed ``` `PYTHONUTF8: "1"` is therefore set in `ci.yml`. Two caveats: CI runs on Linux, where the default codec is already UTF-8, so this is a portability fix for anyone running the gates on Windows rather than a fix for the CI run; and `check_repo.py` is in the Lead's write scope, so the underlying `text=True` call is not changed here. ### 5.2 `native_crash.py` overwrites `regs['pc']` with a backtrace frame (`observed`) Register extraction scans every line of a crash block with one permissive pattern, so a `#NN pc ` backtrace line matches too and the last one wins. On the fixture, the register dump's own `pc 00000000007a111128cc` is not what `regs['pc']` holds; `0x55488` — the last frame's pc — is. The verdict is unaffected (the arranged-fault hint keys off the fault address and the registers holding it, and neither is `pc`), but a reader who trusts `regs['pc']` as the faulting instruction pointer is misled. Pinned in `tests/integration/test_no_device_flows.py::TestTombstoneParsing`. ### 5.3 `coldstart.installed_facts` `versionCode` swallows the rest of the line (`observed`) The extraction is `r'\s*(versionCode|...)=(.*)'`, so on a real `dumpsys package` line — `versionCode=42 minSdk=24 targetSdk=34` — the value is the whole remainder, not `42`. It still works as the before/after fingerprint the caller uses it for, but the field name overstates what it holds. Pinned in `tests/integration/test_adb_command_construction.py`. ### 5.4 The leak scanner reports one `ip:port` twice (`observed`) `192.0.2.10:8080` satisfies both `plain_addr` (`ip:port`) and `plain_addr_ipliteral` (bare IPv4), because the literal rule's trailing `(?![\w.])` is not violated by a colon. Both hits are `weak`, so the default gate stays green; the effect is report volume, not a wrong verdict. Pinned, with the reasoning, in `tests/unit/test_scan_leaks.py`. ### 5.5 `usb_net_proxy.py` does something with no arguments (`observed`) With no arguments it neither prints usage nor exits: it ran past both a 12 s and a 20 s timeout during probing. Every other script either requires arguments or prints its own documentation. Not diagnosed further here — the fix belongs to whoever owns that script. It is classified as device/network bound, so the suite does not execute it. ### 5.6 Stray artifacts in the tree that this work item did not create (`observed`) Found while checking its own side effects, left in place because they are not this work item's to delete and may belong to a concurrent one: - `--help/` at the repository root, containing `planted/{package,device,appkey,endpoint,path,token}.txt` and `exempt/benign.txt`. The names say "leak-scanner decoy set", so some scanner test wrote its fixtures into a directory literally named `--help` — most likely an argument-parsing bug in that test, not a scan result. - `skills/apk-reverse/scripts/example.preferences_pb` and `skills/apk-reverse/scripts/snapshots/` (containing `snap.png`, 3.97 MB, and `_snap_ui.xml`). Timestamps 01:17 and 01:18, unchanged across this work item's later runs, so they are not its output. --- ## 6. The established regression marker The red line recorded in `REBUILD-BRIEF.md` §2 is reproduced by the suite rather than re-invented: ``` $ python skills/apk-reverse/scripts/dex_patch_bytes.py \ tools/_work/bench/l1l3/dex/classes.dex \ --spec tools/_work/bench/l1l3/l1_root.spec.json \ -o /classes.patched.dex == header checksum 0x7fb7d9fa -> 0x86e1d928 signature b7fafe72cb521450 -> eb632328fd1ba03c self-verify: checksum_ok=True signature_ok=True == wrote /classes.patched.dex (5528 bytes, delta 0) $ sha256sum /classes.patched.dex 67ae1e58e7a44777e5ac8f9f823ab21e55e56c6122d2d64c3627232d902a2bba observed ``` Identical to the stored `tools/_work/bench/l1l3/dex/classes.patched.dex`, byte for byte. The suite asserts the hash **and** where the difference is allowed to be: bytes 8..32 (checksum and signature, recomputed) plus the four bytes at `0x9d0`. Two of those four keep their original value — `39000e00` → `00000000` clears two bytes — and a recomputed checksum can keep a byte by coincidence, so the assertion is containment plus "these must have moved", not equality of the diff set. That is the honest form: an equality assertion would fail on a correct build. The test copies the bench input into `tmp_path` before patching, so a test run cannot overwrite the stored expectation. `tools/_work/bench/**` is git-ignored, so on a fresh checkout and in CI this test **skips** with the printed reason instead of turning red. A regression marker that fails because a sample is absent is a marker people learn to ignore. --- ## 7. Files added by this work item ``` pytest.ini repository root tests/README.md how to run it, and the ground rules tests/run_tests.py entry point; exits 3 without pytest tests/conftest.py fixtures, subprocess runner, isolated cwd tests/unit/test_dexutil.py tests/unit/test_protobuf_decode_raw.py tests/unit/test_blob_decode.py tests/unit/test_scan_leaks.py tests/unit/test_dex_patch_bytes_regression.py tests/cli/contract.py usage parsing and the contract constants tests/cli/test_cli_contract.py tests/integration/test_adb_command_construction.py tests/integration/test_no_device_flows.py tests/fixtures/tombstone_sample.txt synthetic, 1.3 KB tests/fixtures/logcat_sample.txt synthetic, 1.0 KB .github/workflows/ci.yml skills/apk-reverse/evals/evals.json skeleton, never executed ``` Not touched: `skills/apk-reverse/scripts/**`, `tests/test_so_constpatch.py`, `tests/fixtures/apk/**`, `SKILL.md`, `README.md`, the root `check_*.py` scripts. No commit, no push. The only write this work item made inside the shipped tree was its own accident (§3.1), and it was reverted. Fixture budget: two text fixtures, 2.3 KB total, well under the 32 KB per-file limit. The leak decoys are synthesised at run time and are not committed.