58 lines
15 KiB
Markdown
58 lines
15 KiB
Markdown
# Benchmark set — the regression baseline for this skill
|
||
|
||
This file is a **test matrix**, not a sample collection. It answers one question per row: *when an
|
||
agent meets this shape of target, does the route this repository documents actually work, and what
|
||
is the strength of the evidence?* Every claim in the `Result` column must be reproducible from the
|
||
`Target` column plus the named script — nothing here is a summary of what "should" happen.
|
||
|
||
**Samples are never committed.** Each row names a public source; download it into `tools/_work/bench/`
|
||
(ignored by git) before running the row. Record the sample hash when you run it, because two rows on
|
||
different sample revisions are not comparable. Hashes used in this pass:
|
||
|
||
```
|
||
UnCrackable-Level1.apk 66651 B 1da8bf57d266109f…
|
||
UnCrackable-Level2.apk 901022 B 4c7980ef1f6cc295…
|
||
UnCrackable-Level3.apk 1460555 B 6827f776b7d28443…
|
||
r2pay-v1.0.apk 4293620 B 850d7c98bed8df53…
|
||
```
|
||
|
||
Strength labels follow the repository's own three-tier rule (`references/long-task-discipline.md`
|
||
reserves `observed` for a claim with an exact command and its output behind it):
|
||
|
||
- `observed` — a command was run and its output is quoted in the evidence file.
|
||
- `inferred` — reasoned from documented mechanism or from a neighbouring measurement.
|
||
- `unverified` — assumed, or reported by someone else and not reproduced here.
|
||
|
||
| # | Dimension | Target (public source) | Scripts / documents the row exercises | Result | Strength | Evidence |
|
||
|---|---|---|---|---|---|---|
|
||
| B1 | Equal-length dex patch + repack + re-sign installability (G2/G4) | MASTG `UnCrackable-Level1.apk` | `dex_patch_bytes.py`, `repack.py`, `byte-level-patching.md`, `repack-and-sign.md` | **The route works end to end, and the change is visible on screen.** 2 bytes of the first root-check branch in `MainActivity.onCreate` (`0x9d0: 39000e00` -> `00000000`) patched inside a 5528-byte dex; the whole-file diff is those 4 bytes plus bytes 8..32, and the written file self-verifies (`adler32(d[12:])` and `sha1(d[32:])` both reproduce, proving signature-before-checksum order). `repack.py` produced a STORED + 4-byte-aligned `resources.arsc`, apksigner reported v1+v2+v3 all true, the root install path succeeded with the on-device `base.apk` sha256 equal to the local build, and the blocking `Root detected!` dialog is gone from the screenshots. A zero-change control through the identical pipeline still shows it | observed | `EXTENSION-benchmark-l1-l3.md` §2 |
|
||
| B2 | Anti-debug + self-destruct race (patch before the check fires) | MASTG `UnCrackable-Level3.apk` | `spawn_patch_detach.py`, `hook_patch_only.js`, `dynamic-frida.md` | **Split: the tooling works, the patch does not.** Time-to-death with a probe attached is 5.34 s. After an ordering fix (resume -> wait for `PATCHED` -> detach) the probe wrote `libfoo.so` base+0x2ff0/+0x2ff8/+0x3080 to `mov w0,#0 ; ret` 0.04-0.05 s after resume and the writes survived the detach — the mechanism is real. The patch still does not stop the death: the patched run **and** a read-and-write-back control both die with `SIGABRT` at `libfoo.so (goodbye()+12)` 3.22 s after resume, so neutralising the terminate routine plus both of its branch inputs is insufficient; the region is a dense cluster of `mov w0,#imm ; ret` stubs and locating the real second site needs a full `.text` disassembly. The alternative ordering ("hold paused until `PATCHED`") cannot work — a frozen process has no libraries mapped, so `Process.findModuleByName` never returns (0.11 s after resume it appears). Independently, the pristine original already reports tamper on this device, so the row's assumed clean baseline does not exist here | observed (tooling) / unverified (a surviving target) | `EXTENSION-benchmark-l1-l3.md` §3 |
|
||
| B3 | Extraction-shell skeleton detection (`trivial-body` routing) | self-built 11-variant skeleton set from the `r2pay-v1.0.apk` dex, with a zero-change control | `dex_dump_validate.py`, `advanced-unpacking.md` | **The "stub% threshold" claim this skill used to make is wrong.** `stub%` is *bimodal*, not thresholded: a real skeleton reads ≈100% (`allstub` 100.0%) while a control reads 1.9%, and the shapes real shells actually use are invisible — nop-cleared bodies 0.0% (below the control), throw stubs 1.9% (indistinguishable), partial extraction at 25/50/75% 1.7/1.7/2.1%. `advanced-unpacking.md`'s original "tens of percent means a skeleton" had no support and was deleted. Three script defects were found and fixed (all-nop bodies ranked as real; a legal minimal body mislabelled malformed; the ranking pointing at a 25%-extracted image as "most likely original" — caught by independent review). The new `minimal-form` counter *does* track the extraction ratio monotonically (1299/2524/3736/4946) and now participates in the ranking | observed | `EXTENSION-extraction-shell-bench.md` §1–3 |
|
||
| B4 | Dex-VMP boundary: refuse-to-repack and downgrade the deliverable | DexPatcher `assets/ezAndroid.apk` | `advanced-unpacking.md`, `java2c-and-jni-sinking.md`, `lsposed_scaffold.py`, `frida_rpc_serve.py` | **The sample is not a Dex VMP — it is JNI sinking**: 7 `native` declarations, each with a matching exported `Java_*` symbol, no `JNI_OnLoad` / `RegisterNatives` symbol, 34,566 instructions with zero structural errors under the official `dexdump`. A zero-change repack succeeded (v2 signing, alignment, `classes.dex` CRC verified). Static criteria **can** rule a dex-VMP *out*; they cannot rule runtime-applied hardening out. The first verdict here was the opposite — a hand-written opcode table reported 1393/22424 undecodable bodies and called it a VMP; `dexdump` showed the table was missing an arm64 branch opcode, and that rule (never declare VMP without an official disassembler's failure count) is now in the reference. Installation / launch / refusal remain untested | observed (static + repack) / unverified (install, launch) | `EXTENSION-extraction-shell-bench.md` §4 |
|
||
| B5 | Java2C recognition (do not misdiagnose as an extraction shell) | self-built with `amimo/dcc` (`--no-build`) + real JNI samples | `java2c_probe.py`, `java2c-and-jni-sinking.md` | Native-declaration density separates the shapes by ~2000x: **0.03–0.04%** on three real JNI samples vs **82.76%** on a Java2C-shaped fixture, and `Java_*` symbols matched dex native-method counts 1:1 per ABI (2:2, 3:3). **No Dex-to-C compiler output was ever built** (no NDK, no host clang, no device clang, WSL unavailable), so every Java2C-specific criterion is inferred while the JNI-sinking side is measured. A symbol search fails silently on real Java2C: dcc emits `-fvisibility=hidden`, and C++ `jni.h` inlines `RegisterNatives` so it leaves no symbol | observed (density, JNI sinking) / inferred (Java2C-specific) | `EXTENSION-java2c.md` |
|
||
| B6 | Native tracing: does `Stalker.exclude()` change the outcome? | MASTG `UnCrackable-Level1.apk`, module `libc.so` followed on the main thread for 6 s | `stalker_trace.js`, `native-dbi-and-deobfuscation.md` | **Split result.** A no-follow baseline survives; the same follow with `excludeModules: []` **kills the process**; with 20 system modules excluded it survives — so exclusion is what keeps the target alive (`observed`, with the baseline arm ruling out "frida killed it anyway"). But the treatment arm still reported `blocks=0 blk=0 calls=0`: **exclusion does not fix the zero-event trace**, which partially contradicts the community framing that lists "no events" among its symptoms. The pass also fixed two harness traps that manufacture a false zero-event result (a load-time default follow, and a suspended spawn) and verified a new zero-event `WARN`. No real OLLVM trace was produced | observed (survival, delivery) / unverified (the multiplier, the cause of zero delivery) | `EXTENSION-stalker-exclude.md` |
|
||
| B7 | Native certificate pinning | CyReverse Cronet sample | `protocol-reverse.md` | not run this pass — the CyReverse Cronet sample needs a gradle+NDK build this machine cannot do | unverified | — |
|
||
| B8 | Schemaless protobuf decoding | synthetic fixtures + the official runtime + a real AndroidX DataStore container | `protobuf_decode_raw.py`, `protocol-reverse.md` | 26/26 built-in fixtures, **21/21 against the official protobuf runtime** (re-encode byte-identical to `SerializeToString()`), 8/8 framing checks. A real 81-byte DataStore file round-tripped byte-exact and an edited value re-encoded (81->80 B, length prefix recomputed) was read back by `datastore_inject.py`. Both traps reproduced on real bytes: `0801120178` is a valid nested message *and* a valid packed array; a non-presence `0` is byte-identical to "never assigned". Four script defects were found by these checks and fixed | observed | `EXTENSION-protobuf-raw.md` |
|
||
| B9 | Split APK / App Bundle handling | two real split sets pulled from a device (8 members = base + 7 density splits each carrying its own `resources.arsc`; 9 members = base + ABI split with only `lib/` + 7 density splits with 40-byte placeholder tables) | `repack.py` (split mode), `split-apk.md` | Both branches of the decision ran on real sets: after `resign` every member shares one certificate with v1+v2+v3 all true; `merge` **correctly refused** the set whose members carry their own `resources.arsc` (exit 1, before writing anything) and **succeeded** on the code/native-only set, folding 2 `.so` files into `base`. The single-APK legacy path still passes its regression (`--dexdir` replacement and a zero-change round trip). Signing has no `uber-apk-signer` on this host; the measured call is `zipalign -p -f 4` then `apksigner.jar --v1 --v2 --v3`, and the order cannot be reversed (`[-124]`). `adb install-multiple` and launch confirmation were **not** executed | observed (analyze/resign/merge) / unverified (install, launch) | `EXTENSION-split-apk.md` |
|
||
| B10 | Kernel-side syscall answer forging | KernelSU/APatch module template, host toolchain, device kernel | `kernel-and-environment-hardening.md`, `kernelsu_syscall_mask.py` | The generator produced an 11-file userspace module skeleton plus KPM/LKM/eBPF templates and `verify` reports 0 problems — but **no kernel-side artefact was compiled or loaded**: the `<DEVICE>` kernel is 4.14.186 (eBPF needs 5.10+), the host has no `aarch64-none-elf-gcc` / `make` / `ndk-build`, and there is no kernel source. Documentation correction with teeth: **an ordinary KernelSU module is a userspace module and structurally cannot change a syscall return value** | observed (generation) / unverified (kernel side) | `EXTENSION-kernel-weapons.md` |
|
||
| B11 | JNI dynamic registration discovery (`RegisterNatives` with no `Java_*` symbol) | a real system library + CyReverse `plugin/` | `java2c-and-jni-sinking.md`, `java2c_probe.py` | A real dynamically-registering library contains **zero `RegisterNatives` symbols** (the C++ `jni.h` inlines the call), so symbol-based discovery fails structurally. The combination that did fire on a real library is "**exports `JNI_OnLoad` and zero `Java_*`**" | observed | `EXTENSION-java2c.md` |
|
||
| B12 | Dart AOT string-table format across ABIs (the premise any string patch rests on) | a locally supplied hardened Flutter APK (360 shell + AOT payload, both ABIs; uncommitted, `tools/_work/dart/`) | `dart-aot.md` §7/§10, `scripts/dart_pool_strings.py` | **Two claims tested, both were wrong before this row.** (a) arm64 packed `[0x80\|(len<<1)][payload]` is confirmed — `dart_pool_strings.py` keeps 4,980 chained entries and the tag byte equals `0x80\|(len<<1)` at every literal checked (`AdmobBanner` 11 chars → `0x96`, `showSplashAd` and `AdPgl_Splash` 12 → `0x98`). (b) The armv7 form "`[len*2 u32le][UTF-16LE]`" is **refuted**: the same extractor keeps **0** entries on the 32-bit snapshot, and the byte layer shows why — each 4-byte-aligned record reads `[header u32][byte-count u32le][UTF-8 payload]`, with the two adjacent literals `SystemChrome.setEnabledSystemUIMode` (36) + `SystemChrome.setSystemUIOverlayStyle` (34) carrying lengths `0x46` (70) and `0x48` (72) that exactly equal their byte lengths. The extractor's zero is a format mismatch, not an empty string table. A third claim, "the engine banner gives `x.y.z (stable)`", is also refuted on this target: the banner is 66 bytes of `0.0.1` with a blank build id. Full detail and the raw probes: `EXTENSION-dart-aot-formats.md` | observed (both formats, banner) / unverified (an end-to-end device patch — no repack+install was run) | `EXTENSION-dart-aot-formats.md` |
|
||
| B13 | Naming the check that fires, and the environment it reads (anti-instrumentation triage) | MASTG `owasp.mstg.uncrackable3` (public target) for the three arms; device `libc.so` and a hardened shell library for `svc_scan.py` | `anti_detect_probe.js`, `detection-and-anti-analysis.md` Step 3, `svc_scan.py` | **The check was named, and the asymmetry is the finding.** Three arms on one target with a single variable: control (`spawn` + `resume`, no script) ends `alive_at_end=True`; the arm that also loads the observer probe ends `alive_at_end=False`, with the stream showing `strstr("frida")` at t=303 ms and, in an earlier same-shape arm, the kill itself at t=352 ms — process gone with **no tombstone, no crash and no ANR record**. The probe's own self-report on the same arm: `TracerPid=0` while the target's `/proc/self/maps` carries **4 frida-named mappings**, so a `TracerPid`-only detector passes and a maps/thread-name detector does not. Reproducibility is honestly bounded: 3 same-shape arms, 1 delivered a full sequence (a ≈300 ms self-destruct leaves a sub-second window). Separately, `svc_scan.py`'s two independent decoders — a hand-written word scan and capstone — produced an **identical 214-site set** (set difference empty), and a structural criterion fell out: the device `libc.so`'s 4 termination sites are libc's own exports (so a libc-level hook is structurally able to see them), while the hardened shell library's 21 `svc` sites are **all data** (neighbours are `udf`/NEON filler) with no termination site | observed (the arms, the named check, decode agreement) / unverified (dual-dumper byte agreement — no second producer existed; a dump from the hardened sample) | `EXTENSION-detection-pipeline.md` §1, §4, §5 |
|
||
|
||
## How to read a row
|
||
|
||
`Result` states what actually happened, including negative outcomes — "the route worked", "the route
|
||
worked only with `Stalker.exclude()`", "the tool refused to run at all". A negative result that is
|
||
recorded is worth more than a positive one that is assumed, because the next agent can route around
|
||
it without re-paying the cost.
|
||
|
||
A row whose `Strength` is `unverified` is a statement about *this repository's evidence*, not about
|
||
the mechanism's truth. It means: nobody has yet put a command and its output behind it here.
|
||
|
||
**Rows that changed a conclusion this repository had already recorded.** This is the outcome the
|
||
matrix exists to produce, so it is listed rather than buried: B3 (the trivial-body "threshold" became
|
||
a measured bimodal result), B4 (a VMP verdict was withdrawn after an official disassembler disagreed
|
||
with a hand-written opcode table), B6 (exclusion turns out to keep the process alive but does not fix
|
||
zero-event delivery, contradicting the community framing), and B1 (an earlier "`repack.py` has no
|
||
apksigner signing path" conclusion expired mid-pass when a second writer added one).
|