reverse-skills-max/vendor-new/apk-reverse/tests/benchmark.md

Benchmark set — the regression baseline for this skill

This file is a test matrix, not a sample collection. It answers one question per row: when an agent meets this shape of target, does the route this repository documents actually work, and what is the strength of the evidence? Every claim in the Result column must be reproducible from the Target column plus the named script — nothing here is a summary of what "should" happen.

Samples are never committed. Each row names a public source; download it into tools/_work/bench/ (ignored by git) before running the row. Record the sample hash when you run it, because two rows on different sample revisions are not comparable. Hashes used in this pass:

UnCrackable-Level1.apk   66651 B  1da8bf57d266109f…
UnCrackable-Level2.apk  901022 B  4c7980ef1f6cc295…
UnCrackable-Level3.apk 1460555 B  6827f776b7d28443…
r2pay-v1.0.apk         4293620 B  850d7c98bed8df53…

Strength labels follow the repository's own three-tier rule (references/long-task-discipline.md reserves observed for a claim with an exact command and its output behind it):

• observed — a command was run and its output is quoted in the evidence file.
• inferred — reasoned from documented mechanism or from a neighbouring measurement.
• unverified — assumed, or reported by someone else and not reproduced here.

#	Dimension	Target (public source)	Scripts / documents the row exercises	Result	Strength	Evidence
B1	Equal-length dex patch + repack + re-sign installability (G2/G4)	MASTG UnCrackable-Level1.apk	dex_patch_bytes.py, repack.py, byte-level-patching.md, repack-and-sign.md	The route works end to end, and the change is visible on screen. 2 bytes of the first root-check branch in MainActivity.onCreate (0x9d0: 39000e00 -> 00000000) patched inside a 5528-byte dex; the whole-file diff is those 4 bytes plus bytes 8..32, and the written file self-verifies (adler32(d[12:]) and sha1(d[32:]) both reproduce, proving signature-before-checksum order). repack.py produced a STORED + 4-byte-aligned resources.arsc, apksigner reported v1+v2+v3 all true, the root install path succeeded with the on-device base.apk sha256 equal to the local build, and the blocking Root detected! dialog is gone from the screenshots. A zero-change control through the identical pipeline still shows it	observed	EXTENSION-benchmark-l1-l3.md §2
B2	Anti-debug + self-destruct race (patch before the check fires)	MASTG UnCrackable-Level3.apk	spawn_patch_detach.py, hook_patch_only.js, dynamic-frida.md	Split: the tooling works, the patch does not. Time-to-death with a probe attached is 5.34 s. After an ordering fix (resume -> wait for PATCHED -> detach) the probe wrote libfoo.so base+0x2ff0/+0x2ff8/+0x3080 to mov w0,#0 ; ret 0.04-0.05 s after resume and the writes survived the detach — the mechanism is real. The patch still does not stop the death: the patched run and a read-and-write-back control both die with SIGABRT at libfoo.so (goodbye()+12) 3.22 s after resume, so neutralising the terminate routine plus both of its branch inputs is insufficient; the region is a dense cluster of mov w0,#imm ; ret stubs and locating the real second site needs a full .text disassembly. The alternative ordering ("hold paused until PATCHED") cannot work — a frozen process has no libraries mapped, so Process.findModuleByName never returns (0.11 s after resume it appears). Independently, the pristine original already reports tamper on this device, so the row's assumed clean baseline does not exist here	observed (tooling) / unverified (a surviving target)	EXTENSION-benchmark-l1-l3.md §3
B3	Extraction-shell skeleton detection (trivial-body routing)	self-built 11-variant skeleton set from the r2pay-v1.0.apk dex, with a zero-change control	dex_dump_validate.py, advanced-unpacking.md	The "stub% threshold" claim this skill used to make is wrong. stub% is bimodal, not thresholded: a real skeleton reads ≈100% (allstub 100.0%) while a control reads 1.9%, and the shapes real shells actually use are invisible — nop-cleared bodies 0.0% (below the control), throw stubs 1.9% (indistinguishable), partial extraction at 25/50/75% 1.7/1.7/2.1%. advanced-unpacking.md's original "tens of percent means a skeleton" had no support and was deleted. Three script defects were found and fixed (all-nop bodies ranked as real; a legal minimal body mislabelled malformed; the ranking pointing at a 25%-extracted image as "most likely original" — caught by independent review). The new minimal-form counter does track the extraction ratio monotonically (1299/2524/3736/4946) and now participates in the ranking	observed	EXTENSION-extraction-shell-bench.md §1–3
B4	Dex-VMP boundary: refuse-to-repack and downgrade the deliverable	DexPatcher assets/ezAndroid.apk	advanced-unpacking.md, java2c-and-jni-sinking.md, lsposed_scaffold.py, frida_rpc_serve.py	The sample is not a Dex VMP — it is JNI sinking: 7 native declarations, each with a matching exported Java_* symbol, no JNI_OnLoad / RegisterNatives symbol, 34,566 instructions with zero structural errors under the official dexdump. A zero-change repack succeeded (v2 signing, alignment, classes.dex CRC verified). Static criteria can rule a dex-VMP out; they cannot rule runtime-applied hardening out. The first verdict here was the opposite — a hand-written opcode table reported 1393/22424 undecodable bodies and called it a VMP; dexdump showed the table was missing an arm64 branch opcode, and that rule (never declare VMP without an official disassembler's failure count) is now in the reference. Installation / launch / refusal remain untested	observed (static + repack) / unverified (install, launch)	EXTENSION-extraction-shell-bench.md §4
B5	Java2C recognition (do not misdiagnose as an extraction shell)	self-built with amimo/dcc (--no-build) + real JNI samples	java2c_probe.py, java2c-and-jni-sinking.md	Native-declaration density separates the shapes by ~2000x: 0.03–0.04% on three real JNI samples vs 82.76% on a Java2C-shaped fixture, and Java_* symbols matched dex native-method counts 1:1 per ABI (2:2, 3:3). No Dex-to-C compiler output was ever built (no NDK, no host clang, no device clang, WSL unavailable), so every Java2C-specific criterion is inferred while the JNI-sinking side is measured. A symbol search fails silently on real Java2C: dcc emits -fvisibility=hidden, and C++ jni.h inlines RegisterNatives so it leaves no symbol	observed (density, JNI sinking) / inferred (Java2C-specific)	EXTENSION-java2c.md
B6	Native tracing: does Stalker.exclude() change the outcome?	MASTG UnCrackable-Level1.apk, module libc.so followed on the main thread for 6 s	stalker_trace.js, native-dbi-and-deobfuscation.md	Split result. A no-follow baseline survives; the same follow with excludeModules: [] kills the process; with 20 system modules excluded it survives — so exclusion is what keeps the target alive (observed, with the baseline arm ruling out "frida killed it anyway"). But the treatment arm still reported blocks=0 blk=0 calls=0: exclusion does not fix the zero-event trace, which partially contradicts the community framing that lists "no events" among its symptoms. The pass also fixed two harness traps that manufacture a false zero-event result (a load-time default follow, and a suspended spawn) and verified a new zero-event WARN. No real OLLVM trace was produced	observed (survival, delivery) / unverified (the multiplier, the cause of zero delivery)	EXTENSION-stalker-exclude.md
B7	Native certificate pinning	CyReverse Cronet sample	protocol-reverse.md	not run this pass — the CyReverse Cronet sample needs a gradle+NDK build this machine cannot do	unverified	—
B8	Schemaless protobuf decoding	synthetic fixtures + the official runtime + a real AndroidX DataStore container	protobuf_decode_raw.py, protocol-reverse.md	26/26 built-in fixtures, 21/21 against the official protobuf runtime (re-encode byte-identical to SerializeToString()), 8/8 framing checks. A real 81-byte DataStore file round-tripped byte-exact and an edited value re-encoded (81->80 B, length prefix recomputed) was read back by datastore_inject.py. Both traps reproduced on real bytes: 0801120178 is a valid nested message and a valid packed array; a non-presence 0 is byte-identical to "never assigned". Four script defects were found by these checks and fixed	observed	EXTENSION-protobuf-raw.md
B9	Split APK / App Bundle handling	two real split sets pulled from a device (8 members = base + 7 density splits each carrying its own resources.arsc; 9 members = base + ABI split with only lib/ + 7 density splits with 40-byte placeholder tables)	repack.py (split mode), split-apk.md	Both branches of the decision ran on real sets: after resign every member shares one certificate with v1+v2+v3 all true; merge correctly refused the set whose members carry their own resources.arsc (exit 1, before writing anything) and succeeded on the code/native-only set, folding 2 .so files into base. The single-APK legacy path still passes its regression (--dexdir replacement and a zero-change round trip). Signing has no uber-apk-signer on this host; the measured call is zipalign -p -f 4 then apksigner.jar --v1 --v2 --v3, and the order cannot be reversed ([-124]). adb install-multiple and launch confirmation were not executed	observed (analyze/resign/merge) / unverified (install, launch)	EXTENSION-split-apk.md
B10	Kernel-side syscall answer forging	KernelSU/APatch module template, host toolchain, device kernel	kernel-and-environment-hardening.md, kernelsu_syscall_mask.py	The generator produced an 11-file userspace module skeleton plus KPM/LKM/eBPF templates and verify reports 0 problems — but no kernel-side artefact was compiled or loaded: the <DEVICE> kernel is 4.14.186 (eBPF needs 5.10+), the host has no aarch64-none-elf-gcc / make / ndk-build, and there is no kernel source. Documentation correction with teeth: an ordinary KernelSU module is a userspace module and structurally cannot change a syscall return value	observed (generation) / unverified (kernel side)	EXTENSION-kernel-weapons.md
B11	JNI dynamic registration discovery (RegisterNatives with no Java_* symbol)	a real system library + CyReverse plugin/	java2c-and-jni-sinking.md, java2c_probe.py	A real dynamically-registering library contains zero RegisterNatives symbols (the C++ jni.h inlines the call), so symbol-based discovery fails structurally. The combination that did fire on a real library is "exports JNI_OnLoad and zero Java_*"	observed	EXTENSION-java2c.md
B12	Dart AOT string-table format across ABIs (the premise any string patch rests on)	a locally supplied hardened Flutter APK (360 shell + AOT payload, both ABIs; uncommitted, tools/_work/dart/)	dart-aot.md §7/§10, scripts/dart_pool_strings.py	Two claims tested, both were wrong before this row. (a) arm64 packed [0x80|(len<<1)][payload] is confirmed — dart_pool_strings.py keeps 4,980 chained entries and the tag byte equals 0x80|(len<<1) at every literal checked (AdmobBanner 11 chars → 0x96, showSplashAd and AdPgl_Splash 12 → 0x98). (b) The armv7 form "[len*2 u32le][UTF-16LE]" is refuted: the same extractor keeps 0 entries on the 32-bit snapshot, and the byte layer shows why — each 4-byte-aligned record reads [header u32][byte-count u32le][UTF-8 payload], with the two adjacent literals SystemChrome.setEnabledSystemUIMode (36) + SystemChrome.setSystemUIOverlayStyle (34) carrying lengths 0x46 (70) and 0x48 (72) that exactly equal their byte lengths. The extractor's zero is a format mismatch, not an empty string table. A third claim, "the engine banner gives x.y.z (stable)", is also refuted on this target: the banner is 66 bytes of 0.0.1 with a blank build id. Full detail and the raw probes: EXTENSION-dart-aot-formats.md	observed (both formats, banner) / unverified (an end-to-end device patch — no repack+install was run)	EXTENSION-dart-aot-formats.md
B13	Naming the check that fires, and the environment it reads (anti-instrumentation triage)	MASTG owasp.mstg.uncrackable3 (public target) for the three arms; device libc.so and a hardened shell library for svc_scan.py	anti_detect_probe.js, detection-and-anti-analysis.md Step 3, svc_scan.py	The check was named, and the asymmetry is the finding. Three arms on one target with a single variable: control (spawn + resume, no script) ends alive_at_end=True; the arm that also loads the observer probe ends alive_at_end=False, with the stream showing strstr("frida") at t=303 ms and, in an earlier same-shape arm, the kill itself at t=352 ms — process gone with no tombstone, no crash and no ANR record. The probe's own self-report on the same arm: TracerPid=0 while the target's /proc/self/maps carries 4 frida-named mappings, so a TracerPid-only detector passes and a maps/thread-name detector does not. Reproducibility is honestly bounded: 3 same-shape arms, 1 delivered a full sequence (a ≈300 ms self-destruct leaves a sub-second window). Separately, svc_scan.py's two independent decoders — a hand-written word scan and capstone — produced an identical 214-site set (set difference empty), and a structural criterion fell out: the device libc.so's 4 termination sites are libc's own exports (so a libc-level hook is structurally able to see them), while the hardened shell library's 21 svc sites are all data (neighbours are udf/NEON filler) with no termination site	observed (the arms, the named check, decode agreement) / unverified (dual-dumper byte agreement — no second producer existed; a dump from the hardened sample)	EXTENSION-detection-pipeline.md §1, §4, §5

How to read a row

Result states what actually happened, including negative outcomes — "the route worked", "the route worked only with Stalker.exclude()", "the tool refused to run at all". A negative result that is recorded is worth more than a positive one that is assumed, because the next agent can route around it without re-paying the cost.

A row whose Strength is unverified is a statement about this repository's evidence, not about the mechanism's truth. It means: nobody has yet put a command and its output behind it here.

Rows that changed a conclusion this repository had already recorded. This is the outcome the matrix exists to produce, so it is listed rather than buried: B3 (the trivial-body "threshold" became a measured bimodal result), B4 (a VMP verdict was withdrawn after an official disassembler disagreed with a hand-written opcode table), B6 (exclusion turns out to keep the process alive but does not fix zero-event delivery, contradicting the community framing), and B1 (an earlier "repack.py has no apksigner signing path" conclusion expired mid-pass when a second writer added one).