21 KiB
Extension verification — schema-free protobuf decoding
Covers the pass that added skills/apk-reverse/scripts/protobuf_decode_raw.py and the measured half
of references/protocol-reverse.md §1 ("The decoder that ships here"). Strength labels as defined in
this directory's README: observed (exact command + output recorded here), inferred (follows
from observed facts, step not executed), unverified (assumed or externally reported, not
independently confirmed here).
Reference environment for this pass: Windows host, Python 3.14.0, protobuf 6.33.6. No device
was used (see §7) — every result below is host-side, offline, and reproducible with the standard
library alone. Workbench artifacts live in tools/_work/t5-proto/ (not part of the skill, and
tools/ is gitignored).
The delivered script imports no third-party module. google.protobuf is used here for one
purpose only: to generate reference bytes from a real schema, so the hand-written decoder is
checked against the reference implementation instead of against itself.
0. Summary of what this pass established
| Claim | Label |
|---|---|
protobuf_decode_raw.py --help, --selftest, hex/file/stdin inputs, --json, --max-depth, --reencode all run on this host |
observed |
| 26/26 built-in known-answer checks pass (field decode and byte-exact round trip) | observed |
The decoder agrees field-by-field with a message serialized by google.protobuf 6.33.6, and its re-encode is byte-identical to SerializeToString() |
observed |
A real AndroidX DataStore file written by scripts/datastore_inject.py decodes to the correct structure and round-trips byte-identically |
observed |
Editing one value in the JSON tree and re-encoding produced a file that datastore_inject.py read back as the edited value |
observed |
| proto3 zero semantics: an implicit (no-presence) field set to 0 writes nothing; a presence-carrying field set to 0 writes its tag plus a zero byte | observed |
| A packed field and a nested message are both legal readings of the same length-delimited bytes (measured on real encoded bytes, not argued) | observed |
--split varint-length is protobuf's writeDelimitedTo framing; handed a gRPC frame it silently produced 5 frames instead of failing |
observed |
| Decoding a device-side protobuf file (real app data read over adb) | not done (device lock held by another agent; §7) |
| Any protobuf payload recovered from the benchmark APKs | not done — the one APK scanned carries no .pb/.proto resource (§7) |
blackboxprotobuf / protobuf-inspector behaviour |
unverified (not installed on this host; the reference still says so) |
1. Input forms, measured
$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --help
usage: protobuf_decode_raw.py [-h] [--hex STR] [--json] [--max-depth N]
[--no-expand] [--split {none,varint-length}]
[--reencode JSON] [--out PATH] [--check PATH]
[--selftest]
[input]
... exit = 0
Hex text, a binary file and stdin are all accepted, and the two stdin shapes are told apart without a flag:
$ '0a220a176578616d706c655f6578706972795f65706f63685f6d7312072080b08fe6b277' \
| python skills/apk-reverse/scripts/protobuf_decode_raw.py -
input: stdin (hex text), 36 byte(s)
frame 0 bytes 0..36 (36 B) status=end_of_buffer
f1 len-delimited(34) view=nested_message
...
$ cmd /c "python skills\apk-reverse\scripts\protobuf_decode_raw.py - < tools\_work\t5-proto\ds.preferences_pb"
input: stdin (raw bytes), 81 byte(s)
frame 0 bytes 0..81 (81 B) status=end_of_buffer
Separators and escapes, in one command (two frames, 0x prefixes, spaces, a \xNN escape):
$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --hex "0x08 0x96 0x01 | \x12\x07testing"
frame 0 bytes 0..3 (3 B) status=end_of_buffer
f1 varint 150
frame 1 bytes 3..12 (9 B) status=end_of_buffer
f2 len-delimited(7) view=utf8_string
...
string: 'testing'
08,96,\n01 (commas and a newline) and the single token 0x089601 both decode to the same field.
This command is the one that found the script's first real defect — see §6.
2. Known-answer fixtures: --selftest
Twelve fixtures are built byte by byte inside the script (no external file, no schema), covering a
canonical varint, a multi-byte varint, uint64 maximum, two-level nesting, a packed field, fixed64
and fixed32, an explicit zero, a deprecated group, a non-canonical varint, a 10-byte overflow, two
top-level messages in one stream, and varint-length framing. Each is decoded with an expected value
and passed through the round trip.
$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --selftest
== known-answer fixtures (bytes produced by hand in this script) ==
canonical varint 150 089601
field 1 varint value PASS 150
unsigned max uint64 08ffffffffffffffffff01
field 1 varint value PASS 18446744073709551615
nested two levels 089601120774657374696e671a060801120210072206038e029ea705...
field 3 nested child count PASS 2
deprecated group 3/4 43080544
field 8 has children PASS 1
non-canonical varint 08968100
field 1 varint value PASS 150
10-byte varint overflow 08ffffffffffffffffff7f
field 1 varint overflow PASS True
...
== round trip: decode -> re-encode -> byte comparison ==
canonical varint 150 MATCH (1 fields, 1 identical)
nested two levels MATCH (7 fields, 7 identical)
non-canonical varint MATCH (1 fields, 0 identical) re-encode emits the minimal
form 089601, original 08968100, and the report states why
varint length framing MATCH (7 fields, 7 identical)
== the two wire-format traps ==
explicit zero bytes 2800 decodes to value 0; absent field decodes to no fields at all: PASS
packed payload 038e029ea705 offers live=['bytes', 'packed_varint'], rejected=['nested_message',
'utf8_string'] -> view=packed_varint : PASS
selftest: 26/26 PASS exit = 0
The non-canonical fixture is the one deliberate exception: the re-encode is expected to differ
(08968100 -> 089601, the minimal form), and the tool reports the difference with its reason
rather than normalising it away silently. That is the honest form of "round trip": a difference that
is explained is evidence; a difference that is not is a bug.
3. Cross-check against the official protobuf runtime
tools/_work/t5-proto/official_ref.py builds a real FileDescriptorProto (Outer with int64 /
string / message / repeated int32 (packed) / fixed64 / float / enum / proto3-optional / bytes /
fixed32 fields, plus a nested Inner), serializes it with SerializeToString(deterministic=True),
and then asks the hand-written decoder to reproduce it.
$ python tools\_work\t5-proto\official_ref.py
== official runtime bytes (protobuf 6.33.6) ==
58 B: 089601120774657374696e671a0508011201782206038e029ea705290807060504030201350000c03f380240004a06038e029ea70555efbeadde
== field-by-field agreement with the schema ==
field 1 count == 150 PASS decoded 150
field 2 name == "testing" PASS decoded 'testing'
field 3 inner walked as a nested message PASS view=nested_message children=2
field 4 nums read as packed [3,270,86942] PASS view=packed_varint values=[3, 270, 86942]
field 5 stamp fixed64 hex 0807060504030201 PASS 0807060504030201
field 6 ratio fixed32 hex 0000c03f (1.5f) PASS 0000c03f
field 7 color enum == 2 PASS
field 8 explicit_zero is present and 0 PASS
field 9 blob hex 038e029ea705 PASS
field 10 crc fixed32 hex efbeadde PASS efbeadde
== round trip against the official bytes ==
re-encode is byte-identical to SerializeToString() PASS 10/10 fields identical
no field was rewritten PASS []
official-runtime cross-check: 21/21 PASS exit = 0
Field 5 is the useful false-friend: the decoder offers as_double_repr for those 8 bytes, and the
official value is a fixed64 integer. Nothing on the wire distinguishes them, which is why the
candidate list exists instead of a single typed value.
3.1 The four-way tie, measured on real bytes
A really-encoded nested message — Inner{id:1, tag:"x"} inside field 3, inner payload 0801120178
— is reported as:
nested message payload 0801120178 -> {'nested_message': 0.5, 'packed_varint': 0.5,
'utf8_string': 0.35, 'bytes': 0.2}
0801120178 is simultaneously a valid two-field message and a valid five-element packed varint
array. The decoder refuses to choose; the tie: line states that the displayed view is a display
default. A real packed field (2206038e029ea705) yields packed_varint 0.55 with
nested_message rejected — the rejection is printed with its reason instead of being omitted.
3.2 proto3 zero — the reference sentence, corrected by measurement
The reference previously stated that 28 00 is byte-identical to an absent field. The runtime shows
that the sentence needs splitting into two halves:
implicit int64 count = 0 : 120178 (3 B)
count never touched : 120178 (3 B)
implicit (no presence) zero is byte-identical to absent PASS
that byte string contains no tag for field 1 PASS
optional int32 explicit_zero=0: 1201784000 (5 B)
a presence-carrying field DOES put tag+0x00 on the wire PASS 1201784000
the optional zero costs 2 bytes the implicit zero never costs PASS 5 B vs 3 B
the decoder sees that 0 and flags the ambiguity PASS ["proto3_explicit_zero"]
Read it as two separate facts:
- For a field without presence,
value = 0and "never set" are the same bytes (no bytes at all), so a field missing from a decode is not evidence that the value in use was 0. - A
0that is on the wire was emitted on purpose — a presence-carrying field (40 00here, field 8) or a hand-rolled writer. The decoder cannot tell which, so it flags the case.
references/protocol-reverse.md §1 consequence 3 now carries that correction.
4. A real AndroidX DataStore file
scripts/datastore_inject.py writes the real AndroidX Preferences protobuf container
(map<string, Value>); the decoder has never seen that schema.
$ python skills/apk-reverse/scripts/datastore_inject.py --file tools\_work\t5-proto\ds.preferences_pb \
--key example_expiry_epoch_ms --type long --value 4102444800000 \
--extra example_flag=true:bool --extra example_name=hello:string
[ok] wrote tools\_work\t5-proto\ds.preferences_pb (81 bytes)
$ python skills/apk-reverse/scripts/protobuf_decode_raw.py tools\_work\t5-proto\ds.preferences_pb
input: tools\_work\t5-proto\ds.preferences_pb, 81 byte(s)
frame 0 bytes 0..81 (81 B) status=end_of_buffer
f1 len-delimited(34) view=nested_message
...
f1 len-delimited(23) view=utf8_string
string: 'example_expiry_epoch_ms'
f2 len-delimited(7) view=nested_message
...
f4 varint 4102444800000
f1 len-delimited(18) view=nested_message <- the second map entry
...
The map-entry structure, the key strings and the long value all come out correctly with no schema;
every length-delimited field still shows its ties (nested_message vs packed_varint vs
utf8_string) because those ties are real.
4.1 Round trip, and an edit confirmed by the other tool
$ python skills/apk-reverse/scripts/protobuf_decode_raw.py tools\_work\t5-proto\ds.preferences_pb --json > tools\_work\t5-proto\ds.tree.json
$ python skills/apk-reverse/scripts/protobuf_decode_raw.py --reencode tools\_work\t5-proto\ds.tree.json \
--out tools\_work\t5-proto\ds.reenc.bin --check tools\_work\t5-proto\ds.preferences_pb
re-encoded 81 byte(s) from 3 field(s)
fields re-encoded byte-identically: 3/3
written to tools\_work\t5-proto\ds.reenc.bin
check against tools\_work\t5-proto\ds.preferences_pb (81 B, the whole file): MATCH exit = 0
Then one value is edited in the JSON tree and re-encoded:
editing field 4 from 4102444800000
re-encoded 80 byte(s) from 3 field(s)
fields re-encoded byte-identically: 2/3
CHANGED field 1: expected 0a22...80b08fe6b277, got 0a21...20ffc7afa025 (the tree was edited)
$ python skills/apk-reverse/scripts/datastore_inject.py --in tools\_work\t5-proto\ds.edited.bin --list
[in] tools\_work\t5-proto\ds.edited.bin (80 bytes)
example_expiry_epoch_ms long=9999999999 (value=6 bytes)
example_flag bool=True (value=2 bytes)
example_name string='hello' (value=7 bytes)
Two independent facts in one run: the file shrank 81 -> 80 B because a 6-byte varint became a 5-byte one and the enclosing length prefix was recomputed (the re-encode rebuilds lengths, it does not copy them), and a different tool of this repository read the edited bytes back as the new value. A round trip that only agrees with itself would not have caught either.
5. Framing: writeDelimitedTo is not a gRPC frame
tools/_work/t5-proto/framing_check.py:
message (20 B): 089601120774657374696e672206038e029ea705
writeDelimitedTo form (21 B): 14089601120774657374696e672206038e029ea705
varint-length framing: exactly one frame PASS got 1
a second message makes two frames PASS
gRPC frame (25 B): 0000000014089601120774657374696e672206038e029ea705
decoded with --split varint-length: 5 frame(s): [(1, 1, 'end_of_buffer'), (2, 2, ...),
(3, 3, ...), (4, 4, ...), (5, 25, ...)]
gRPC frames are NOT varint-length framed (the split misfires) PASS got 5 frame(s)
stripping the 5-byte gRPC prefix decodes the message correctly PASS field 1=150, field 4=[3, 270, 86942]
'|' separates two independently framed messages PASS frames=2
framing check: 8/8 PASS exit = 0
| Framing | Bytes in front of the message | --split varint-length |
|---|---|---|
protobuf writeDelimitedTo |
varint length | correct — 1 frame per message |
| gRPC | 1 compression-flag byte + 4-byte big-endian length | wrong, and it does not fail: the 4 zero bytes become four empty frames |
The same gRPC bytes decoded with no splitting reported stray_end_group (exit 1), and a
writeDelimitedTo file decoded with no splitting reported the same error — that error is a refusal,
whereas the wrongful split is a silent reinterpretation. Strip the 5-byte gRPC prefix yourself
(references/protocol-reverse.md §3), and check the frame count either way.
A --split varint-length round trip compares frame bodies only, and says so:
check against tools\_work\t5-proto\two_frames.bin (26 B, frame bodies only: 2 of 26 byte(s) are
length prefixes (framing, not message data)): MATCH
6. Defects these checks found (in the code, not in the target)
Four defects, three of them in the delivered script, all found by the checks above rather than by reading the code.
| # | Defect (observed) | How it surfaced | Fix |
|---|---|---|---|
| 1 | When two candidate readings tied, both were lowered to the same confidence as the bytes fallback, and the tie was then broken by alphabetical order — bytes sorts before nested_message, so a real nested message was never expanded and --reencode copied it as an opaque blob |
--selftest: nested two levels / field 3 nested child count returned 0 instead of 2 |
bytes now scores 0.2 and ties are broken by an explicit VIEW_PRIORITY, structural candidates first |
| 2 | In the \xNN path, whitespace and commas were kept as data bytes, so `"08 96 01 |
\x12\x07testing"decoded the space after the bar asfield 4 varint 18and then hitinvalid_wire_type` |
the input-form check in §1 |
| 3 | Two fixtures were mis-designed, not the code: a bare packed stream and a length-prefixed frame were round-tripped as if each were one top-level message, which reported FAIL for correct behaviour |
--selftest round-trip section |
the packed fixture is now a field value; the framing fixture compares frame bodies |
| 4 | The cross-check's own assertion expected 0800 for the explicit zero, but the field under test is number 8, whose tag is 0x40 — the official bytes were ...4000, and the assertion, not the decoder, was wrong |
official_ref.py |
assertion compares 4000 and the 2-byte cost |
| 5 | --no-expand did nothing: the expansion branch never consulted the flag, so a 7-level payload was still walked to the bottom |
the deep-payload check below | the branch honours --no-expand and records expansion: suppressed by --no-expand on the field |
| 6 | Hitting --max-depth was invisible: the depth cut-off only appeared as a rejected note inside one candidate, so the report showed a shallow tree with no statement that anything had been left unexamined |
the same check | a depth_limit ambiguity is emitted, naming the path and the limit |
Defect 1 is worth its own line in any future reader's mind: a heuristic tie-break silently changed which bytes the tool re-emitted, and it was caught only because a fixture asserted a child count. Defects 5 and 6 are the other direction — the tool silently did more than asked, and silently stopped without saying so; both are ways a report can be read as complete when it is not.
6.1 The deep-payload check
A 7-level nesting (08 01 wrapped six times in field 1, 14 bytes) is the fixture:
$ python skills/apk-reverse/scripts/protobuf_decode_raw.py tools\_work\t5-proto\deep.bin
frame 0 bytes 0..14 (14 B) status=end_of_buffer
[depth_limit] frame0.1.1.1.1.1.1 @2: a nested reading of this field was not attempted:
--max-depth 5 reached. The bytes here may well be a message; re-run with a larger --max-depth
before concluding that they are not
$ python ... tools\_work\t5-proto\deep.bin --max-depth 2
[depth_limit] frame0.1.1.1 @2: ... --max-depth 2 reached. ...
$ python ... tools\_work\t5-proto\deep.bin --no-expand
expansion: suppressed by --no-expand
The truncation is reported as an ambiguity with the path of the field it stopped at, so "no nested message here" can never be read off a truncated tree.
7. What was not done
- No device was used. The pass intended to pull a real app's
.preferences_pbfrom<DEVICE>:python tools\_work\devlock.py acquire proto --timeout 1800waited 120 s and timed out with the lock held by another agent (held by bench for 297s, note "L1 repack control+patched install & screen; L3 frida"). Per the two-strike rule the device source was dropped rather than contended for, and the real-file evidence comes from the DataStore file this repository's own tool writes (§4). Reading a device-side protobuf file with this decoder is therefore unverified. Device facts used elsewhere in this repository (attach-by-name, the brokenlogdchannel) were not re-tested here. - No protobuf payload was recovered from the benchmark APKs. Scanning the available sample
(
<PKG>-style de-identified, 4,293,620 B, 435 zip entries) for.pb/.proto/protobufresources found none; the MASTG crackmes carry no protobuf at all. So no wild-captured payload was decoded in this pass —inferredfor any claim that a real traffic body looks like these fixtures, though §3's bytes come from the reference implementation and §4's from a real AndroidX container. - No live traffic. Nothing was proxied, so the gRPC section of the reference is exercised only as framing (§5), not as captured HTTP/2.
blackboxprotobuf/protobuf-inspectorwere not installed or compared; the reference still labels them unverified.--max-depthis measured up to a 7-level fixture (§6.1); no real payload deeper than that was decoded, and no payload was decoded whose only reading is a deep message.
8. Artifacts produced by this pass
tools/_work/t5-proto/ (not committed): official_ref.py (official-runtime cross-check, 21 checks),
framing_check.py (8 checks), ds.preferences_pb (81 B, real DataStore container),
ds.tree.json / ds.edited.json (decoded trees), ds.reenc.bin (byte-identical round trip),
ds.edited.bin (80 B, edited value read back by datastore_inject.py), two_frames.bin,
frames.tree.json, frames.reenc.bin.
Files changed in the repository by this pass: skills/apk-reverse/scripts/protobuf_decode_raw.py
(new), and skills/apk-reverse/references/protocol-reverse.md (a new measured subsection under §1,
the corrected proto3-zero wording in §1 consequence 3, and a new framing failure mode in §6).
SKILL.md and README.md were deliberately not touched; the index rows for the new script are
handed to the lead to register.