Files

3.3 KiB

name, description, domain, subdomain, tags, version, author, license, mitre_attack, d3fend, references
name description domain subdomain tags version author license mitre_attack d3fend references
reverse-engineering-custom-c2-protocols Reverses proprietary command-and-control protocols: locating send/recv routines, recovering the message framing and encryption/encoding, and reconstructing the command set to build a decoder or emulator. Activates for requests to reverse a custom C2 protocol, decode beacon traffic, or document a malware command structure. cybersecurity reverse-engineering
reverse-engineering
command-and-control
protocol
network
decryption
beacon
1.0.0 analyst-ai-pack Apache-2.0
T1071
T1095
T1573
D3-NTA
D3-DA
MITRE ATT&CK T1071 Application Layer Protocol — https://attack.mitre.org/techniques/T1071/
MITRE ATT&CK T1573 Encrypted Channel — https://attack.mitre.org/techniques/T1573/

Reverse Engineering Custom C2 Protocols

When to Use

  • A sample uses a non-standard protocol (or a custom layer over HTTP/TCP) and you must decode its traffic.
  • You need to recover message framing, the encryption/encoding scheme, and the command set.
  • You want to build a standalone decoder or emulator to parse captured beacon traffic.

Do not use this for documented protocols handled by existing dissectors — use the protocol analyzer directly instead of reversing from scratch.

Prerequisites

  • A disassembler/debugger and a network capture (PCAP) of the sample's traffic where possible.
  • The crypto/obfuscation skills for recovering keys and encodings.

Workflow

Step 1: Locate the network routines

Find send/recv/WSASend/WinHTTP calls (or the framework wrappers) and the buffers they operate on. These bracket the serialization and crypto.

Step 2: Recover framing

Trace how outbound buffers are built: magic bytes, length prefixes, sequence/ID fields, and type/opcode fields. Document the header layout.

[ magic(4) ][ len(4 LE) ][ opcode(1) ][ flags(1) ][ payload(len) ]

Step 3: Recover crypto/encoding

Identify the transform applied before send / after recv (XOR with embedded key, RC4, AES, base64). Recover keys from the binary or from the key-setup routine.

Step 4: Reconstruct the command set

Map opcodes to handlers (shell, download, upload, sleep/jitter, exit) by following the dispatch switch. Build a table of command → behavior.

Step 5: Build a decoder and validate

Implement a parser/decryptor and run it against captured traffic; confirm it yields coherent commands and recovers the beacon's config.

python scripts/analyst.py decode capture.bin --key 0x5a --magic 4d5a4331

Validation

  • The decoder parses every message in a real capture without desync.
  • Decrypted payloads are coherent (printable commands / structured config).
  • The opcode table matches the dispatch logic observed in the binary.

Pitfalls

  • Assuming fixed keys when the protocol negotiates a session key in the first exchange.
  • Missing a length/sequence field and desynchronizing after the first message.
  • Confusing transport (TLS) with the inner custom layer; decode the inner layer.

References