reverse-skills-max: initial public release (restored, redacted, hardened)
This commit is contained in:
baradok609hecl1pq6qi
@@ -0,0 +1,201 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to the Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by the Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding any notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. Please do not remove or change
|
||||
the license header comment from a contributed file except when
|
||||
necessary.
|
||||
|
||||
Copyright 2026 mukul975
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,396 @@
|
||||
---
|
||||
name: extracting-iocs-from-malware-samples
|
||||
description: Extracts indicators of compromise (IOCs) from malware samples, including
|
||||
file hashes, network indicators (IPs, domains, URLs, PCAP indicators), host artifacts
|
||||
(file paths, registry keys, mutexes), and behavioral patterns, using tools like
|
||||
CyberChef, then defangs and exports them in standard threat-intel formats. Use
|
||||
for IOC extraction, threat indicator harvesting, or building detection content
|
||||
from a sample.
|
||||
domain: cybersecurity
|
||||
subdomain: malware-analysis
|
||||
tags:
|
||||
- malware
|
||||
- IOC-extraction
|
||||
- threat-intelligence
|
||||
- indicators
|
||||
- detection
|
||||
version: 1.0.0
|
||||
author: mahipal
|
||||
license: Apache-2.0
|
||||
nist_csf:
|
||||
- DE.AE-02
|
||||
- RS.AN-03
|
||||
- ID.RA-01
|
||||
- DE.CM-01
|
||||
mitre_attack:
|
||||
- T1027
|
||||
- T1055
|
||||
- T1140
|
||||
- T1497
|
||||
---
|
||||
|
||||
# Extracting IOCs from Malware Samples
|
||||
|
||||
## When to Use
|
||||
|
||||
- A malware analysis (static or dynamic) is complete and actionable indicators need to be extracted for defense teams
|
||||
- Building blocklists for firewalls, proxies, and DNS sinkholes from analyzed samples
|
||||
- Creating YARA rules, Snort/Suricata signatures, or SIEM detection content from malware artifacts
|
||||
- Contributing to threat intelligence sharing platforms (MISP, OTX, ThreatConnect)
|
||||
- Tracking malware campaigns by correlating IOCs across multiple samples
|
||||
|
||||
**Do not use** for IOCs from unverified sources without validation; false positives in blocklists can disrupt legitimate business operations.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.8+ with `iocextract`, `pefile`, `yara-python` libraries installed
|
||||
- Completed malware analysis report (static analysis, dynamic analysis, or reverse engineering)
|
||||
- Access to PCAP files, memory dumps, or sandbox reports from the analysis
|
||||
- MISP instance or STIX/TAXII server for structured IOC sharing
|
||||
- VirusTotal API key for IOC enrichment and validation
|
||||
- CyberChef for decoding obfuscated indicators
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Extract File-Based IOCs
|
||||
|
||||
Compute hashes and identify file metadata indicators:
|
||||
|
||||
```bash
|
||||
# Generate all standard hashes
|
||||
md5sum malware_sample.exe
|
||||
sha1sum malware_sample.exe
|
||||
sha256sum malware_sample.exe
|
||||
|
||||
# Generate ssdeep fuzzy hash for similarity matching
|
||||
ssdeep malware_sample.exe
|
||||
|
||||
# Generate imphash (import hash) for PE files
|
||||
python3 -c "
|
||||
import pefile
|
||||
pe = pefile.PE('malware_sample.exe')
|
||||
print(f'Imphash: {pe.get_imphash()}')
|
||||
"
|
||||
|
||||
# Generate TLSH (Trend Micro Locality Sensitive Hash)
|
||||
python3 -c "
|
||||
import tlsh
|
||||
with open('malware_sample.exe', 'rb') as f:
|
||||
h = tlsh.hash(f.read())
|
||||
print(f'TLSH: {h}')
|
||||
"
|
||||
|
||||
# Compile file metadata IOCs
|
||||
python3 << 'PYEOF'
|
||||
import pefile
|
||||
import os
|
||||
import hashlib
|
||||
import datetime
|
||||
|
||||
pe = pefile.PE("malware_sample.exe")
|
||||
|
||||
print("FILE IOCs:")
|
||||
with open("malware_sample.exe", "rb") as f:
|
||||
data = f.read()
|
||||
print(f" MD5: {hashlib.md5(data).hexdigest()}")
|
||||
print(f" SHA-1: {hashlib.sha1(data).hexdigest()}")
|
||||
print(f" SHA-256: {hashlib.sha256(data).hexdigest()}")
|
||||
print(f" File Size: {len(data)} bytes")
|
||||
|
||||
ts = pe.FILE_HEADER.TimeDateStamp
|
||||
print(f" Compile: {datetime.datetime.utcfromtimestamp(ts)} UTC")
|
||||
print(f" Imphash: {pe.get_imphash()}")
|
||||
PYEOF
|
||||
```
|
||||
|
||||
### Step 2: Extract Network IOCs
|
||||
|
||||
Pull network indicators from strings, PCAP, and sandbox reports:
|
||||
|
||||
```python
|
||||
# Extract network IOCs from strings
|
||||
import re
|
||||
|
||||
with open("malware_sample.exe", "rb") as f:
|
||||
data = f.read()
|
||||
|
||||
# Extract ASCII and Unicode strings
|
||||
ascii_strings = re.findall(b'[ -~]{4,}', data)
|
||||
unicode_strings = re.findall(b'(?:[ -~]\x00){4,}', data)
|
||||
|
||||
all_strings = [s.decode('ascii', errors='ignore') for s in ascii_strings]
|
||||
all_strings += [s.decode('utf-16-le', errors='ignore') for s in unicode_strings]
|
||||
|
||||
# IP addresses (excluding private ranges for C2 indicators)
|
||||
ip_pattern = re.compile(r'\b(?:(?:25[0-5]|2[0-4]\d|1\d{2}|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d{2}|[1-9]?\d)\b')
|
||||
ips = set()
|
||||
for s in all_strings:
|
||||
for ip in ip_pattern.findall(s):
|
||||
# Filter out private/reserved ranges
|
||||
octets = [int(o) for o in ip.split('.')]
|
||||
if octets[0] not in [10, 127, 0] and not (octets[0] == 172 and 16 <= octets[1] <= 31) and not (octets[0] == 192 and octets[1] == 168):
|
||||
ips.add(ip)
|
||||
|
||||
# Domain names
|
||||
domain_pattern = re.compile(r'\b[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z]{2,})+\b')
|
||||
domains = set()
|
||||
for s in all_strings:
|
||||
for d in domain_pattern.findall(s):
|
||||
if not d.endswith(('.dll', '.exe', '.sys', '.com.au')):
|
||||
domains.add(d)
|
||||
|
||||
# URLs
|
||||
url_pattern = re.compile(r'https?://[^\s<>"{}|\\^`\[\]]+')
|
||||
urls = set()
|
||||
for s in all_strings:
|
||||
for u in url_pattern.findall(s):
|
||||
urls.add(u)
|
||||
|
||||
print("NETWORK IOCs:")
|
||||
print(f" IPs: {ips}")
|
||||
print(f" Domains: {domains}")
|
||||
print(f" URLs: {urls}")
|
||||
```
|
||||
|
||||
### Step 3: Extract Host-Based IOCs
|
||||
|
||||
Identify file paths, registry keys, mutexes, and services:
|
||||
|
||||
```python
|
||||
# Extract host-based IOCs from sandbox report
|
||||
import json
|
||||
|
||||
with open("cuckoo_report.json") as f:
|
||||
report = json.load(f)
|
||||
|
||||
print("HOST IOCs:")
|
||||
|
||||
# File paths created or modified
|
||||
print("\nFile Paths:")
|
||||
for f in report["behavior"]["summary"].get("files", []):
|
||||
if any(p in f.lower() for p in ["temp", "appdata", "system32", "programdata"]):
|
||||
print(f" [DROPPED] {f}")
|
||||
|
||||
# Registry keys for persistence
|
||||
print("\nRegistry Keys:")
|
||||
for key in report["behavior"]["summary"].get("write_keys", []):
|
||||
if any(p in key.lower() for p in ["run", "service", "startup", "shell"]):
|
||||
print(f" [PERSIST] {key}")
|
||||
|
||||
# Mutexes (unique to malware family)
|
||||
print("\nMutexes:")
|
||||
for mutex in report["behavior"]["summary"].get("mutexes", []):
|
||||
if mutex not in ["Local\\!IETld!Mutex", "RasPbFile"]: # Filter known Windows mutexes
|
||||
print(f" [MUTEX] {mutex}")
|
||||
|
||||
# Created services
|
||||
print("\nServices:")
|
||||
for svc in report["behavior"]["summary"].get("started_services", []):
|
||||
print(f" [SERVICE] {svc}")
|
||||
```
|
||||
|
||||
### Step 4: Extract Network IOCs from PCAP
|
||||
|
||||
Parse network captures for additional indicators:
|
||||
|
||||
```bash
|
||||
# Extract DNS queries from PCAP
|
||||
tshark -r capture.pcap -T fields -e dns.qry.name -Y "dns.flags.response == 0" | sort -u
|
||||
|
||||
# Extract HTTP hosts and URLs
|
||||
tshark -r capture.pcap -T fields -e http.host -e http.request.uri -Y "http.request" | sort -u
|
||||
|
||||
# Extract TLS server names (SNI)
|
||||
tshark -r capture.pcap -T fields -e tls.handshake.extensions_server_name -Y "tls.handshake.type == 1" | sort -u
|
||||
|
||||
# Extract JA3 hashes
|
||||
tshark -r capture.pcap -T fields -e tls.handshake.ja3 -Y "tls.handshake.type == 1" | sort -u
|
||||
|
||||
# Extract unique destination IPs
|
||||
tshark -r capture.pcap -T fields -e ip.dst -Y "ip.src == 10.0.2.15" | sort -u
|
||||
|
||||
# Extract User-Agent strings
|
||||
tshark -r capture.pcap -T fields -e http.user_agent -Y "http.user_agent" | sort -u
|
||||
```
|
||||
|
||||
### Step 5: Defang and Validate IOCs
|
||||
|
||||
Defang indicators for safe sharing and validate against threat intelligence:
|
||||
|
||||
```python
|
||||
# Defang IOCs for safe sharing
|
||||
def defang_ip(ip):
|
||||
return ip.replace(".", "[.]")
|
||||
|
||||
def defang_url(url):
|
||||
return url.replace("http", "hxxp").replace(".", "[.]")
|
||||
|
||||
def defang_domain(domain):
|
||||
return domain.replace(".", "[.]")
|
||||
|
||||
# Validate IOCs against VirusTotal
|
||||
import requests
|
||||
|
||||
VT_API_KEY = "your_api_key"
|
||||
|
||||
def check_vt_ip(ip):
|
||||
resp = requests.get(f"https://www.virustotal.com/api/v3/ip_addresses/{ip}",
|
||||
headers={"x-apikey": VT_API_KEY})
|
||||
data = resp.json()
|
||||
stats = data["data"]["attributes"]["last_analysis_stats"]
|
||||
return stats["malicious"]
|
||||
|
||||
def check_vt_domain(domain):
|
||||
resp = requests.get(f"https://www.virustotal.com/api/v3/domains/{domain}",
|
||||
headers={"x-apikey": VT_API_KEY})
|
||||
data = resp.json()
|
||||
stats = data["data"]["attributes"]["last_analysis_stats"]
|
||||
return stats["malicious"]
|
||||
|
||||
# Validate each IOC
|
||||
for ip in ips:
|
||||
detections = check_vt_ip(ip)
|
||||
print(f" {defang_ip(ip)} - VT: {detections} detections")
|
||||
```
|
||||
|
||||
### Step 6: Export IOCs in Standard Formats
|
||||
|
||||
Generate structured IOC outputs for sharing and ingestion:
|
||||
|
||||
```python
|
||||
# Export as STIX 2.1 bundle
|
||||
from stix2 import Indicator, Bundle, Malware, Relationship
|
||||
import datetime
|
||||
|
||||
indicators = []
|
||||
|
||||
# File hash indicator
|
||||
indicators.append(Indicator(
|
||||
name="Malware SHA-256 Hash",
|
||||
pattern=f"[file:hashes.'SHA-256' = '{sha256_hash}']",
|
||||
pattern_type="stix",
|
||||
valid_from=datetime.datetime.now(datetime.timezone.utc),
|
||||
labels=["malicious-activity"]
|
||||
))
|
||||
|
||||
# IP indicator
|
||||
for ip in ips:
|
||||
indicators.append(Indicator(
|
||||
name=f"C2 IP Address {ip}",
|
||||
pattern=f"[ipv4-addr:value = '{ip}']",
|
||||
pattern_type="stix",
|
||||
valid_from=datetime.datetime.now(datetime.timezone.utc),
|
||||
labels=["malicious-activity"]
|
||||
))
|
||||
|
||||
# Domain indicator
|
||||
for domain in domains:
|
||||
indicators.append(Indicator(
|
||||
name=f"C2 Domain {domain}",
|
||||
pattern=f"[domain-name:value = '{domain}']",
|
||||
pattern_type="stix",
|
||||
valid_from=datetime.datetime.now(datetime.timezone.utc),
|
||||
labels=["malicious-activity"]
|
||||
))
|
||||
|
||||
bundle = Bundle(objects=indicators)
|
||||
with open("iocs_stix.json", "w") as f:
|
||||
f.write(bundle.serialize(pretty=True))
|
||||
|
||||
# Export as CSV for SIEM ingestion
|
||||
import csv
|
||||
with open("iocs.csv", "w", newline="") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow(["type", "value", "context", "confidence"])
|
||||
writer.writerow(["sha256", sha256_hash, "malware_sample", "high"])
|
||||
for ip in ips:
|
||||
writer.writerow(["ipv4", ip, "c2_server", "high"])
|
||||
for domain in domains:
|
||||
writer.writerow(["domain", domain, "c2_domain", "high"])
|
||||
for url in urls:
|
||||
writer.writerow(["url", url, "c2_url", "high"])
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| **IOC (Indicator of Compromise)** | Forensic artifact observed in a network or system that indicates a potential intrusion: hashes, IPs, domains, file paths, registry keys |
|
||||
| **Defanging** | Modifying IOCs to prevent accidental activation (e.g., replacing dots with [.] in URLs and IPs for safe sharing in reports) |
|
||||
| **Imphash** | MD5 hash of the import table functions in a PE file; samples from the same malware family often share the same imphash |
|
||||
| **STIX/TAXII** | Structured Threat Information Expression / Trusted Automated Exchange; standards for encoding and transmitting threat intelligence |
|
||||
| **JA3/JA3S** | TLS client/server fingerprint based on ClientHello/ServerHello parameters; identifies specific malware families by their TLS implementation |
|
||||
| **Fuzzy Hashing (ssdeep)** | Context-triggered piecewise hashing that identifies similar files even with minor modifications; useful for malware variant detection |
|
||||
| **MISP** | Malware Information Sharing Platform; open-source threat intelligence platform for collecting, storing, and sharing IOCs |
|
||||
|
||||
## Tools & Systems
|
||||
|
||||
- **iocextract (Python)**: Automated IOC extraction library supporting IPs, URLs, domains, hashes, and YARA rules from text
|
||||
- **MISP**: Open-source threat intelligence sharing platform for structured IOC management and distribution
|
||||
- **CyberChef**: Web-based tool for decoding, decrypting, and transforming data useful for deobfuscating encoded IOCs
|
||||
- **tshark**: Command-line network protocol analyzer for extracting network IOCs from PCAP files
|
||||
- **VirusTotal**: Online service for validating and enriching IOCs with community detection results and threat intelligence
|
||||
|
||||
## Common Scenarios
|
||||
|
||||
### Scenario: Building a Comprehensive IOC Package from a Ransomware Sample
|
||||
|
||||
**Context**: A ransomware incident requires rapid IOC extraction for blocking across the enterprise while the full investigation continues. Multiple data sources are available: the sample binary, PCAP from network monitoring, and a Cuckoo sandbox report.
|
||||
|
||||
**Approach**:
|
||||
1. Compute all file hashes (MD5, SHA-1, SHA-256, imphash, ssdeep) for the ransomware binary and any dropped files
|
||||
2. Extract network IOCs from strings in the binary (hardcoded C2 addresses)
|
||||
3. Parse the PCAP for DNS queries, HTTP requests, and TLS SNI fields
|
||||
4. Extract host IOCs from the sandbox report (file paths, registry keys, mutexes, ransom note filenames)
|
||||
5. Validate all network IOCs against VirusTotal to confirm malicious status and check for known associations
|
||||
6. Defang all indicators and compile into STIX 2.1 format for sharing and CSV for SIEM ingestion
|
||||
7. Submit to MISP event for organizational and community sharing
|
||||
|
||||
**Pitfalls**:
|
||||
- Including IP addresses of legitimate CDNs or cloud services without validating context (e.g., AWS IPs used for hosting, not inherently malicious)
|
||||
- Not defanging URLs and IPs in reports, leading to accidental clicks or DNS resolution
|
||||
- Extracting strings from packed binaries (IOCs from packed samples are unreliable; unpack first)
|
||||
- Forgetting to include dropped file hashes (the initial dropper and the final payload are separate IOCs)
|
||||
|
||||
## Output Format
|
||||
|
||||
```
|
||||
IOC EXTRACTION REPORT
|
||||
======================
|
||||
Sample: ransomware.exe
|
||||
Analysis Date: 2025-09-15
|
||||
Analyst: [Name]
|
||||
|
||||
FILE INDICATORS
|
||||
SHA-256: e3b0c44298fc1c149afbf4c8996fb924...
|
||||
SHA-1: da39a3ee5e6b4b0d3255bfef95601890afd80709
|
||||
MD5: d41d8cd98f00b204e9800998ecf8427e
|
||||
Imphash: a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6
|
||||
ssdeep: 3072:kJh3bN7fY+aUkJh3bN7fY+aU:kJh3R7aUkJh3R7aU
|
||||
|
||||
NETWORK INDICATORS
|
||||
C2 IPs: 185.220.101[.]42, 91.215.85[.]17
|
||||
C2 Domains: update.malicious[.]com, backup.evil[.]net
|
||||
C2 URLs: hxxps://update.malicious[.]com/gate.php
|
||||
hxxps://backup.evil[.]net/gate.php
|
||||
JA3 Hash: a0e9f5d64349fb13191bc781f81f42e1
|
||||
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0)
|
||||
|
||||
HOST INDICATORS
|
||||
File Paths: C:\Users\Public\svchost.exe
|
||||
C:\Users\%USER%\AppData\Local\Temp\payload.dll
|
||||
C:\Users\%USER%\Desktop\README_DECRYPT.txt
|
||||
Registry Keys: HKCU\Software\Microsoft\Windows\CurrentVersion\Run\WindowsUpdate
|
||||
Mutexes: Global\CryptLocker_2025_Q3
|
||||
Services: FakeWindowsUpdate
|
||||
|
||||
CONFIDENCE ASSESSMENT
|
||||
High Confidence: SHA-256, C2 IPs (validated via VT), Mutexes
|
||||
Medium Confidence: Domains (could be compromised legitimate sites)
|
||||
Low Confidence: User-Agent (common string, high false positive risk)
|
||||
|
||||
EXPORT FILES
|
||||
stix_bundle.json - STIX 2.1 format for TIP ingestion
|
||||
iocs.csv - Flat CSV for SIEM blocklist import
|
||||
yara_rule.yar - YARA detection rule
|
||||
```
|
||||
@@ -0,0 +1,77 @@
|
||||
# API Reference: Malware IOC Extraction Agent
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| pefile | >=2023.2 | PE file parsing for imphash, sections, imports |
|
||||
| yara-python | >=4.3 | YARA rule scanning against malware samples |
|
||||
| requests | >=2.28 | VirusTotal API v3 IOC validation |
|
||||
|
||||
## CLI Usage
|
||||
|
||||
```bash
|
||||
python scripts/agent.py \
|
||||
--sample /cases/malware.exe \
|
||||
--yara-rules /rules/malware.yar \
|
||||
--vt-key YOUR_VT_API_KEY \
|
||||
--output-dir /cases/analysis/ \
|
||||
--output ioc_report.json
|
||||
```
|
||||
|
||||
## Functions
|
||||
|
||||
### `compute_hashes(file_path) -> dict`
|
||||
Computes MD5, SHA-1, SHA-256 and file size for the sample.
|
||||
|
||||
### `extract_pe_metadata(file_path) -> dict`
|
||||
Parses PE headers via pefile: imphash, compile timestamp, section entropy, import table.
|
||||
|
||||
### `extract_strings(file_path, min_length) -> list`
|
||||
Extracts ASCII and Unicode strings (min 4 chars) from the binary.
|
||||
|
||||
### `extract_network_iocs(strings) -> dict`
|
||||
Regex extraction of IPs, domains, URLs, emails from strings. Filters private IP ranges.
|
||||
|
||||
### `extract_host_iocs(strings) -> dict`
|
||||
Identifies Windows file paths, registry keys, and mutex names from strings.
|
||||
|
||||
### `run_yara_scan(file_path, rules_path) -> list`
|
||||
Compiles and runs YARA rules against the sample. Returns matched rule names, tags, and string offsets.
|
||||
|
||||
### `validate_ioc_virustotal(ioc_value, ioc_type, api_key) -> dict`
|
||||
Queries VirusTotal API v3 for IP, domain, or file hash. Returns malicious/suspicious counts.
|
||||
|
||||
### `defang_ioc(value) -> str`
|
||||
Defangs IOCs by replacing `http` with `hxxp` and `.` with `[.]`.
|
||||
|
||||
### `export_stix_bundle(iocs, sha256) -> dict`
|
||||
Builds a STIX 2.1 indicator bundle with file hash, IP, and domain patterns.
|
||||
|
||||
### `export_csv(iocs, hashes, output_path)`
|
||||
Writes IOCs to CSV format (type, value, context, confidence) for SIEM ingestion.
|
||||
|
||||
### `run_extraction(sample_path, output_dir, yara_rules, vt_key) -> dict`
|
||||
Orchestrates the full extraction pipeline and generates all output files.
|
||||
|
||||
## Regex Patterns
|
||||
|
||||
| Pattern | Target |
|
||||
|---------|--------|
|
||||
| `\b(?:(?:25[0-5]\|...)\.){3}...\b` | IPv4 addresses |
|
||||
| `\b[a-zA-Z0-9]...\.[a-zA-Z]{2,}+\b` | Domain names |
|
||||
| `https?://[^\s<>"'{}]+` | URLs |
|
||||
| `[a-zA-Z0-9_.+-]+@...` | Email addresses |
|
||||
|
||||
## Output Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"hashes": {"md5": "...", "sha256": "...", "sha1": "..."},
|
||||
"pe_metadata": {"imphash": "...", "compile_time": "...", "sections": []},
|
||||
"network_iocs": {"ips": [], "domains": [], "urls": []},
|
||||
"host_iocs": {"file_paths": [], "registry_keys": [], "mutexes": []},
|
||||
"yara_matches": [{"rule": "APT28_dropper", "tags": ["apt"]}],
|
||||
"summary": {"ips": 3, "domains": 5, "yara_hits": 1}
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,271 @@
|
||||
#!/usr/bin/env python3
|
||||
# For authorized testing in lab/CTF environments only
|
||||
"""IOC extraction agent using pefile, yara-python, and requests for VirusTotal validation."""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from typing import List, Set
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
try:
|
||||
import pefile
|
||||
except ImportError:
|
||||
sys.exit("pefile required: pip install pefile")
|
||||
|
||||
try:
|
||||
import yara
|
||||
except ImportError:
|
||||
yara = None
|
||||
logger.warning("yara-python not installed; YARA scanning disabled")
|
||||
|
||||
try:
|
||||
import requests
|
||||
except ImportError:
|
||||
requests = None
|
||||
logger.warning("requests not installed; VT validation disabled")
|
||||
|
||||
IP_RE = re.compile(r"\b(?:(?:25[0-5]|2[0-4]\d|1?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|1?\d\d?)\b")
|
||||
DOMAIN_RE = re.compile(r"\b[a-zA-Z0-9](?:[a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z]{2,})+\b")
|
||||
URL_RE = re.compile(r"https?://[^\s<>\"'{}|\\^`\[\]]+")
|
||||
EMAIL_RE = re.compile(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z]{2,}")
|
||||
|
||||
PRIVATE_IP_PREFIXES = ("10.", "127.", "0.", "192.168.", "169.254.")
|
||||
FALSE_DOMAIN_SUFFIXES = (".dll", ".exe", ".sys", ".ocx", ".drv", ".pdb")
|
||||
|
||||
|
||||
def compute_hashes(file_path: str) -> dict:
|
||||
"""Compute MD5, SHA-1, SHA-256 hashes of a file."""
|
||||
with open(file_path, "rb") as f:
|
||||
data = f.read()
|
||||
return {
|
||||
"md5": hashlib.md5(data).hexdigest(),
|
||||
"sha1": hashlib.sha1(data).hexdigest(),
|
||||
"sha256": hashlib.sha256(data).hexdigest(),
|
||||
"size_bytes": len(data),
|
||||
}
|
||||
|
||||
|
||||
def extract_pe_metadata(file_path: str) -> dict:
|
||||
"""Extract PE file metadata including imphash and compile time."""
|
||||
try:
|
||||
pe = pefile.PE(file_path)
|
||||
meta = {
|
||||
"imphash": pe.get_imphash(),
|
||||
"compile_time": datetime.utcfromtimestamp(pe.FILE_HEADER.TimeDateStamp).isoformat(),
|
||||
"sections": [],
|
||||
"imports": [],
|
||||
}
|
||||
for section in pe.sections:
|
||||
name = section.Name.rstrip(b"\x00").decode("ascii", errors="replace")
|
||||
meta["sections"].append({
|
||||
"name": name, "entropy": round(section.get_entropy(), 2),
|
||||
"virtual_size": section.Misc_VirtualSize, "raw_size": section.SizeOfRawData,
|
||||
})
|
||||
if hasattr(pe, "DIRECTORY_ENTRY_IMPORT"):
|
||||
for entry in pe.DIRECTORY_ENTRY_IMPORT:
|
||||
dll_name = entry.dll.decode("ascii", errors="replace")
|
||||
funcs = [imp.name.decode("ascii", errors="replace") for imp in entry.imports if imp.name]
|
||||
meta["imports"].append({"dll": dll_name, "functions": funcs[:20]})
|
||||
pe.close()
|
||||
return meta
|
||||
except pefile.PEFormatError:
|
||||
return {"error": "Not a valid PE file"}
|
||||
|
||||
|
||||
def extract_strings(file_path: str, min_length: int = 4) -> List[str]:
|
||||
"""Extract ASCII and Unicode strings from binary."""
|
||||
with open(file_path, "rb") as f:
|
||||
data = f.read()
|
||||
ascii_strs = [s.decode("ascii") for s in re.findall(b"[ -~]{%d,}" % min_length, data)]
|
||||
unicode_strs = [s.decode("utf-16-le", errors="ignore")
|
||||
for s in re.findall(b"(?:[ -~]\x00){%d,}" % min_length, data)]
|
||||
return ascii_strs + unicode_strs
|
||||
|
||||
|
||||
def extract_network_iocs(strings: List[str]) -> dict:
|
||||
"""Extract IPs, domains, URLs, emails from string list."""
|
||||
ips: Set[str] = set()
|
||||
domains: Set[str] = set()
|
||||
urls: Set[str] = set()
|
||||
emails: Set[str] = set()
|
||||
|
||||
for s in strings:
|
||||
for ip in IP_RE.findall(s):
|
||||
if not any(ip.startswith(p) for p in PRIVATE_IP_PREFIXES):
|
||||
octets = ip.split(".")
|
||||
if not (int(octets[0]) == 172 and 16 <= int(octets[1]) <= 31):
|
||||
ips.add(ip)
|
||||
for d in DOMAIN_RE.findall(s):
|
||||
if not any(d.lower().endswith(sfx) for sfx in FALSE_DOMAIN_SUFFIXES):
|
||||
domains.add(d.lower())
|
||||
for u in URL_RE.findall(s):
|
||||
urls.add(u)
|
||||
for e in EMAIL_RE.findall(s):
|
||||
emails.add(e.lower())
|
||||
|
||||
return {"ips": sorted(ips), "domains": sorted(domains),
|
||||
"urls": sorted(urls), "emails": sorted(emails)}
|
||||
|
||||
|
||||
def extract_host_iocs(strings: List[str]) -> dict:
|
||||
"""Extract file paths, registry keys, and mutexes from strings."""
|
||||
file_paths = set()
|
||||
registry_keys = set()
|
||||
mutexes = set()
|
||||
|
||||
for s in strings:
|
||||
if re.match(r"[A-Z]:\\", s) and len(s) > 5:
|
||||
file_paths.add(s)
|
||||
if re.match(r"(?i)(HKLM|HKCU|HKCR|HKU|HKCC)\\", s):
|
||||
registry_keys.add(s)
|
||||
if re.match(r"(?i)(Global\\|Local\\)", s):
|
||||
mutexes.add(s)
|
||||
|
||||
return {"file_paths": sorted(file_paths)[:30], "registry_keys": sorted(registry_keys)[:20],
|
||||
"mutexes": sorted(mutexes)[:10]}
|
||||
|
||||
|
||||
def run_yara_scan(file_path: str, rules_path: str) -> List[dict]:
|
||||
"""Scan file with YARA rules."""
|
||||
if not yara:
|
||||
return [{"error": "yara-python not installed"}]
|
||||
try:
|
||||
rules = yara.compile(filepath=rules_path)
|
||||
matches = rules.match(file_path)
|
||||
return [{"rule": m.rule, "tags": m.tags, "meta": m.meta,
|
||||
"strings": [(s.identifier, s.instances[0].offset if s.instances else 0)
|
||||
for s in m.strings][:10]}
|
||||
for m in matches]
|
||||
except yara.Error as exc:
|
||||
return [{"error": str(exc)}]
|
||||
|
||||
|
||||
def validate_ioc_virustotal(ioc_value: str, ioc_type: str, api_key: str) -> dict:
|
||||
"""Validate a single IOC against VirusTotal API v3."""
|
||||
if not requests or not api_key:
|
||||
return {"validated": False}
|
||||
endpoints = {"ip": f"https://www.virustotal.com/api/v3/ip_addresses/{ioc_value}",
|
||||
"domain": f"https://www.virustotal.com/api/v3/domains/{ioc_value}",
|
||||
"hash": f"https://www.virustotal.com/api/v3/files/{ioc_value}"}
|
||||
url = endpoints.get(ioc_type)
|
||||
if not url:
|
||||
return {"validated": False}
|
||||
try:
|
||||
resp = requests.get(url, headers={"x-apikey": api_key}, timeout=10)
|
||||
if resp.status_code == 200:
|
||||
stats = resp.json()["data"]["attributes"]["last_analysis_stats"]
|
||||
return {"validated": True, "malicious": stats.get("malicious", 0),
|
||||
"suspicious": stats.get("suspicious", 0)}
|
||||
except Exception:
|
||||
pass
|
||||
return {"validated": False}
|
||||
|
||||
|
||||
def defang_ioc(value: str) -> str:
|
||||
"""Defang an IOC for safe sharing."""
|
||||
return value.replace("http", "hxxp").replace(".", "[.]")
|
||||
|
||||
|
||||
def export_stix_bundle(iocs: dict, sha256: str) -> dict:
|
||||
"""Build a minimal STIX 2.1 bundle from extracted IOCs."""
|
||||
indicators = []
|
||||
ts = datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
indicators.append({"type": "indicator", "spec_version": "2.1",
|
||||
"pattern": f"[file:hashes.'SHA-256' = '{sha256}']",
|
||||
"pattern_type": "stix", "valid_from": ts, "name": "Malware Hash"})
|
||||
for ip in iocs.get("ips", []):
|
||||
indicators.append({"type": "indicator", "spec_version": "2.1",
|
||||
"pattern": f"[ipv4-addr:value = '{ip}']",
|
||||
"pattern_type": "stix", "valid_from": ts, "name": f"C2 IP {ip}"})
|
||||
for domain in iocs.get("domains", [])[:20]:
|
||||
indicators.append({"type": "indicator", "spec_version": "2.1",
|
||||
"pattern": f"[domain-name:value = '{domain}']",
|
||||
"pattern_type": "stix", "valid_from": ts, "name": f"C2 Domain {domain}"})
|
||||
return {"type": "bundle", "id": "bundle--ioc-extract", "objects": indicators}
|
||||
|
||||
|
||||
def export_csv(iocs: dict, hashes: dict, output_path: str) -> None:
|
||||
"""Export IOCs to CSV for SIEM ingestion."""
|
||||
with open(output_path, "w", newline="", encoding="utf-8") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow(["type", "value", "context", "confidence"])
|
||||
writer.writerow(["sha256", hashes["sha256"], "malware_sample", "high"])
|
||||
writer.writerow(["md5", hashes["md5"], "malware_sample", "high"])
|
||||
for ip in iocs.get("ips", []):
|
||||
writer.writerow(["ipv4", ip, "c2_server", "high"])
|
||||
for d in iocs.get("domains", []):
|
||||
writer.writerow(["domain", d, "c2_domain", "medium"])
|
||||
for u in iocs.get("urls", []):
|
||||
writer.writerow(["url", u, "c2_url", "medium"])
|
||||
logger.info("Exported IOCs to %s", output_path)
|
||||
|
||||
|
||||
def run_extraction(sample_path: str, output_dir: str, yara_rules: str = "",
|
||||
vt_key: str = "") -> dict:
|
||||
"""Run full IOC extraction pipeline."""
|
||||
report = {"analysis_date": datetime.utcnow().isoformat(), "sample": sample_path}
|
||||
|
||||
report["hashes"] = compute_hashes(sample_path)
|
||||
report["pe_metadata"] = extract_pe_metadata(sample_path)
|
||||
|
||||
strings = extract_strings(sample_path)
|
||||
report["string_count"] = len(strings)
|
||||
report["network_iocs"] = extract_network_iocs(strings)
|
||||
report["host_iocs"] = extract_host_iocs(strings)
|
||||
|
||||
if yara_rules and os.path.isfile(yara_rules):
|
||||
report["yara_matches"] = run_yara_scan(sample_path, yara_rules)
|
||||
else:
|
||||
report["yara_matches"] = []
|
||||
|
||||
if vt_key:
|
||||
vt_result = validate_ioc_virustotal(report["hashes"]["sha256"], "hash", vt_key)
|
||||
report["virustotal"] = vt_result
|
||||
|
||||
stix = export_stix_bundle(report["network_iocs"], report["hashes"]["sha256"])
|
||||
stix_path = os.path.join(output_dir, "iocs_stix.json")
|
||||
with open(stix_path, "w") as f:
|
||||
json.dump(stix, f, indent=2)
|
||||
|
||||
export_csv(report["network_iocs"], report["hashes"], os.path.join(output_dir, "iocs.csv"))
|
||||
|
||||
report["summary"] = {
|
||||
"ips": len(report["network_iocs"]["ips"]),
|
||||
"domains": len(report["network_iocs"]["domains"]),
|
||||
"urls": len(report["network_iocs"]["urls"]),
|
||||
"file_paths": len(report["host_iocs"]["file_paths"]),
|
||||
"registry_keys": len(report["host_iocs"]["registry_keys"]),
|
||||
"yara_hits": len(report["yara_matches"]),
|
||||
}
|
||||
return report
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Malware IOC Extraction Agent")
|
||||
parser.add_argument("--sample", required=True, help="Path to malware sample")
|
||||
parser.add_argument("--yara-rules", default="", help="Path to YARA rules file")
|
||||
parser.add_argument("--vt-key", default="", help="VirusTotal API key")
|
||||
parser.add_argument("--output-dir", default=".", help="Output directory")
|
||||
parser.add_argument("--output", default="ioc_report.json")
|
||||
args = parser.parse_args()
|
||||
|
||||
os.makedirs(args.output_dir, exist_ok=True)
|
||||
report = run_extraction(args.sample, args.output_dir, args.yara_rules, args.vt_key)
|
||||
out_path = os.path.join(args.output_dir, args.output)
|
||||
with open(out_path, "w") as f:
|
||||
json.dump(report, f, indent=2, default=str)
|
||||
logger.info("Report saved to %s", out_path)
|
||||
print(json.dumps(report, indent=2, default=str))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user