reverse-skills-max: initial public release (restored, redacted, hardened)
This commit is contained in:
baradok609hecl1pq6qi
157
reverse-skill-repo/skills/case-review/SKILL.md
Normal file
157
reverse-skill-repo/skills/case-review/SKILL.md
Normal file
@@ -0,0 +1,157 @@
|
||||
---
|
||||
name: case-review
|
||||
description: Reviews a reverse-skill case package for scope readiness, Evidence to Finding to Path traceability, work item coverage, timeline references, and optional artifact hash integrity before report handoff.
|
||||
---
|
||||
|
||||
# Evidence Graph Review
|
||||
|
||||
Use this skill when a reverse engineering, forensics, CTF, or authorized security case needs a defensible handoff. It audits the existing `work/<case>/` package without changing the case or touching a target.
|
||||
|
||||
## Scope
|
||||
|
||||
This skill covers:
|
||||
|
||||
- Scope metadata and target-activity readiness
|
||||
- Evidence record structure and reproducibility fields
|
||||
- References from work items and timeline entries to Evidence
|
||||
- Structured Findings and Paths in report Markdown
|
||||
- Optional SHA-256 verification for case-local artifacts
|
||||
- A Markdown or JSON review result for a report handoff
|
||||
|
||||
It MUST NOT perform reconnaissance, exploitation, dynamic instrumentation, or target changes. Those actions belong to the routed analysis skill and require the case scope gate.
|
||||
|
||||
## ACTION REQUIRED
|
||||
|
||||
1. `NOW`: read `../field-journal/precedent-reverse.md` and confirm that this is a review of an existing authorized case package.
|
||||
2. `NOW`: confirm the case path and choose read-only review mode.
|
||||
3. `NEXT`: read `../tool-index.md`; this skill uses only Python 3 standard library and does not require bootstrap.
|
||||
4. `NEXT`: run `python3 scripts/review_case.py <case-root> --format markdown`.
|
||||
5. `ACT`: resolve every error, then rerun the review before claiming a handoff is complete.
|
||||
|
||||
## Tool dependencies
|
||||
|
||||
| Tool | Required | Purpose | Auto-bootstrap |
|
||||
|------|----------|---------|---------------|
|
||||
| Python 3.9+ | Yes | Runs the read-only case review script | No, use the platform Python installation |
|
||||
|
||||
No network access or third-party package is required.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Intake
|
||||
|
||||
Run the review against the existing case directory:
|
||||
|
||||
```bash
|
||||
python3 skills/case-review/scripts/review_case.py work/<case> --format markdown
|
||||
```
|
||||
|
||||
Confirm that `scope.md`, `timeline.md`, `workitems.md`, and `evidence/` are present. A non-strict review reports scope warnings while a strict review treats warnings as handoff blockers.
|
||||
|
||||
## 建议下一步（选一个编号）
|
||||
|
||||
1. 修复 scope.md 中的授权、范围或 network_profile 字段
|
||||
2. 继续检查 Evidence 记录的可复现命令和来源
|
||||
3. 导出当前 review 结果并附到阶段性报告
|
||||
4. 换 JSON 输出接入 CI 或其他审查工具
|
||||
5. 暂停，先确认审查范围
|
||||
|
||||
### Phase 2: Traceability
|
||||
|
||||
Review the checks for:
|
||||
|
||||
- Evidence IDs that do not exist
|
||||
- Findings without `evidence_ids`
|
||||
- Paths without an allowed `path_type` or Evidence reference
|
||||
- Work items and timeline entries pointing to unknown Evidence
|
||||
- Unlinked Evidence records
|
||||
- Validated Findings with low confidence
|
||||
|
||||
An offline observation may use `repro_command: n/a` only when its `notes` field explicitly documents the offline limitation.
|
||||
|
||||
Use JSON when another tool needs stable fields:
|
||||
|
||||
```bash
|
||||
python3 skills/case-review/scripts/review_case.py work/<case> --format json
|
||||
```
|
||||
|
||||
## 建议下一步（选一个编号）
|
||||
|
||||
1. 补写缺失的 Evidence，并保留原始命令
|
||||
2. 将候选 Finding 绑定到 Evidence 后重新审查
|
||||
3. 为调用链或攻击链补充 P-id 和 Path 步骤
|
||||
4. 生成 Markdown handoff summary
|
||||
5. 换回 PRIMARY skill 继续分析
|
||||
|
||||
### Phase 3: Fixity verification
|
||||
|
||||
When an Evidence record contains both `content_hash` and `artifact_path`, verify the case-local artifact:
|
||||
|
||||
```bash
|
||||
python3 skills/case-review/scripts/review_case.py work/<case> --verify-hashes --strict
|
||||
```
|
||||
|
||||
The script accepts `sha256:<64 hex characters>` and checks that the artifact remains inside the case root. A hash mismatch is a hard failure.
|
||||
|
||||
The PowerShell Evidence helper can record a hash while appending a record:
|
||||
|
||||
```powershell
|
||||
powershell -File skills/scripts/append-evidence.ps1 -CaseRoot work\<case> -Id E-001 -Title "Sample hash" -ReproCommand "sha256sum evidence/sample.bin" -ArtifactPath "evidence\sample.bin"
|
||||
```
|
||||
|
||||
## 建议下一步（选一个编号）
|
||||
|
||||
1. 修复 hash mismatch 或替换已污染的工作副本
|
||||
2. 为未固定的原始文件补充 SHA-256 和 artifact_path
|
||||
3. 继续进入报告生成阶段
|
||||
4. 导出 JSON 结果供 CI 保存
|
||||
5. 暂停并请求人工复核
|
||||
|
||||
### Phase 4: Handoff
|
||||
|
||||
Use strict mode before a final report or specialist handoff:
|
||||
|
||||
```bash
|
||||
python3 skills/case-review/scripts/review_case.py work/<case> --strict --format markdown > work/<case>/report/case-review.md
|
||||
```
|
||||
|
||||
The command is read-only with respect to the case unless shell redirection is explicitly used to save its output. The review is not legal advice and does not replace organizational evidence handling procedures.
|
||||
|
||||
## 建议下一步（选一个编号）
|
||||
|
||||
1. 将通过的 review 结果交给 `docs-generator/` 生成正式报告
|
||||
2. 回到 PRIMARY skill 补齐新的分析证据
|
||||
3. 归档 Markdown 和 JSON review 结果
|
||||
4. 暂停并请求人工复核
|
||||
|
||||
## Language behavior contract
|
||||
|
||||
- Internal reasoning, tool selection, and phase control: English.
|
||||
- User-visible messages, section labels, reports, and next-step menus: Chinese unless the user requests another language.
|
||||
- Default bilingual labels place Chinese first and English second, separated by `/`.
|
||||
|
||||
## Bootstrap boundary
|
||||
|
||||
This skill has no third-party dependency. If Python 3 is unavailable, the only allowed recovery action is the repository bootstrap path when a Python capability is registered for the current platform. If no such capability is registered, stop and report the missing runtime. Do not guess executable paths, download packages, or perform a manual install from inside this skill.
|
||||
|
||||
## Routing context
|
||||
|
||||
**Upstream entry**: any reverse, forensics, CTF, or authorized security skill that has produced a case package.
|
||||
|
||||
**Downstream exit**: `docs-generator/` for a formal report, or the original PRIMARY skill when the graph is incomplete.
|
||||
|
||||
**Related modules**: `ops/evidence-finding-path.md`, `ops/timeline-workitem.md`, `digital-forensics/`, `reverse-engineering/`, and `docs-generator/`.
|
||||
|
||||
## References
|
||||
|
||||
- [NIST SP 800-86: Guide to Integrating Forensic Techniques into Incident Response](https://csrc.nist.gov/pubs/sp/800/86/final)
|
||||
- [SWGDE Best Practices for Computer Forensic Acquisitions](https://www.swgde.org/documents/published-complete-listing/17-f-002-2-1/)
|
||||
- [SWGDE Best Practices for Archiving Digital and Multimedia Evidence](https://www.swgde.org/documents/published-complete-listing/19-f-003-best-practices-for-archiving-digital-and-multimedia-evidence/)
|
||||
|
||||
## 任务完成自检
|
||||
|
||||
- [ ] 我是否审查了 scope.md、timeline.md、workitems.md 和 evidence/？
|
||||
- [ ] 所有 Finding 是否引用了现存 Evidence？
|
||||
- [ ] 所有 Path 是否包含合法 path_type 和 Evidence 引用？
|
||||
- [ ] 是否执行了 hash verification，或记录了未执行原因？
|
||||
- [ ] 是否以 strict 模式重新运行并保存了 review 结果？
|
||||
453
reverse-skill-repo/skills/case-review/scripts/review_case.py
Normal file
453
reverse-skill-repo/skills/case-review/scripts/review_case.py
Normal file
@@ -0,0 +1,453 @@
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
EVIDENCE_ID = re.compile(r"\bE-[A-Za-z0-9][A-Za-z0-9_-]*\b")
|
||||
WORKITEM_ID = re.compile(r"\bWI-[A-Za-z0-9][A-Za-z0-9_-]*\b")
|
||||
FINDING_ID = re.compile(r"\bF-[A-Za-z0-9][A-Za-z0-9_-]*\b")
|
||||
PATH_ID = re.compile(r"\bP-[A-Za-z0-9][A-Za-z0-9_-]*\b")
|
||||
FIELD_LINE = re.compile(r"^\s*-\s+([A-Za-z0-9_]+):\s*(.*)$")
|
||||
SECTION_HEADING = re.compile(r"^#{2,6}\s+(.+?)\s*$")
|
||||
EVIDENCE_HEADING = re.compile(r"^###\s+(E-[A-Za-z0-9][A-Za-z0-9_-]*)\b", re.MULTILINE)
|
||||
FINDING_HEADING = re.compile(r"^###\s+(F-[A-Za-z0-9][A-Za-z0-9_-]*)\b", re.MULTILINE)
|
||||
PATH_HEADING = re.compile(r"^###\s+(P-[A-Za-z0-9][A-Za-z0-9_-]*)\b", re.MULTILINE)
|
||||
|
||||
SEVERITIES = {"critical", "high", "medium", "low", "info", "n/a", "n/a_re"}
|
||||
EVIDENCE_STATUSES = {"observed", "candidate", "validated", "false_positive", "accepted_risk"}
|
||||
WORKITEM_STATUSES = {"pending", "in_progress", "blocked", "done", "completed", "cancelled"}
|
||||
PATH_TYPES = {"attack", "callflow", "solve"}
|
||||
NETWORK_MODES = {"offline", "lab_only", "authorized_target_only", "unrestricted_lab"}
|
||||
|
||||
|
||||
def field_value(text, name):
|
||||
lines = text.splitlines()
|
||||
for index, line in enumerate(lines):
|
||||
match = FIELD_LINE.match(line)
|
||||
if not match or match.group(1) != name:
|
||||
continue
|
||||
value = match.group(2).strip()
|
||||
if value != "|":
|
||||
return value
|
||||
block = []
|
||||
for continuation in lines[index + 1:]:
|
||||
if FIELD_LINE.match(continuation):
|
||||
break
|
||||
if SECTION_HEADING.match(continuation):
|
||||
break
|
||||
if continuation.strip():
|
||||
block.append(continuation.strip())
|
||||
return "\n".join(block).strip()
|
||||
return ""
|
||||
|
||||
|
||||
def section_text(text, title):
|
||||
pattern = re.compile(
|
||||
r"(?ms)^##\s+" + re.escape(title) + r"\s*$\n?(.*?)(?=^##\s|\Z)"
|
||||
)
|
||||
match = pattern.search(text)
|
||||
return match.group(1) if match else ""
|
||||
|
||||
|
||||
def ids_in(value, pattern):
|
||||
return sorted(set(pattern.findall(value or "")))
|
||||
|
||||
|
||||
def issue(issues, level, code, message, path=""):
|
||||
issues.append({"level": level, "code": code, "message": message, "path": path})
|
||||
|
||||
|
||||
def relative_path(root, path):
|
||||
try:
|
||||
return str(path.resolve().relative_to(root.resolve()))
|
||||
except ValueError:
|
||||
return str(path)
|
||||
|
||||
|
||||
def split_table_row(line):
|
||||
stripped = line.strip()
|
||||
if not stripped.startswith("|") or not stripped.endswith("|"):
|
||||
return []
|
||||
return [cell.strip() for cell in stripped[1:-1].split("|")]
|
||||
|
||||
|
||||
def parse_workitems(root, issues):
|
||||
path = root / "workitems.md"
|
||||
records = {}
|
||||
references = []
|
||||
if not path.is_file():
|
||||
issue(issues, "error", "workitems.missing", "workitems.md is missing", "workitems.md")
|
||||
return records, references
|
||||
|
||||
for line_number, line in enumerate(path.read_text(encoding="utf-8-sig").splitlines(), 1):
|
||||
cells = split_table_row(line)
|
||||
if len(cells) < 8 or not WORKITEM_ID.fullmatch(cells[0]):
|
||||
continue
|
||||
workitem_id = cells[0]
|
||||
status = cells[5].lower()
|
||||
if status not in WORKITEM_STATUSES:
|
||||
issue(
|
||||
issues,
|
||||
"error",
|
||||
"workitem.status",
|
||||
"unsupported work item status: " + status,
|
||||
"workitems.md:" + str(line_number),
|
||||
)
|
||||
evidence_ids = ids_in(cells[6], EVIDENCE_ID)
|
||||
records[workitem_id] = {"status": status, "evidence_ids": evidence_ids}
|
||||
references.extend((evidence_id, "workitems.md:" + str(line_number)) for evidence_id in evidence_ids)
|
||||
|
||||
if not records:
|
||||
issue(issues, "warning", "workitems.empty", "no work item rows were found", "workitems.md")
|
||||
return records, references
|
||||
|
||||
|
||||
def parse_timeline(root, issues):
|
||||
path = root / "timeline.md"
|
||||
references = []
|
||||
events = 0
|
||||
if not path.is_file():
|
||||
issue(issues, "error", "timeline.missing", "timeline.md is missing", "timeline.md")
|
||||
return events, references
|
||||
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
for index, line in enumerate(lines):
|
||||
if not line.startswith("## ") or "|" not in line:
|
||||
continue
|
||||
events += 1
|
||||
end = len(lines)
|
||||
for next_index in range(index + 1, len(lines)):
|
||||
if lines[next_index].startswith("## "):
|
||||
end = next_index
|
||||
break
|
||||
body = "\n".join(lines[index:end])
|
||||
evidence_ids = ids_in(field_value(body, "evidence_ids"), EVIDENCE_ID)
|
||||
references.extend((evidence_id, "timeline.md:" + str(index + 1)) for evidence_id in evidence_ids)
|
||||
|
||||
if events == 0:
|
||||
issue(issues, "warning", "timeline.empty", "no append-only timeline events were found", "timeline.md")
|
||||
return events, references
|
||||
|
||||
|
||||
def parse_scope(root, issues, strict):
|
||||
path = root / "scope.md"
|
||||
result = {"auth_status": "", "network_mode": "", "ready_for_act": "", "assets": []}
|
||||
if not path.is_file():
|
||||
issue(issues, "error", "scope.missing", "scope.md is missing", "scope.md")
|
||||
return result
|
||||
|
||||
text = path.read_text(encoding="utf-8-sig")
|
||||
auth = section_text(text, "auth")
|
||||
scope_section = section_text(text, "in_scope")
|
||||
network = section_text(text, "network_profile")
|
||||
signoff = section_text(text, "signoff")
|
||||
result["auth_status"] = field_value(auth, "status").lower()
|
||||
result["network_mode"] = field_value(network, "mode").lower()
|
||||
result["ready_for_act"] = field_value(signoff, "ready_for_act").lower()
|
||||
|
||||
# Only a top-level "- field:" ends the assets block. Indented Windows
|
||||
# paths such as " - D:\\repo" contain a colon but are asset values.
|
||||
assets_match = re.search(r"(?ms)^\s*-\s+assets:\s*\n(?P<body>.*?)(?=^-\s+[A-Za-z0-9_]+:|\Z)", scope_section)
|
||||
if assets_match:
|
||||
result["assets"] = [
|
||||
line.strip()[2:].strip()
|
||||
for line in assets_match.group("body").splitlines()
|
||||
if re.match(r"^\s+-\s+\S+", line) and line.strip()[2:].strip() != "[]"
|
||||
]
|
||||
|
||||
if not result["auth_status"]:
|
||||
issue(issues, "error", "scope.auth_missing", "auth.status is missing", "scope.md")
|
||||
elif result["auth_status"] not in {"pending", "granted", "denied", "unknown"}:
|
||||
issue(issues, "error", "scope.auth_invalid", "unsupported auth.status: " + result["auth_status"], "scope.md")
|
||||
|
||||
if not result["network_mode"]:
|
||||
issue(issues, "error", "scope.network_missing", "network_profile.mode is missing", "scope.md")
|
||||
elif result["network_mode"] not in NETWORK_MODES:
|
||||
issue(issues, "error", "scope.network_invalid", "unsupported network mode: " + result["network_mode"], "scope.md")
|
||||
|
||||
if not result["ready_for_act"]:
|
||||
issue(issues, "error", "scope.ready_missing", "signoff.ready_for_act is missing", "scope.md")
|
||||
elif result["ready_for_act"] not in {"true", "false"}:
|
||||
issue(issues, "error", "scope.ready_invalid", "ready_for_act must be true or false", "scope.md")
|
||||
|
||||
if result["network_mode"] != "offline" and not result["assets"]:
|
||||
issue(issues, "error", "scope.assets_missing", "in_scope.assets is empty for a network case", "scope.md")
|
||||
|
||||
if result["auth_status"] != "granted" or result["ready_for_act"] != "true":
|
||||
message = "scope is not ready for target ACT"
|
||||
if strict and result["network_mode"] != "offline":
|
||||
issue(issues, "error", "scope.not_ready", message, "scope.md")
|
||||
else:
|
||||
issue(issues, "warning", "scope.not_ready", message, "scope.md")
|
||||
return result
|
||||
|
||||
|
||||
def report_sections(text, heading_pattern):
|
||||
matches = list(heading_pattern.finditer(text))
|
||||
sections = []
|
||||
for index, match in enumerate(matches):
|
||||
end = len(text)
|
||||
for next_match in re.finditer(r"(?m)^#{1,3}\s+", text[match.end():]):
|
||||
end = match.end() + next_match.start()
|
||||
break
|
||||
sections.append((match.group(1), text[match.start():end]))
|
||||
return sections
|
||||
|
||||
|
||||
def parse_reports(root, issues):
|
||||
finding_records = []
|
||||
path_records = []
|
||||
references = []
|
||||
report_root = root / "report"
|
||||
report_files = sorted(report_root.rglob("*.md")) if report_root.is_dir() else []
|
||||
if not report_files:
|
||||
issue(issues, "warning", "report.missing", "no Markdown report was found", "report")
|
||||
return finding_records, path_records, references
|
||||
|
||||
for report_path in report_files:
|
||||
text = report_path.read_text(encoding="utf-8-sig")
|
||||
for finding_id, body in report_sections(text, FINDING_HEADING):
|
||||
status = field_value(body, "status").lower()
|
||||
confidence = field_value(body, "confidence").lower()
|
||||
evidence_ids = ids_in(field_value(body, "evidence_ids"), EVIDENCE_ID)
|
||||
required = ("severity", "evidence_ids", "confidence", "location", "status")
|
||||
for field in required:
|
||||
if not field_value(body, field):
|
||||
issue(issues, "error", "finding.field_missing", finding_id + " is missing " + field, relative_path(root, report_path))
|
||||
if status not in {"candidate", "validated", "false_positive", "accepted_risk"}:
|
||||
issue(issues, "error", "finding.status", finding_id + " has unsupported status", relative_path(root, report_path))
|
||||
if status == "validated" and confidence == "low":
|
||||
issue(issues, "error", "finding.confidence", finding_id + " is validated with low confidence", relative_path(root, report_path))
|
||||
if not evidence_ids:
|
||||
issue(issues, "error", "finding.evidence_missing", finding_id + " has no evidence_ids", relative_path(root, report_path))
|
||||
references.extend((evidence_id, relative_path(root, report_path)) for evidence_id in evidence_ids)
|
||||
finding_records.append({"id": finding_id, "status": status, "evidence_ids": evidence_ids, "path": relative_path(root, report_path)})
|
||||
|
||||
for path_id, body in report_sections(text, PATH_HEADING):
|
||||
path_type = field_value(body, "path_type").lower()
|
||||
evidence_ids = ids_in(body, EVIDENCE_ID)
|
||||
if path_type not in PATH_TYPES:
|
||||
issue(issues, "error", "path.type", path_id + " has unsupported path_type", relative_path(root, report_path))
|
||||
if not evidence_ids:
|
||||
issue(issues, "error", "path.evidence_missing", path_id + " has no evidence reference", relative_path(root, report_path))
|
||||
references.extend((evidence_id, relative_path(root, report_path)) for evidence_id in evidence_ids)
|
||||
path_records.append({"id": path_id, "path_type": path_type, "evidence_ids": evidence_ids, "path": relative_path(root, report_path)})
|
||||
return finding_records, path_records, references
|
||||
|
||||
|
||||
def normalize_hash(value):
|
||||
normalized = value.strip().lower()
|
||||
if normalized.startswith("sha256:"):
|
||||
normalized = normalized[7:]
|
||||
if re.fullmatch(r"[0-9a-f]{64}", normalized):
|
||||
return normalized
|
||||
return ""
|
||||
|
||||
|
||||
def is_within(root, path):
|
||||
try:
|
||||
return os.path.commonpath([str(root), str(path)]) == str(root)
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def sha256_file(path):
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def verify_artifact(root, artifact_path, expected_hash, issues, display_path):
|
||||
if not artifact_path:
|
||||
issue(issues, "warning", "artifact.path_missing", "content_hash is recorded without artifact_path", display_path)
|
||||
return
|
||||
candidate = Path(artifact_path)
|
||||
if not candidate.is_absolute():
|
||||
candidate = root / candidate
|
||||
candidate = candidate.resolve()
|
||||
if not is_within(root.resolve(), candidate):
|
||||
issue(issues, "warning", "artifact.outside_case", "artifact_path points outside the case root", display_path)
|
||||
return
|
||||
if not candidate.is_file():
|
||||
issue(issues, "error", "artifact.missing", "artifact_path does not exist", display_path)
|
||||
return
|
||||
digest = sha256_file(candidate)
|
||||
if digest != expected_hash:
|
||||
issue(issues, "error", "artifact.hash_mismatch", "artifact SHA-256 does not match content_hash", display_path)
|
||||
|
||||
|
||||
def parse_evidence(root, workitems, issues, verify_hashes):
|
||||
evidence_root = root / "evidence"
|
||||
records = {}
|
||||
references = []
|
||||
if not evidence_root.is_dir():
|
||||
issue(issues, "error", "evidence.dir_missing", "evidence directory is missing", "evidence")
|
||||
return records, references
|
||||
|
||||
evidence_files = sorted(path for path in evidence_root.glob("E-*.md") if path.name != "INDEX.md")
|
||||
if not evidence_files:
|
||||
issue(issues, "warning", "evidence.empty", "no Evidence records were found", "evidence")
|
||||
|
||||
for path in evidence_files:
|
||||
evidence_id = path.stem
|
||||
text = path.read_text(encoding="utf-8-sig")
|
||||
heading = EVIDENCE_HEADING.search(text)
|
||||
if not heading:
|
||||
issue(issues, "error", "evidence.heading_missing", "Evidence record has no matching heading", relative_path(root, path))
|
||||
elif heading.group(1) != evidence_id:
|
||||
issue(issues, "error", "evidence.heading_mismatch", "Evidence heading does not match its filename", relative_path(root, path))
|
||||
|
||||
severity = field_value(text, "severity").lower()
|
||||
status = field_value(text, "status").lower()
|
||||
repro_command = field_value(text, "repro_command")
|
||||
notes = field_value(text, "notes").lower()
|
||||
linked_workitems = ids_in(field_value(text, "linked_workitem"), WORKITEM_ID)
|
||||
content_hash_value = field_value(text, "content_hash")
|
||||
artifact_path = field_value(text, "artifact_path")
|
||||
if severity not in SEVERITIES:
|
||||
issue(issues, "error", "evidence.severity", "unsupported severity: " + severity, relative_path(root, path))
|
||||
if status not in EVIDENCE_STATUSES:
|
||||
issue(issues, "error", "evidence.status", "unsupported status: " + status, relative_path(root, path))
|
||||
offline_note = any(marker in notes for marker in ("offline", "离线", "not applicable"))
|
||||
if not repro_command or (repro_command.lower() in {"n/a", "n/a_re"} and not offline_note):
|
||||
issue(issues, "error", "evidence.repro_missing", "Evidence requires a reproducible command or documented offline limitation", relative_path(root, path))
|
||||
expected_hash = normalize_hash(content_hash_value) if content_hash_value.lower() != "n/a" else ""
|
||||
if content_hash_value.lower() != "n/a" and not expected_hash:
|
||||
issue(issues, "error", "evidence.hash_invalid", "content_hash must be SHA-256", relative_path(root, path))
|
||||
if verify_hashes and expected_hash:
|
||||
verify_artifact(root, artifact_path, expected_hash, issues, relative_path(root, path))
|
||||
for workitem_id in linked_workitems:
|
||||
if workitem_id not in workitems:
|
||||
issue(issues, "error", "evidence.workitem_missing", "linked work item does not exist: " + workitem_id, relative_path(root, path))
|
||||
records[evidence_id] = {"severity": severity, "status": status, "artifact_path": artifact_path}
|
||||
|
||||
return records, references
|
||||
|
||||
|
||||
def build_traceability(evidence_ids, references):
|
||||
graph = {evidence_id: {"workitems": 0, "timeline": 0, "reports": 0} for evidence_id in sorted(evidence_ids)}
|
||||
for evidence_id, source in references:
|
||||
if evidence_id not in graph:
|
||||
continue
|
||||
if source.startswith("workitems.md:"):
|
||||
graph[evidence_id]["workitems"] += 1
|
||||
elif source.startswith("timeline.md:"):
|
||||
graph[evidence_id]["timeline"] += 1
|
||||
else:
|
||||
graph[evidence_id]["reports"] += 1
|
||||
return graph
|
||||
|
||||
|
||||
def review_case(case_root, strict=False, verify_hashes=False):
|
||||
root = Path(case_root).expanduser().resolve()
|
||||
issues = []
|
||||
if not root.is_dir():
|
||||
issue(issues, "error", "case.missing", "case root does not exist", str(root))
|
||||
return {
|
||||
"status": "FAIL",
|
||||
"case_root": str(root),
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"summary": {"errors": 1, "warnings": 0, "evidence": 0, "workitems": 0, "timeline_events": 0, "findings": 0, "paths": 0},
|
||||
"issues": issues,
|
||||
"traceability": {},
|
||||
}
|
||||
|
||||
workitems, workitem_refs = parse_workitems(root, issues)
|
||||
timeline_events, timeline_refs = parse_timeline(root, issues)
|
||||
scope = parse_scope(root, issues, strict)
|
||||
evidence, evidence_refs = parse_evidence(root, workitems, issues, verify_hashes)
|
||||
findings, paths, report_refs = parse_reports(root, issues)
|
||||
references = workitem_refs + timeline_refs + evidence_refs + report_refs
|
||||
|
||||
for evidence_id, source in references:
|
||||
if evidence_id not in evidence:
|
||||
issue(issues, "error", "reference.unknown_evidence", "reference points to missing Evidence: " + evidence_id, source)
|
||||
|
||||
traceability = build_traceability(evidence, references)
|
||||
for evidence_id, links in traceability.items():
|
||||
if not any(links.values()):
|
||||
issue(issues, "warning", "evidence.unlinked", "Evidence is not referenced by a work item, timeline, or report", "evidence/" + evidence_id + ".md")
|
||||
|
||||
errors = sum(1 for item in issues if item["level"] == "error")
|
||||
warnings = sum(1 for item in issues if item["level"] == "warning")
|
||||
status = "FAIL" if errors or (strict and warnings) else "WARN" if warnings else "PASS"
|
||||
return {
|
||||
"status": status,
|
||||
"case_root": str(root),
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"scope": scope,
|
||||
"summary": {
|
||||
"errors": errors,
|
||||
"warnings": warnings,
|
||||
"evidence": len(evidence),
|
||||
"workitems": len(workitems),
|
||||
"timeline_events": timeline_events,
|
||||
"findings": len(findings),
|
||||
"paths": len(paths),
|
||||
},
|
||||
"issues": issues,
|
||||
"traceability": traceability,
|
||||
}
|
||||
|
||||
|
||||
def render_markdown(report):
|
||||
summary = report["summary"]
|
||||
lines = [
|
||||
"# Case review",
|
||||
"",
|
||||
"- status: " + report["status"],
|
||||
"- case_root: " + report["case_root"],
|
||||
"- generated_at: " + report["generated_at"],
|
||||
"",
|
||||
"## Summary",
|
||||
"",
|
||||
"| Metric | Value |",
|
||||
"|---|---:|",
|
||||
]
|
||||
for key in ("errors", "warnings", "evidence", "workitems", "timeline_events", "findings", "paths"):
|
||||
lines.append("| " + key + " | " + str(summary[key]) + " |")
|
||||
|
||||
lines.extend(["", "## Checks", "", "| Level | Code | Location | Detail |", "|---|---|---|---|"])
|
||||
if report["issues"]:
|
||||
for item in report["issues"]:
|
||||
location = item["path"].replace("|", "\\|")
|
||||
detail = item["message"].replace("|", "\\|")
|
||||
lines.append("| " + item["level"] + " | " + item["code"] + " | " + location + " | " + detail + " |")
|
||||
else:
|
||||
lines.append("| pass | none | n/a | No review issues found |")
|
||||
|
||||
lines.extend(["", "## Traceability", "", "| Evidence | Work items | Timeline | Reports |", "|---|---:|---:|---:|"])
|
||||
if report["traceability"]:
|
||||
for evidence_id, links in report["traceability"].items():
|
||||
lines.append(
|
||||
"| " + evidence_id + " | " + str(links["workitems"]) + " | " + str(links["timeline"]) + " | " + str(links["reports"]) + " |"
|
||||
)
|
||||
else:
|
||||
lines.append("| n/a | 0 | 0 | 0 |")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description="Review a reverse-skill case for scope and Evidence graph integrity")
|
||||
parser.add_argument("case_root", help="Path to a work/<case> directory")
|
||||
parser.add_argument("--format", choices=("markdown", "json"), default="markdown")
|
||||
parser.add_argument("--strict", action="store_true", help="Treat warnings as a failed review")
|
||||
parser.add_argument("--verify-hashes", action="store_true", help="Verify SHA-256 content_hash values against artifact_path")
|
||||
args = parser.parse_args(argv)
|
||||
report = review_case(args.case_root, strict=args.strict, verify_hashes=args.verify_hashes)
|
||||
if args.format == "json":
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
else:
|
||||
print(render_markdown(report), end="")
|
||||
return 1 if report["status"] == "FAIL" else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
197
reverse-skill-repo/skills/case-review/tests/test_review_case.py
Normal file
197
reverse-skill-repo/skills/case-review/tests/test_review_case.py
Normal file
@@ -0,0 +1,197 @@
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "review_case.py"
|
||||
SPEC = importlib.util.spec_from_file_location("review_case", SCRIPT_PATH)
|
||||
REVIEW_CASE = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(REVIEW_CASE)
|
||||
|
||||
|
||||
class ReviewCaseTests(unittest.TestCase):
|
||||
def write_case(self, root, include_report=True):
|
||||
evidence_dir = root / "evidence"
|
||||
report_dir = root / "report"
|
||||
evidence_dir.mkdir(parents=True)
|
||||
if include_report:
|
||||
report_dir.mkdir(parents=True)
|
||||
|
||||
artifact = evidence_dir / "sample.bin"
|
||||
artifact.write_bytes(b"reverse-skill case review fixture")
|
||||
digest = hashlib.sha256(artifact.read_bytes()).hexdigest()
|
||||
|
||||
(root / "scope.md").write_text(
|
||||
"""# Case Scope
|
||||
|
||||
## auth
|
||||
- status: granted
|
||||
- basis: lab_only
|
||||
|
||||
## in_scope
|
||||
- assets:
|
||||
- sample.bin
|
||||
|
||||
## network_profile
|
||||
- mode: offline
|
||||
|
||||
## signoff
|
||||
- ready_for_act: true
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(root / "workitems.md").write_text(
|
||||
"""# Work Items
|
||||
|
||||
| ID | title | role | targets | surface | status | evidence | notes |
|
||||
|----|-------|------|---------|---------|--------|----------|-------|
|
||||
| WI-001 | Recover sample behavior | cre | sample.bin | binary | done | E-001 | |
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(root / "timeline.md").write_text(
|
||||
"""# Timeline (append-only)
|
||||
|
||||
## 2026-08-02T00:00:00Z | cre | static
|
||||
- action: inspect local sample
|
||||
- evidence_ids: [E-001]
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(evidence_dir / "E-001.md").write_text(
|
||||
"""### E-001
|
||||
- title: Sample hash
|
||||
- severity: info
|
||||
- status: observed
|
||||
- content_hash: sha256:%s
|
||||
- artifact_path: evidence/sample.bin
|
||||
- linked_workitem: WI-001
|
||||
- repro_command: sha256sum evidence/sample.bin
|
||||
- raw_excerpt: |
|
||||
fixture
|
||||
""" % digest,
|
||||
encoding="utf-8",
|
||||
)
|
||||
if include_report:
|
||||
(report_dir / "analysis.md").write_text(
|
||||
"""### F-001
|
||||
- title: Sample behavior recovered
|
||||
- severity: info
|
||||
- category: reverse_algo
|
||||
- status: validated
|
||||
- evidence_ids: [E-001]
|
||||
- location: sample.bin:0x10
|
||||
- impact: n/a
|
||||
- confidence: high
|
||||
- repro_steps: |
|
||||
1. Run the fixture command.
|
||||
- remediation: n/a
|
||||
|
||||
### P-001
|
||||
- title: Static recovery path
|
||||
- path_type: callflow
|
||||
- start: sample.bin
|
||||
- goal: recovered behavior
|
||||
- steps: |
|
||||
1. Hash the sample with E-001.
|
||||
- residual_risks: n/a
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
def test_valid_case_passes_with_hash_verification(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
self.write_case(root)
|
||||
result = REVIEW_CASE.review_case(root, verify_hashes=True)
|
||||
self.assertEqual(result["status"], "PASS")
|
||||
self.assertEqual(result["summary"]["evidence"], 1)
|
||||
self.assertEqual(result["summary"]["findings"], 1)
|
||||
self.assertEqual(result["summary"]["paths"], 1)
|
||||
|
||||
def test_unknown_report_evidence_fails(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
self.write_case(root)
|
||||
report = root / "report" / "analysis.md"
|
||||
report.write_text(report.read_text(encoding="utf-8").replace("E-001", "E-999"), encoding="utf-8")
|
||||
result = REVIEW_CASE.review_case(root)
|
||||
self.assertEqual(result["status"], "FAIL")
|
||||
self.assertTrue(any(item["code"] == "reference.unknown_evidence" for item in result["issues"]))
|
||||
|
||||
def test_hash_mismatch_fails(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
self.write_case(root)
|
||||
(root / "evidence" / "sample.bin").write_bytes(b"changed fixture")
|
||||
result = REVIEW_CASE.review_case(root, verify_hashes=True)
|
||||
self.assertEqual(result["status"], "FAIL")
|
||||
self.assertTrue(any(item["code"] == "artifact.hash_mismatch" for item in result["issues"]))
|
||||
|
||||
def test_markdown_output_contains_traceability_table(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
self.write_case(root)
|
||||
result = REVIEW_CASE.review_case(root)
|
||||
output = REVIEW_CASE.render_markdown(result)
|
||||
self.assertIn("## Traceability", output)
|
||||
self.assertIn("| E-001 | 1 | 1 | 2 |", output)
|
||||
|
||||
def test_documented_offline_evidence_does_not_require_command(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
self.write_case(root)
|
||||
(root / "evidence" / "E-002.md").write_text(
|
||||
"""### E-002
|
||||
- title: Offline observation
|
||||
- severity: info
|
||||
- status: observed
|
||||
- content_hash: n/a
|
||||
- repro_command: n/a
|
||||
- notes: offline-only observation
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
result = REVIEW_CASE.review_case(root)
|
||||
self.assertFalse(any(item["code"] == "evidence.repro_missing" for item in result["issues"]))
|
||||
|
||||
def test_strict_mode_fails_when_report_is_missing(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
self.write_case(root, include_report=False)
|
||||
result = REVIEW_CASE.review_case(root, strict=True)
|
||||
self.assertEqual(result["status"], "FAIL")
|
||||
self.assertTrue(any(item["code"] == "report.missing" for item in result["issues"]))
|
||||
|
||||
def test_completed_workitem_status_matches_current_case_convention(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
self.write_case(root)
|
||||
workitems = root / "workitems.md"
|
||||
workitems.write_text(
|
||||
workitems.read_text(encoding="utf-8").replace("| done |", "| completed |"),
|
||||
encoding="utf-8",
|
||||
)
|
||||
result = REVIEW_CASE.review_case(root)
|
||||
self.assertFalse(any(item["code"] == "workitem.status" for item in result["issues"]))
|
||||
|
||||
def test_windows_drive_path_is_parsed_as_scope_asset(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
self.write_case(root)
|
||||
scope = root / "scope.md"
|
||||
scope.write_text(
|
||||
scope.read_text(encoding="utf-8")
|
||||
.replace(" - sample.bin", r" - D:\reverse-skill")
|
||||
.replace("- mode: offline", "- mode: lab_only"),
|
||||
encoding="utf-8",
|
||||
)
|
||||
result = REVIEW_CASE.review_case(root)
|
||||
self.assertEqual(result["scope"]["assets"], [r"D:\reverse-skill"])
|
||||
self.assertFalse(any(item["code"] == "scope.assets_missing" for item in result["issues"]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user