feat: 1.0.0
This commit is contained in:
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "marcus-skills",
|
||||
"owner": {
|
||||
"name": "Marcus"
|
||||
},
|
||||
"metadata": {
|
||||
"description": "把可靠的工程方法沉淀为可复用的 Agent Skills。"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "marcus-skills",
|
||||
"source": "./",
|
||||
"description": "可复用的工程方法 Agent Skills。当前包含 bugfix：证明根因，修复整类缺陷而不只是单个实例。",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"name": "marcus-skills",
|
||||
"displayName": "Marcus Skills",
|
||||
"version": "1.0.0",
|
||||
"description": "可复用的工程方法 Agent Skills。当前包含 bugfix：证明根因，修复整类缺陷而不只是单个实例。",
|
||||
"author": {
|
||||
"name": "Marcus",
|
||||
"url": "https://www.qflink.xyz"
|
||||
},
|
||||
"keywords": ["bugfix", "debugging", "root-cause", "skills"],
|
||||
"skills": ["./bugfix"]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# 统一使用 LF，避免 Windows / Linux 之间来回转换；shell 脚本必须是 LF 才能在 bash 中运行
|
||||
* text=auto eol=lf
|
||||
*.sh text eol=lf
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
*.zip binary
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
.claude/settings.local.json
|
||||
.omc/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
.bugfix-scratch.*
|
||||
|
||||
# 系统文件
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
@@ -0,0 +1,185 @@
|
||||
# Agent Skills
|
||||
|
||||
**把可靠的工程方法沉淀为可复用的 Agent Skills。**
|
||||
|
||||
本仓库收录符合 [Agent Skills 规范](https://agentskills.io/specification) 的技能。每个 Skill 独立存放，包含供 Agent 执行的 `SKILL.md`，以及按需加载的参考资料；仓库另外提供模板、结构检查脚本和可复现的验证案例。
|
||||
|
||||
## Skills
|
||||
|
||||
| Skill | 版本 | 解决的问题 |
|
||||
| --- | --- | --- |
|
||||
| **[bugfix](bugfix/SKILL.md)** | 1.0.0 | 引导编程 Agent 证明 Bug 根因，识别同一缺陷机制的其他实例，从源头修复，并验证防复发检查是否有效。 |
|
||||
|
||||
## bugfix：修复一类缺陷，而不只是一处报错
|
||||
|
||||
> **Fix the class, not just the instance.**
|
||||
|
||||
AI 编程 Agent 往往能快速处理被报告的错误，但“当前场景不再报错”不等于问题已经解决。一个 Bug 可能只是共享错误假设的其中一处表现：相同的缺陷可能出现在旧版实现、其他调用方、不同语法写法中，甚至已经影响缓存、队列或业务数据。
|
||||
|
||||
`bugfix` 的目标不是让每次修复都扩大成项目重构，而是让 Agent 回答四个问题：
|
||||
|
||||
1. **为什么会错？** 用可复现、因果、可逆和机制性证据确认根因，而不满足于“异常消失”。
|
||||
2. **还有哪里会错？** 定义缺陷类别，再沿复制代码、共享契约、错误假设、历史实现和已写入数据等方向查找同源实例。
|
||||
3. **应该在哪里修？** 优先消除错误机制；只有共享契约确实一致时才抽取公共修复，否则在已确认的位置进行协调修复。
|
||||
4. **如何防止再犯？** 用实际验证过的测试或检查阻止新的同类错误，并确认真实执行路径已使用修复后的代码。
|
||||
|
||||
### 核心能力
|
||||
|
||||
| 能力 | 具体做法 |
|
||||
| --- | --- |
|
||||
| **根因证明** | 以复现、因果验证、可逆验证和代码机制说明四项证据约束结论。 |
|
||||
| **同源缺陷搜索** | 不只搜索同一段代码，还检查不同写法、相同契约的读写端、平行路径和受影响数据。 |
|
||||
| **候选项判定** | 每个搜索结果都要标为已确认、已排除或待确认；不能把“看起来相似”直接当成 Bug。 |
|
||||
| **按风险分级** | Quick / Standard / Full 三种调查深度；遇到共享契约、安全、数据损坏等风险时升级。 |
|
||||
| **防复发守卫** | 要求证明守卫能够拦住新的同类实例，而不是仅确认已有回归测试通过。 |
|
||||
| **运行时与回归验证** | 对比修改前后的测试基线，核实真实入口与实际运行的产物；无法验证时明确说明。 |
|
||||
| **安全边界** | 重大契约变化、依赖、数据修复、共享环境操作和部署等事项需要先获得授权。 |
|
||||
|
||||
### 调查流程
|
||||
|
||||
```text
|
||||
┌─ Quick ──────────────┐
|
||||
Phase 0 风险分级 ─────┤ ├─ 修复与验证 → 报告
|
||||
└─ Standard / Full ────┘
|
||||
│
|
||||
复现问题 → 根因证明
|
||||
│
|
||||
定义缺陷类别 → 同源实例清单
|
||||
│
|
||||
影响范围评估 → 从源头修复
|
||||
│
|
||||
守卫验收 → 回归及运行时验证
|
||||
│
|
||||
结果与遗留风险报告
|
||||
```
|
||||
|
||||
- **Quick**：明显、局部、低风险的问题。仍需精简的复现和因果证据，以及一次同类问题检查。
|
||||
- **Standard**：局限在少量文件、未涉及共享契约的普通逻辑缺陷。
|
||||
- **Full**：安全、数据损坏、共享代码或契约、并发、跨进程等高风险问题；执行更完整的范围分析与验证。
|
||||
|
||||
调查深度根据证据动态升级。**这不是要求所有简单问题都进行全项目扫描，也不是要求每次修复都引入新抽象。**
|
||||
|
||||
### 一个简化示例
|
||||
|
||||
**报告的问题**：商品搜索标签 `art` 时，错误返回包含 `cart` 的商品。
|
||||
|
||||
- 只修当前报错：将新版本搜索接口的子串判断改为精确标签匹配，然后结束。
|
||||
- 根因导向的调查：确认错误来自“把子串包含误当成标签成员关系”；继续查找旧版搜索、正则匹配以及权限判定等是否存在同一错误假设；区分真正受影响的位置与合法的集合成员查询；再完成针对性修复和验证。
|
||||
|
||||
这个案例来自仓库的 [`inventory` 验证夹具](tests/bugfix/fixtures/inventory/)。它体现的是**同一错误机制可能有不同代码写法和更高的安全影响**，而不是“发现相似代码就全部修改”。
|
||||
|
||||
## 安装
|
||||
|
||||
只需安装 `bugfix/` 目录（包括 `SKILL.md` 和完整的 `references/`）；`tests/`、`scripts/` 和 `templates/` 是仓库开发与验证工具，**不需要复制到 Agent 的 Skill 目录**。
|
||||
|
||||
**作为 Claude Code 插件安装**（仓库已包含 `.claude-plugin/`）：
|
||||
|
||||
```text
|
||||
/plugin marketplace add https://git.qflink.xyz/jiawei/skills
|
||||
/plugin install marcus-skills@marcus-skills
|
||||
```
|
||||
|
||||
安装后技能以插件命名空间调用，例如 `/marcus-skills:bugfix`。
|
||||
|
||||
## 使用
|
||||
|
||||
安装后直接描述需要修复的 Bug，Agent 可以根据 `SKILL.md` 的 `description` 判断是否启用；也可以显式调用。
|
||||
|
||||
**Claude Code**：
|
||||
|
||||
```text
|
||||
/bugfix 修复用户刷新页面后登录状态丢失的问题
|
||||
```
|
||||
|
||||
**Codex**：
|
||||
|
||||
```text
|
||||
$bugfix 排查这个接口为何间歇性返回错误数据，并检查是否存在同源问题
|
||||
```
|
||||
|
||||
**自然语言**：
|
||||
|
||||
```text
|
||||
修复订单状态偶发回退的问题。请先找到根因，再确认其他路径是否存在相同的错误机制。
|
||||
```
|
||||
|
||||
适用于真实的 Bug、异常、错误结果、回归、间歇性故障及因缺陷导致的性能问题；**不用于**新功能开发、纯代码评审或不存在已知缺陷的普通性能调优。Skill 的自动触发由 Agent 判断，不保证每次描述 Bug 都会自动启用。
|
||||
|
||||
### 结果报告
|
||||
|
||||
Skill 要求最终明确归入以下四种状态之一，避免用笼统的“已修复”掩盖证据缺口：
|
||||
|
||||
| 状态 | 含义 |
|
||||
| --- | --- |
|
||||
| **未修复** | 原始问题尚未得到代码层面的修复及验证。 |
|
||||
| **部分修复** | 仍有已确认或未解决的同类实例。 |
|
||||
| **本地已修复，线上未验证** | 代码与检查已通过，但执行中的实际产物尚未核实，或部署、重启待授权。 |
|
||||
| **整类缺陷已修复** | 已确认的实例、守卫验收及实际运行验证均完成，达到 Skill 定义的完成条件。 |
|
||||
|
||||
当运行环境没有浏览器、服务访问或测试执行能力时，Agent 应报告验证限制，而不是声称已经验证。
|
||||
|
||||
## 验证与测试
|
||||
|
||||
仓库提供的验证设施不只是检查 `SKILL.md` 是否符合格式，还包括对 Agent 行为的盲测。
|
||||
|
||||
### 规范与链接检查
|
||||
|
||||
从仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python scripts/check_skills.py
|
||||
# 只检查 bugfix
|
||||
python scripts/check_skills.py bugfix
|
||||
```
|
||||
|
||||
该脚本检查 frontmatter、名称、描述和 Markdown 相对链接，并对正文长度提供提示。完整的 Agent Skills 规范验证可另行使用官方参考校验器。
|
||||
|
||||
### 盲测案例
|
||||
|
||||
| 案例 | 仅报告的症状 | 主要考察 |
|
||||
| --- | --- | --- |
|
||||
| [`inventory`](tests/bugfix/fixtures/inventory/) | 标签搜索发生错误匹配 | 不同写法的同源问题、权限影响、缓存错误和正确代码的排除。 |
|
||||
| [`scheduler`](tests/bugfix/fixtures/scheduler/) | 重试延迟比配置短得多 | 秒/毫秒契约不一致、相关调用端、队列中已写入的错误记录。 |
|
||||
|
||||
仓库的 [`RESULTS.md`](tests/bugfix/RESULTS.md) 记录了 **2026-10-13** 的两次 Codex CLI 内部盲测：上述两个案例均按预先编写的评分标准得到 **18/18**。同时记录了因测试暴露的问题而修改 Skill 的过程，以及守卫回退配方的安全回归检查。
|
||||
|
||||
**注意**：这只是两个受控夹具的测试记录，不能据此推断真实工程中的普遍修复成功率、可靠性提升幅度或跨模型泛化能力。完整的运行方法、评分方式及安全前提参见 [`tests/bugfix/README.md`](tests/bugfix/README.md)。
|
||||
|
||||
在 Linux、macOS 或 Windows Git Bash 环境中，可从仓库根目录准备盲测夹具：
|
||||
|
||||
```bash
|
||||
bash tests/bugfix/setup_fixture.sh scheduler /tmp/bf-scheduler > /tmp/bf-scheduler.prompt.md
|
||||
```
|
||||
|
||||
再将生成的提示词交给被测 Agent，在夹具仓库中执行并根据独立的 `rubrics/` 评分。测试时应核查实际代码差异与运行结果，而不仅仅阅读 Agent 的文字报告。
|
||||
|
||||
## 仓库结构
|
||||
|
||||
```text
|
||||
.
|
||||
├── .claude-plugin/ # Claude Code 插件清单与 marketplace
|
||||
├── bugfix/
|
||||
│ ├── SKILL.md # 技能入口、分级流程、执行规则
|
||||
│ └── references/ # 按阶段加载的分析与验证材料
|
||||
│ ├── root-cause.md # 根因分析
|
||||
│ ├── class-analysis.md # 同源缺陷搜索与守卫验收
|
||||
│ ├── system-rca.md # 跨组件、跨进程问题
|
||||
│ ├── bug-patterns.md # 常见缺陷模式
|
||||
│ ├── frontend-patterns.md # 前端故障模式
|
||||
│ ├── backend-patterns.md # 后端故障模式
|
||||
│ ├── blind-spots.md # 影响范围中的常见盲区
|
||||
│ ├── verification.md # 基线、回归及运行时验证
|
||||
│ └── project-knowledge.md # 项目知识的使用与维护
|
||||
├── tests/bugfix/ # 夹具、提示词、评分标准、验证记录
|
||||
├── scripts/check_skills.py # Skill 结构检查
|
||||
├── templates/skill-template/ # 新 Skill 模板
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## 设计边界
|
||||
|
||||
- **不替代 Agent 的判断**：规则提供调查框架，但结论仍需事实与实验支撑。
|
||||
- **不默认进行全仓库重构**：先定义缺陷类别，再根据证据确定搜索与修改范围。
|
||||
- **不把搜索命中视作缺陷**：必须有失败样例或可追踪的不变量违例，才确认同源实例。
|
||||
- **不擅自进行高风险操作**：公共契约变更、新依赖、权威数据修复、共享服务操作及部署等需征得用户同意。
|
||||
- **不伪造完成状态**：无法运行的测试、未验证的线上版本和未处理的数据必须明确列出。
|
||||
+249
@@ -0,0 +1,249 @@
|
||||
---
|
||||
name: bugfix
|
||||
description: >-
|
||||
Root-cause bug fixing that repairs the whole defect class, not just the reported instance.
|
||||
Reproduces with evidence, proves the root cause, defines the defect class, finds every sibling
|
||||
(copies, other spellings, other consumers of the same contract, data the bug already wrote),
|
||||
fixes at the origin, adds a tested guard against recurrence, and verifies the fix is live with no
|
||||
regressions. Use when something is broken, wrong, erroring, flaky, slow, or regressed: "fix this
|
||||
bug", "debug", "not working", "error", "exception", "crash", "wrong output", "regression",
|
||||
"intermittent", "console warning". Not for new features, pure code review, or performance tuning
|
||||
with no defect.
|
||||
metadata:
|
||||
version: "1.0.0"
|
||||
---
|
||||
|
||||
# Bugfix: fix the class, not the instance
|
||||
|
||||
A reported bug is one **instance** of a **mechanism**. Mechanisms get copied, imitated, and re-written by people holding the same wrong assumption. This skill proves the cause, finds every place the mechanism lives, fixes it at the origin, and leaves a guard so the next instance fails a check instead of reaching users.
|
||||
|
||||
**Promise:** Prove the cause. Fix every instance. Make the next one hard to write. Break nothing.
|
||||
|
||||
**How to read this skill**
|
||||
- Paths are relative to this file. Load a `references/` file only when a phase points to it.
|
||||
- "Search", "run", and "edit" mean whatever tools your environment provides.
|
||||
- **Capabilities vary.** If you can't execute code, run a browser, reach a service, or write files, do what you can. Label anything unverified as **provisional** and list the checks the user must run.
|
||||
- Project-specific knowledge lives in the project ([references/project-knowledge.md](references/project-knowledge.md)). **Never modify this skill's files while fixing a bug.**
|
||||
|
||||
---
|
||||
|
||||
## Iron rules
|
||||
|
||||
1. **Evidence before fixes.** Reproduce first, ideally as a failing automated test that asserts the *correct* result.
|
||||
2. **Proven root cause.** The cause must pass four gates: reproducible, causal, reversible, mechanistic. "Causal" means the intended behavior is **restored**. If the error merely disappears (catch-and-ignore, fallback), it is not causal.
|
||||
3. **Fix the class.** Define the class (preconditions + mechanism + violated invariant + failure) and search every axis for siblings *before writing the fix*. (Diagnostic probes in Phase 2 are fine; revert them.) Your combined queries must recover every known instance (recall check).
|
||||
4. **Candidates are not defects.** A search hit is confirmed only by a witness (a failing test or a traced invariant violation) and excluded only with a reason.
|
||||
5. **Fix at the origin.** Use a shared fix only when the sites share the same contract. Otherwise fix each site the same coordinated way. Never patch downstream symptoms.
|
||||
6. **Leave a tested guard.** At least one check must fail on a *new* instance of the class. Prove it with the guard acceptance test for your depth.
|
||||
7. **Know the blast radius** before editing: consumers, contracts, invariants, call sites, duplicates.
|
||||
8. **Verify what actually runs.** Show that the executing artifact is the fixed one and that the real entry point behaves correctly. Handle data the bug already wrote per the safety boundaries.
|
||||
9. **No regressions.** Capture the baseline before changing anything; compare after.
|
||||
10. **Check framework behavior in the installed source** or with an experiment. Docs and memory are hypotheses.
|
||||
|
||||
## Safety boundaries
|
||||
|
||||
| Free to do | Ask the user first |
|
||||
|---|---|
|
||||
| Read code; run local tests, linters, and builds; edit the working copy; add tests; create scratch files and isolated copies (and remove them) | Contract/schema changes (including changed behavior for edge inputs of a public interface), new dependencies, migrations |
|
||||
| Invalidate **disposable** data the bug produced, but only when **all** of these hold, and disclose it in the report:<br>• it is **derived** (a cache or build output, not a record of truth);<br>• it is used **only locally** (no other consumer or sync);<br>• its **current contents are exactly recoverable** (committed and unmodified since the baseline, or regenerable).<br>Uncertain or mixed data counts as authoritative. | Repairing **authoritative** data (records, queues, user data), even when the file is local; touching shared or remote environments, caches, or queues; deploying |
|
||||
| | Restarting or killing processes you didn't start; writing project knowledge files; anything costly or irreversible |
|
||||
|
||||
Finding a command (in a README, CI file, or this skill) **doesn't authorize running it**. Target processes by exact PID or command line, never by name pattern. Data repair needs a backup or rollback plan and a dry run with counts. Probes and logs must redact secrets, tokens, and personal data, and must be removed or deliberately kept.
|
||||
|
||||
---
|
||||
|
||||
## Workflow
|
||||
|
||||
```
|
||||
Phase 0 Triage ─┬─ Quick ─────────► 1 → mini-3 → 5 → 6 → 7 (short report)
|
||||
├─ Standard ─┐
|
||||
└─ Full ─────┴─► 1 Reproduce → 2 Root cause → 3 Class analysis
|
||||
→ 4 Scope & impact → 5 Fix → 6 Verify → 7 Report & learn
|
||||
```
|
||||
|
||||
### Phase 0: Triage
|
||||
|
||||
| Depth | When (any trigger in a higher row wins) |
|
||||
|---|---|
|
||||
| **Full** | P0/P1, security, data loss or corruption, shared code or contract, schema, concurrency, multi-process, more than 3 files |
|
||||
| **Standard** | Logic bug in 1–3 files, P2/P3, nothing shared |
|
||||
| **Quick** | Typo, wrong constant or config value, obvious one-liner, no behavior change beyond the bug |
|
||||
|
||||
- **Severity:** P0 = down, data loss, or security. P1 = a core feature broken or data corrupted. P2 = a non-core feature or the UI. P3 = cosmetic.
|
||||
- Risk triggers override reported severity and small patch size. Re-triage whenever a trigger appears, especially after Phase 3: **class severity = worst confirmed instance**.
|
||||
- **Quick depth still needs proof, compressed:** a check that fails before the change and passes after (that covers reproducible, causal, and reversible), plus the one-line mechanism.
|
||||
- **P0/P1 in production: contain first.** Propose an authorized mitigation (rollback, feature flag off, rate limit), get approval, and apply it. Then do the permanent repair. Agree on a status-report interval with the user; report after every phase and on every containment change.
|
||||
- **No-progress budget:** if two consecutive rounds of experiments yield no new discriminating evidence, change the method (probes, bisect, smaller repro) or report what's eliminated and ask. Rejected hypotheses count as progress.
|
||||
|
||||
### Phase 1: Reproduce
|
||||
|
||||
| Bug type | Minimum evidence |
|
||||
|---|---|
|
||||
| Backend / CLI | Exact input, full error and stack trace, version/commit, config |
|
||||
| UI | Runtime evidence from the actual surface plus exact steps:<br>• **browser**: screenshot, console, DOM/styles, network<br>• **native app**: screenshot, widget/accessibility tree, logs<br>• **terminal**: captured output, input/event trace<br>**Never diagnose UI from code reading alone.** Without that runtime, the diagnosis is provisional. |
|
||||
| Performance | Before-metrics plus profile or query count |
|
||||
| Intermittent | Failure rate over N runs, plus timing, concurrency, and environment conditions |
|
||||
| Data | Example bad records plus the path that wrote them |
|
||||
|
||||
- Turn the repro into a **failing automated test** where feasible. It becomes the regression test and the reversible gate.
|
||||
- **Baseline:** find the project's real commands ([references/verification.md](references/verification.md) sections 0–1). Run them on unmodified code and record the tests that already fail.
|
||||
- Can't reproduce? Follow [references/root-cause.md](references/root-cause.md) section 6 (narrow the environment, bisect, instrument). Never present a speculative fix as confirmed.
|
||||
|
||||
### Phase 2: Root cause
|
||||
|
||||
1. **Known matches first:** check the project's knowledge files, then [references/bug-patterns.md](references/bug-patterns.md) and the backend/frontend playbooks. Use only rows that fit the stack. A match is a lead, not a proof.
|
||||
2. **Hypothesis ladder:** unless the evidence already pins one mechanism, write 3–5 falsifiable hypotheses; cheapest rejection tests first; one variable at a time. Then five whys until you reach a mechanism plus its trigger condition. See [references/root-cause.md](references/root-cause.md). If the bug crosses process or service boundaries, also use [references/system-rca.md](references/system-rca.md).
|
||||
3. **Origin layer:** input, config, data, contract, shared code, repeated local mistake, single component, architecture, external dependency, or model capability. The fix goes there.
|
||||
4. **Four gates:**
|
||||
|
||||
| Gate | Evidence |
|
||||
|---|---|
|
||||
| Reproducible | Failing test or scripted repro |
|
||||
| Causal | A minimal change restores the **intended** output; a negative control stays correct |
|
||||
| Reversible | Removing the change (in a test or fixture) brings the failure back |
|
||||
| Mechanistic | file:line, the violated invariant, and a step-by-step explanation |
|
||||
|
||||
The causal/reversible change is a **diagnostic probe**. Apply it in a test, a scratch copy, or in place, then revert it. The real fix comes in Phase 5, shaped by Phases 3–4.
|
||||
|
||||
### Phase 3: Class analysis (the core)
|
||||
|
||||
Method and verified search recipes: [references/class-analysis.md](references/class-analysis.md). Produce these, in order:
|
||||
|
||||
1. **Class definition**: preconditions + mechanism + violated invariant + observable failure. Add the mechanism's **spellings**, the **wrong assumption** behind it, and **legitimate neighbors** (code that looks similar but is correct).
|
||||
2. **Search universe**: roots and file classes covered (untracked, ignored, submodules, generated, sibling repos). Anything you can't search is a **coverage limit**, not "no hits".
|
||||
3. **Searches across the axes**: clones, same syntax, same mechanism in other syntax, same contract (writers *and* readers), same assumption, parallel paths, history, data and artifacts, config/env/platform.
|
||||
4. **Recall check**:
|
||||
- The combined queries return every known instance.
|
||||
- Each complementary query hits a positive example from its own target population.
|
||||
- At least one variant spelling is probed.
|
||||
5. **Inventory**: one row per candidate, with a disposition:
|
||||
- **confirmed** (needs a witness);
|
||||
- **excluded** (needs a reason);
|
||||
- **unresolved** (gets reported).
|
||||
|
||||
Data rows count too. Re-triage on the worst confirmed row.
|
||||
6. **Fix level** (highest that fits the evidence and approved scope):
|
||||
1. **Eliminate**: a type, a shared helper, boundary validation, or a safe default. Only if the sites share the contract.
|
||||
2. **Centralize + guard.**
|
||||
3. **Coordinated site fixes + guard.**
|
||||
7. **Guard plan**, plus how you will run the guard acceptance test for your depth (Step 6 of the reference). Reverse only the production fix, in an isolated copy, so the user's uncommitted work is never touched.
|
||||
|
||||
**Quick depth** does a mini version: the class definition in one line, one search for the same mistake with a recall check, and an escalation if it finds siblings.
|
||||
|
||||
### Phase 4: Scope & impact (before editing)
|
||||
|
||||
Full depth: five gates. Standard depth: the quick check below.
|
||||
|
||||
| # | Gate | Output |
|
||||
|---|---|---|
|
||||
| 1 | Consumers | All dependents of the code you will change, including re-exports, dynamic dispatch, and registrations |
|
||||
| 2 | Contracts | Interfaces, schemas, and behaviors that change, **including return types and shapes and behavior on inputs outside the bug** (empty, null, case, whitespace, errors). Every change gets listed in the report; public ones need approval first. |
|
||||
| 3 | Invariants | What must stay true |
|
||||
| 4 | Call sites | Each one: affected / unaffected / needs update |
|
||||
| 5 | Duplicates | Parallel implementations covered by the inventory |
|
||||
|
||||
**Impact chain:**
|
||||
- **code**: out to the user-facing surface;
|
||||
- **data**: corrupted records, repair needed?;
|
||||
- **time**: when introduced, how long exposed;
|
||||
- **events**: queues, streams, jobs.
|
||||
|
||||
Then check the applicable entries in [references/blind-spots.md](references/blind-spots.md) and the project's own list, and decide **go / no-go**. For Full depth with competing approaches, compare size, blast radius, regression risk, and rollback.
|
||||
|
||||
Quick impact check (Standard depth):
|
||||
|
||||
```markdown
|
||||
- Change: <one line> - Origin layer: <layer>
|
||||
- Consumers: <list or "local only"> - Inventory: <n confirmed / n excluded / n unresolved>
|
||||
- Could break: <prediction> - Blind spots checked: <ids> - Decision: GO / NO-GO
|
||||
```
|
||||
|
||||
### Phase 5: Fix
|
||||
|
||||
- The failing test exists first, then the code change.
|
||||
- Fix at the origin layer, using the chosen rung. Per-site edits stay minimal. Identical edits across confirmed sites are fine. Unrelated refactors are not: propose them instead.
|
||||
- One concern per change: the class fix, the guard, and the tests.
|
||||
- When the fix matches strings or patterns, define the token grammar and test the boundaries: equal, prefix, suffix, embedded, empty, case, unicode, multiline. Prefer parse-then-compare over a cleverer regex.
|
||||
- For data repair or migrations, follow the safety boundaries: backup or rollback plan, dry run, approval.
|
||||
- Where the fix depends on framework behavior, leave a short comment with the evidence (source location or experiment).
|
||||
|
||||
### Phase 6: Verify
|
||||
|
||||
Details: [references/verification.md](references/verification.md).
|
||||
|
||||
1. **Correctness**: the repro test passes; every confirmed site is verified; edge and failure paths work; **the guard passes its acceptance test**.
|
||||
2. **Regression**: run the affected suites, then compare the full relevant suite with the baseline. Check performance and concurrency when relevant.
|
||||
3. **Runtime**: show that the executing artifact is the fixed one and that the real entry point is correct. Acceptable evidence:
|
||||
- a version or commit the running process reports;
|
||||
- an artifact hash or image digest;
|
||||
- a process started **after** the change whose loaded files are the edited ones.
|
||||
|
||||
A new PID, a loaded file path alone, or a 200 health check proves nothing. Only touch caches or processes when evidence implicates them, within the safety boundaries.
|
||||
4. **Review**: use an independent reviewer (a code-review tool, a second agent, or a person) if one is available. Name it in the report and cite its verdict. Anything else is **self-review**; call it that. Check that:
|
||||
- the diff matches the inventory;
|
||||
- there are no unrelated changes and no secrets;
|
||||
- the tests fail without the fix;
|
||||
- error paths are handled.
|
||||
|
||||
### Phase 7: Report & learn
|
||||
|
||||
Pick exactly one **outcome**. If several apply, report the first that matches in this order, and list the rest as follow-ups. Prerequisites: [references/verification.md](references/verification.md) section 6.
|
||||
1. **Not fixed**: the reported instance isn't fixed and verified in code. Include evidence, eliminated hypotheses, and the next experiment.
|
||||
2. **Partially fixed**: some confirmed or unresolved rows remain (code or data), listed with reasons.
|
||||
3. **Fixed locally, not live**: every row fixed in code with checks passing, but runtime is unverified, or a deploy or restart awaits approval.
|
||||
4. **Class fixed**: everything above is done, the guard is accepted, and the runtime is verified.
|
||||
|
||||
```markdown
|
||||
## Bugfix report: <short symptom>
|
||||
- Outcome: <one of the four> Severity/depth: <P?>/<Quick|Standard|Full> (class severity: <P?>)
|
||||
- Symptom: <user-visible>
|
||||
- Root cause: <mechanism, file:line, violated invariant>. Gates: reproducible ✓ causal ✓ reversible ✓ mechanistic ✓
|
||||
- Class: <preconditions + mechanism + invariant + failure>; legitimate neighbors: <...>
|
||||
- Search universe / coverage limits: <...>
|
||||
- Queries + recall check: <axes, queries, known instances recovered, variant probe>
|
||||
- Inventory: <table: site | disposition | evidence>
|
||||
- Fix level: <rung>. Why not higher: <reason>
|
||||
- Guard: <what>. Acceptance items passed: <list>; limits: <...>
|
||||
- Verification: <baseline → after; entry point exercised> Runtime/data: <evidence or "unverified: why">
|
||||
- Contract changes (incl. edge inputs): <none | list> Review: <independent: who + verdict | self-review>
|
||||
- Needs approval / follow-ups: <reported rows, containment, data repair, deploy, higher-rung proposal>
|
||||
- Escape analysis: <why this class escaped (missing test/type/review/knowledge) and what now closes it>
|
||||
```
|
||||
|
||||
Quick depth: Outcome, Symptom, Root cause, Class (one line), the sibling search and its result, Fix, Verification.
|
||||
|
||||
Afterwards:
|
||||
- Propose project knowledge entries ([references/project-knowledge.md](references/project-knowledge.md)). Write them only with consent.
|
||||
- If the fix caused a regression, add a **regression autopsy**: what broke, why it wasn't predicted, which gate or check would have caught it.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
| Don't | Do |
|
||||
|---|---|
|
||||
| Fix from code reading, without a repro | Reproduce; failing test asserting the correct result |
|
||||
| Accept "the error went away" as the cause | Intended behavior restored + negative control |
|
||||
| Fix the reported file only | Define the class → search all axes → inventory |
|
||||
| Trust a query that never hit anything | Validate every query against a known positive of its population |
|
||||
| "Fix" every search hit | Witness for confirmed, reason for excluded |
|
||||
| Build a shared abstraction because code looks alike | Centralize only for a shared contract; otherwise coordinated site fixes |
|
||||
| Ship a guard you never saw fail | Guard acceptance test |
|
||||
| Treat reported severity as final | Class severity = worst confirmed instance |
|
||||
| "Tests pass" = done | Executing artifact + real entry point + data checked |
|
||||
| Run whatever the docs say, restart by name, clean caches by habit | Safety boundaries; act on evidence only |
|
||||
| Round a partial fix up to "fixed" | Report exactly one outcome state |
|
||||
|
||||
## References
|
||||
|
||||
| File | Load when |
|
||||
|---|---|
|
||||
| [references/class-analysis.md](references/class-analysis.md) | Phase 3, always (Quick: Steps 1 and 3) |
|
||||
| [references/root-cause.md](references/root-cause.md) | Phase 2; can't reproduce; intermittent bugs |
|
||||
| [references/system-rca.md](references/system-rca.md) | Bug crosses processes, services, queues, streams, or browser automation |
|
||||
| [references/bug-patterns.md](references/bug-patterns.md) | Phase 2 known-match search; Phase 3 sibling search ideas |
|
||||
| [references/backend-patterns.md](references/backend-patterns.md) | Server, API, DB, or LLM-integration bugs |
|
||||
| [references/frontend-patterns.md](references/frontend-patterns.md) | UI, client state, or browser bugs |
|
||||
| [references/verification.md](references/verification.md) | Phase 1 baseline; Phase 6; outcome states |
|
||||
| [references/blind-spots.md](references/blind-spots.md) | Phase 4 (Standard/Full) |
|
||||
| [references/project-knowledge.md](references/project-knowledge.md) | Phase 2 (read) and Phase 7 (propose or write) |
|
||||
@@ -0,0 +1,105 @@
|
||||
# Backend Playbook
|
||||
|
||||
Detailed symptom → detection → fix → verify entries for server-side bugs. These apply to any language. Library names are **examples**: verify the behavior against the version actually installed. The summary table lives in [bug-patterns.md](bug-patterns.md).
|
||||
|
||||
## Quick index by symptom
|
||||
|
||||
| Symptom | Likely section |
|
||||
|---|---|
|
||||
| HTTP 200 but the operation failed | 1.2 |
|
||||
| Field sent, not saved | 1.3 |
|
||||
| OPTIONS 401/403, CORS error | 1.4 |
|
||||
| Hangs / pool exhausted | 2.1, 4.2 |
|
||||
| Duplicate writes / orders | 2.2 |
|
||||
| Half-written data | 3.1 |
|
||||
| Lost updates | 3.2 |
|
||||
| Slow as data grows | 4.1 |
|
||||
| Off-by-one day, wrong schedule | 5.1 |
|
||||
| Garbled text | 5.2 |
|
||||
| Precision loss | 5.3 |
|
||||
| Only "500" in logs | 6.1 |
|
||||
| Unknown column / dialect error / FK error | 7 |
|
||||
| LLM / tool-calling misbehavior | 8 |
|
||||
|
||||
---
|
||||
|
||||
## 1. API behavior & contract
|
||||
|
||||
**1.1 Schema drift.** *Symptom*: client parse errors, missing fields. *Detect*: compare the contract (OpenAPI or schema) with the actual responses, including the error branches. *Fix*: restore the contract, or version it; add a compatibility layer (aliases, defaults). *Class search*: every consumer of the endpoint. *Verify*: contract tests for the success and error shapes; old clients still work.
|
||||
|
||||
**1.2 Transport vs business status.** *Symptom*: 200 with `{"ok": false}`, or 4xx codes the client can't distinguish. *Fix*: one error model (status + code + message + correlation ID) mapped in a single middleware. *Class search*: every client call site that checks only the status code. *Verify*: the client can tell retryable from non-retryable from user-correctable errors.
|
||||
|
||||
**1.3 Field silently dropped by a request schema / serializer.** *Symptom*: field sent, not persisted. *Mechanism*: many validators **ignore undeclared fields** by default (their "extra" policy). Separately, "exclude unset" style serialization drops *declared* fields the client didn't supply, which is correct for partial updates. *Detect*: compare the request payload with the parsed model and the stored row; check the schema's extra-field policy, aliases, and the set of fields actually supplied. *Fix*: declare the field on every schema where the contract says it is editable (create/update/read). Keep partial-update semantics. *Class search*: all editable fields × all schemas. *Verify*: PATCH with the field omitted, explicitly null, and with a value: each behaves as the contract says.
|
||||
|
||||
**1.4 CORS preflight rejected.** *Symptom*: browser-only failure; the preflight `OPTIONS` (with `Access-Control-Request-Method`) gets 401/403 or lacks the CORS headers. *Fix*: handle preflight **before** the middleware that rejects it; set the allowed origins, methods, and headers per environment; credentials need an explicit origin (no `*`). *Verify*: for an allowed origin, the preflight returns the correct `Access-Control-Allow-*` headers. A disallowed origin is refused. The actual protected request **without** auth is still rejected. Status code alone proves nothing.
|
||||
|
||||
## 2. Timeouts, retries, idempotency, pagination
|
||||
|
||||
**2.1 Missing timeout / deadline.** Many clients' timeouts are per phase (connect, read inactivity), so a peer that trickles bytes never trips them. *Fix*: an **end-to-end deadline** per operation on HTTP, DB, RPC, subprocess, and lock waits, set in one client factory. Propagate the remaining budget across nested calls and retries. *Guard*: a CI search for raw client calls without a deadline. *Verify*: inject a stalled **and** a trickling dependency; the call fails within the budget, resources are released after cancellation, and the failure is visible.
|
||||
|
||||
**2.2 Retry without idempotency.** A timeout doesn't prove the operation failed. It may have committed. *Fix*: make the mutation idempotent (key or unique constraint) or reconcile before retrying. Retry only retryable errors, with backoff and jitter, a deadline, and an attempt cap. Account for retries the SDK already does. *Verify*: replaying the same request produces one effect, including the "effect committed, response lost" case.
|
||||
|
||||
**2.3 Unstable pagination.** *Fix*: first state the consistency contract (a snapshot? "no duplicates, may miss new rows"?). A deterministic sort with a unique tie-breaker fixes ordering. But **offset** pagination still shifts when rows are inserted or deleted between pages. For consistency under writes, use keyset (cursor) pagination, a snapshot, or a cutoff timestamp. *Verify*: inserts, deletes, and sort-key updates between page fetches behave as the contract says.
|
||||
|
||||
## 3. Consistency & transactions
|
||||
|
||||
**3.1 Partial write.** *Fix*: one transaction for related writes; compensation for external side effects. *Verify*: after fault injection, the invariants hold.
|
||||
|
||||
**3.2 Lost update.** *Fix*: optimistic version column, pessimistic lock, or atomic update statement. *Verify*: a concurrent stress test loses no updates.
|
||||
|
||||
## 4. Performance that shows up as bugs
|
||||
|
||||
**4.1 N+1 queries / external calls in a loop.** *Fix*: batch, eager-load, or join; cache hotspots. *Guard*: a query-count assertion in tests.
|
||||
|
||||
**4.2 Pool exhaustion.** *Fix*: release resources in `finally`/context managers; bounded pools; circuit breaker. *Verify*: connection count returns to baseline after a load test.
|
||||
|
||||
## 5. Serialization, encoding, time
|
||||
|
||||
**5.1 Time.** First classify each value:
|
||||
- **instant** (when something happened): UTC-aware;
|
||||
- **calendar date** (birthday, due date): a date type, not a timestamp;
|
||||
- **wall-clock or recurring time** ("9:00 every Monday in Berlin"): local time plus a timezone ID.
|
||||
|
||||
Use ISO-8601 on the wire. *Class search*: zone-less `now()`, parse, and format calls; dates stored as timestamps. *Verify*: DST transitions, midnight crossings, nonexistent or ambiguous local times.
|
||||
|
||||
**5.2 Encoding.** UTF-8 end to end (source files, DB, `Content-Type`, console). *Verify*: mixed-script and emoji round-trips.
|
||||
|
||||
**5.3 JSON numbers.** Send 64-bit IDs and decimals as strings; store money in integer minor units or a decimal type.
|
||||
|
||||
## 6. Errors & observability
|
||||
|
||||
**6.1 Swallowed error / missing context.** *Fix*: one exception middleware; structured logs; correlation ID propagated across hops. *Verify*: any failure can be traced from the user-visible ID.
|
||||
|
||||
**6.2 Sensitive data in logs.** *Fix*: redaction at the logging layer; no body logging on auth or payment endpoints. *Verify*: a log scan finds no secrets.
|
||||
|
||||
## 7. Persistence (any ORM; SQLAlchemy/Alembic and similar as examples)
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `Unknown column` / missing table | Schema-affecting model change without a migration | Autogenerated migrations are **candidates**: review every operation (they miss some changes: renames, some type and constraint changes). Prefer the tool's non-writing drift check where the installed version has one. Test the upgrade against representative existing data. Creating, applying, and repairing data are separate steps, each needing evidence; applying to shared DBs needs approval. |
|
||||
| `Data too long` | Column too small | Reviewed migration to widen it, plus validation at the boundary |
|
||||
| Enum lookup error on read | Stored values don't match what the mapping expects (names vs values, renamed members, values written by another writer) | Inspect the actual stored values, the enum's names and values, and the mapping config. Pick an explicitly compatible mapping or a reviewed data migration. Test existing rows and new writes. |
|
||||
| FK constraint error | **Insert**: parent missing or not yet created. **Delete**: children still reference the parent. | Insert: fix ordering or validation. Delete: choose the policy by domain meaning. Keep DB enforcement; don't change the policy just to silence the error. |
|
||||
| Syntax error on one DB | Dialect-specific SQL (`ILIKE`, `::type`, `CAST AS TEXT`) | ORM abstraction or portable SQL; test every supported DB |
|
||||
| Fix applied, old behavior persists | A different process, artifact, or installed copy is running | See verification.md section 5 |
|
||||
|
||||
## 8. AI / LLM integration
|
||||
|
||||
Provider and model capabilities change often. Verify the current behavior of the exact provider, model, and API version before relying on any row.
|
||||
|
||||
| Symptom | Mechanism | Fix direction | Class search |
|
||||
|---|---|---|---|
|
||||
| Tool call parsed wrong; arguments garbled or merged into one | Text-protocol tool calls parsed by fragile regex (single-arg assumption, no boundaries, greedy or multiline anchors) | Parse with a real parser (AST / JSON schema); exact name comparison; test multi-arg, nested-quote, and `)`-at-line-end cases | **Every** parser of model output, including duplicates in alternate execution paths |
|
||||
| Wrong parameter filled | Fallback remapping guesses parameter names (e.g. by sort order) | Fix the producer or parser; ambiguous mappings must be **rejected**, not guessed | All fallback, remap, and "best effort" code |
|
||||
| Tools never invoked for some models | The integration routes this model through a text-protocol fallback that doesn't handle its output format, or the capability table is stale | Capability detection per provider, model, and version; test each supported model family | Model capability tables and switches |
|
||||
| Empty or truncated responses | Token limits, safety stops, streaming fields not read (e.g. separate reasoning content) | Handle every finish reason; read all streamed fields; retry or degrade | All stream consumers |
|
||||
| Embedding dimension mismatch | Model changed, index not rebuilt | Version the index by model; rebuild | All stores keyed by that embedding model |
|
||||
| Tool server (subprocess or remote) unavailable | Lifecycle: process died, env missing, timeout | Health check, restart policy, explicit timeout, surfaced error | All tool server registrations |
|
||||
|
||||
## Minimal backend verification
|
||||
|
||||
- [ ] Repro test fails before the fix and passes after
|
||||
- [ ] At least one automated test at the right level (unit, contract, or integration)
|
||||
- [ ] Timeout/retry/idempotency/transaction behavior tested where relevant
|
||||
- [ ] Every inventoried site verified; the guard bites
|
||||
- [ ] Fixed code verified live (process, artifact, cache)
|
||||
@@ -0,0 +1,46 @@
|
||||
# Agent Blind-Spot Checklist
|
||||
|
||||
These are mistakes coding agents repeatedly make while fixing bugs. In Phase 4, scan the list, run the checks that apply to this bug, and note them in your impact check. Skip the ones that clearly don't apply; you don't need to list N/A for each.
|
||||
|
||||
This file is a **read-only seed list**. Do not edit it during a bugfix. Blind spots specific to one project go in that project's knowledge folder (see [project-knowledge.md](project-knowledge.md)). That file uses the same table format and is checked **in addition** to this one.
|
||||
|
||||
| ID | Blind spot | Check |
|
||||
|---|---|---|
|
||||
| BS-01 | Fixed one copy; a duplicate implementation or parallel path still has the bug | Clones and parallel-paths axes of class analysis; search the function name and distinctive literals project-wide |
|
||||
| BS-02 | Searched one spelling of the mechanism | Spellings list + variant probe (class-analysis Step 3) |
|
||||
| BS-03 | Search tool skipped files (untracked, ignored, submodules, generated, other repo or service) | Separate passes per class-analysis Step 2a (`--recurse-submodules`; `--untracked --no-exclude-standard`); check generated sources and sibling repos named in the project profile |
|
||||
| BS-04 | Indirect callers missed (re-exports, dynamic dispatch, string-registered handlers, reflection) | Search the exported name, the string keys, and the registration tables, not just direct calls |
|
||||
| BS-05 | Tests and fixtures not updated alongside the source | Search test files for the changed symbols and the old behavior |
|
||||
| BS-06 | Fixed code, but the running system still uses the old code or data | verification.md section 5: show the executing artifact is the fixed one (reported version, hash, digest, or a process started after the change) plus fixed behavior on the real entry point |
|
||||
| BS-07 | Data already corrupted by the bug left in place | Data & artifacts axis. Disposable (derived, local-only, exactly recoverable): invalidate and disclose. Otherwise: repair plan with approval. |
|
||||
| BS-08 | New pattern or regex in the fix matches too much (substring, multiline anchors, case, unicode) | Boundary tests: equal, prefix, suffix, embedded, empty, case, multiline input |
|
||||
| BS-09 | Assumed a response or input shape matches the declared types | Log or assert the real runtime shape once |
|
||||
| BS-10 | Only the happy path verified | Run failure paths explicitly: error status, timeout, null/empty, invalid input |
|
||||
| BS-11 | Concurrency ignored | Ask "can this run twice at once, or out of order?" If yes, test it |
|
||||
| BS-12 | Cache or derived state not invalidated after a write | Search reads of the mutated entity in caches and query layers |
|
||||
| BS-13 | Fallback / guessing logic hides the real failure (auto-remap, defaults, retries) | Find fallback branches on the path; make ambiguity fail loudly or log loudly |
|
||||
| BS-14 | Config fixed in one environment only | Enumerate all env files, manifests, and CI variables |
|
||||
| BS-15 | Scope creep: refactored beyond the class | Diff contains only class fix + guard + tests; anything else is reported, not done |
|
||||
| BS-16 | Under-triaged: a sibling is worse than the reported bug (security, data loss) | Class severity = worst instance; re-triage after the inventory |
|
||||
| BS-17 | Trusted docs or memory about framework behavior | Verify against the installed source or a minimal experiment |
|
||||
| BS-18 | Guard written but never shown to fail | Run the guard against the pre-fix code |
|
||||
| BS-19 | Pre-existing failing tests blamed on (or hidden by) the fix | Compare against the baseline captured before editing |
|
||||
| BS-20 | Secrets or personal data copied into logs, probes, test fixtures, or the report | Redact before writing anything persistent |
|
||||
| BS-21 | Fix silently changed the contract: behavior on inputs outside the bug (empty → all vs none, null, case, whitespace, error codes), or return type/shape (int → float, list → generator) | Compare old and new behavior on edge inputs **and** the returned types; list every change under Contracts in the report |
|
||||
| BS-22 | Called a self-check an "independent review" | Independent means a separate reviewer whose output you can cite; otherwise write "self-review" |
|
||||
|
||||
## Witness cases worth reusing
|
||||
|
||||
These are concrete tests that caught real misses. Use the ones that fit the bug's shape:
|
||||
- **Alternate entry points agree**: the same input through every parallel path (v1/v2, sync/async, both executors) gives the same valid result.
|
||||
- **Ambiguity is rejected, not guessed**: when a fallback can't map inputs unambiguously (e.g. two unnamed values for two required parameters), it errors. It must not pick by order or alphabet.
|
||||
- **Content containing the delimiter**: a closing token (`)`, `"`, `,`, `]`) at the end of a line *inside* the payload doesn't end the parse early (multiline anchors, greedy vs lazy).
|
||||
- **Names that contain each other**: `x` vs `xy` vs `yx` vs `x_y` vs a unicode-prefixed `éx`. Only exact intended matches succeed.
|
||||
- **Read-after-write across surfaces**: after a mutation, every view, cache, or consumer of the entity shows the new value.
|
||||
|
||||
## Maintaining the project's blind-spot file
|
||||
|
||||
After each fix (Phase 7):
|
||||
1. Did a check here or in the project file catch something? Note it in the report.
|
||||
2. Did you miss something that no check covers? Propose a new project entry: `| P-NN | <what was missed> | <date> | <specific check> |`.
|
||||
3. If an entry keeps proving useful across projects, propose promoting it to this seed list in the report. Don't edit this file yourself.
|
||||
@@ -0,0 +1,115 @@
|
||||
# Bug Pattern Catalogue (universal)
|
||||
|
||||
Use this in Phase 2, step 1, to look for a known match, and in Phase 3 for ideas on how to find siblings. A match is a **lead, not a proof**: the root cause must still pass the four gates.
|
||||
|
||||
Every row has a **Class search** column (how to find all siblings) and a **Source fix / guard** column (how to stop the class, not just the instance). Stack-specific playbooks: [backend-patterns.md](backend-patterns.md), [frontend-patterns.md](frontend-patterns.md).
|
||||
|
||||
**Applicability:** use only the rows that fit the project's languages, frameworks, and data stores. Don't run ORM, browser, or language-specific checks on a project that doesn't have them. Library behavior is version-dependent: confirm it against the installed version before relying on a row's fix.
|
||||
|
||||
---
|
||||
|
||||
## 1. Matching, parsing & escaping
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Match without boundaries (substring hits a longer token, identifier or name) | Wrong item, tool, route, or permission matched; "a" matches "ab" | Every substring test on names or tokens: `in`, `includes`, `indexOf`, `find`, `contains`, `strings.Contains`, `LIKE '%x%'`, regex without anchors or word boundaries | Define the token/identifier grammar (delimiters, case, unicode); parse or tokenize, then compare exactly. Lookarounds like `(?<![A-Za-z0-9_])` are ASCII-only and miss unicode identifiers. Test "x" against "xy", "yx", "éx", "" |
|
||||
| Regex `$`/`^` with multiline flag | Match stops at the first line end inside the content | Regexes with MULTILINE / `m` flag | Use `\A` / `\z` (or the language's equivalent) for whole-string anchors |
|
||||
| Unescaped user input used as a regex or query fragment | Crash on special characters, or injection | Regex compiled from variables; string-built SQL, shell, or HTML | Escape at construction, or use parameterized APIs; a lint rule bans the raw form |
|
||||
| Double escaping / unnecessary preprocessing | Valid input broken after a "fix" | Every pre-parse transformation of the same payload | Trust the parser; normalize once, at the boundary |
|
||||
| Format assumption (case, whitespace, BOM, line endings) | Valid input rejected or mismatched | Comparisons of external strings without normalization | Normalize at the boundary; test with BOM/CRLF/trailing spaces |
|
||||
|
||||
## 2. Data flow & contracts
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Write/read format mismatch | Saved OK but displayed wrong; selection not reflected | All writers and readers of the field, key, or file | One codec or serializer used by both sides; round-trip test |
|
||||
| Schema / field silently dropped | Field sent but not stored or returned | Every request/response/update schema for the entity; extra-field policy; serializer exclude options | Per-operation contract tests (create/update/read), including **negative** tests that immutable and privileged fields are rejected. Don't force the field sets to be equal; operations legitimately differ. |
|
||||
| Producer/consumer drift | Consumer crashes or misreads after a producer change | All consumers of the endpoint, topic, file, or event | Versioned contract plus contract tests on both sides |
|
||||
| Implicit vs explicit state | Inherited or default state shown as "unset" | Read paths that assume stored-explicit values | Derive the effective state in one function |
|
||||
| Null / missing / empty not handled | Crash or wrong default on edge input | Same accessor pattern across modules | Validate at the boundary; types that forbid null where invalid |
|
||||
| Off-by-one / boundary index | Missing or duplicate first or last element; pagination gaps | Same loop, slice, or page arithmetic elsewhere | Shared helper; boundary-value tests |
|
||||
| Numeric representation | Money rounding, precision loss in JSON, overflow | Float arithmetic on money; big IDs in JSON numbers | Decimal or integer minor units; IDs as strings; type guard |
|
||||
|
||||
## 3. State, caching & derived data
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Missing invalidation after mutation | Stale lists or details after an edit | Every cache, memo, or query-cache read of the mutated entity | Invalidate in the mutation layer, not per screen; test read-after-write |
|
||||
| Derived data written by a past bug | Code fixed, wrong results persist | Caches, indexes, exports, materialized views, queued jobs produced by the buggy path | Repair or rebuild script (dry run first); cache key versioning |
|
||||
| Status mismatch (transport OK, business failed) | UI shows success on failure | All call sites checking only HTTP or transport status | One response-handling helper that checks business status |
|
||||
| Orphaned state / missing cleanup | Leaks; stuck "in progress" | Every exit path (early return, exception, cancel) of the same resource | `finally` / context manager / RAII; leak test |
|
||||
| Over-sync overwrites user choice | User selection reset by a refresh | Effects or subscriptions that write user-owned state | Sync only on explicit triggers; separate server and user state |
|
||||
|
||||
## 4. Concurrency, async & time
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Race / ordering assumption | Intermittent, order-dependent failures | Shared mutable state; "fire and forget" calls; responses applied without a sequence check | Serialize, lock, use atomic ops, or apply only the latest response; stress test |
|
||||
| Stale captured value (closure, handler, timer) | Uses an old value, or applies work to the wrong context | Callbacks and async work capturing context at creation time | Decide per value. **Operation identity** (tenant, account, request) stays bound to the operation; on a mismatch, cancel or reject. **"Latest wins" settings** are read at execution time. Test context switches during in-flight work. |
|
||||
| Retry without idempotency | Duplicate writes or orders | All retry wrappers (including SDK built-in retries) around mutations | Make mutations idempotent (key or unique constraint) or reconcile **before** retrying them; retry only retryable errors with deadline and attempt cap. Test "effect committed, response lost". |
|
||||
| Missing timeout | Hangs; exhausted pools | Every outbound call: HTTP, DB, RPC, subprocess, lock wait | Shared client with a default deadline; CI check for raw calls |
|
||||
| Wrong temporal type | Off-by-one day; wrong schedule; DST errors | `now()` without a zone; parse/format without a zone; dates stored as timestamps | Classify each value first: instant → UTC-aware; calendar date → date type; wall-clock or recurring time → local time plus a timezone ID. Test DST transitions and midnight crossings. |
|
||||
|
||||
## 5. Configuration & environment
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Fix applied to one config source only | Works in prod, fails in dev (or the reverse) | Every config source the project actually uses: env files, manifests, CI variables, secret stores, startup scripts; and their precedence | One config schema, validated at startup in all envs |
|
||||
| Merge-order assumption | Wrong value wins | Every config loader or merge site | Read the loader source; document priority; test the precedence |
|
||||
| Missing default / required var | Crash only in some environments | All reads of the variable | Central config object with defaults and validation |
|
||||
| Secret exposure | Tokens or PII in logs or UI | Logging of request/response bodies, errors, config dumps | Redaction at the log layer; test that logs contain no secrets |
|
||||
| Non-critical dependency without degradation | Whole feature fails when an optional service is down | All calls to the optional dependency | Explicit fallback path; failure-injection test |
|
||||
|
||||
## 6. Propagation & parallel implementations
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Duplicate implementation (copy-paste, v1/v2, sync/async, client/server) | Fixed in one path, still broken in the other | Function name, distinctive literals, "copied from" comments, legacy/compat folders | If both paths promise the same contract, make one delegate to the other (or delete the dead one). If their contracts differ, fix both and add a parity test on the shared behavior. Test every entry point. |
|
||||
| Partial rollout of a behavior change | New behavior only in some views or endpoints | All usages of the old pattern or component | If the usages share one contract and owner, centralize; otherwise update each with review. Lint for the old form. |
|
||||
| Incomplete prop / parameter propagation | Works in one place, not in another | Every caller or parent of the changed component or function | Required parameters (no silent default); type checker |
|
||||
| Platform branch drift | Bug only on one OS or browser | `if platform`/OS/UA branches doing the same job | Shared core plus a thin platform adapter; CI on all platforms |
|
||||
|
||||
## 7. Errors & observability
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Swallowed error | Silent failure; "nothing happens" | Empty catch/except; `catch` that only logs and returns success; ignored error returns | Propagate or convert explicitly; lint for empty handlers |
|
||||
| Error without context | Only "500" or "failed" visible | Generic error constructors or handlers | Structured errors with correlation IDs at one middleware |
|
||||
| Fallback masks failure | Wrong result instead of an error (remapping, guessing, defaults) | Fallback, guess, or "best effort" branches | Fail loudly on ambiguity; log and count fallback usage |
|
||||
|
||||
## 8. Persistence & schema (any ORM / DB)
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Schema-affecting model change without migration | "Unknown column" / missing table | Model diffs vs migration history (only changes that alter the DB schema) | Generated migrations are candidates: review every operation; test the upgrade on representative data; CI drift check |
|
||||
| Dialect-specific SQL | Works on one DB, fails on another | Raw SQL, casts, `ILIKE`, `::type`, vendor functions | Use the ORM abstraction; test against every supported DB |
|
||||
| FK constraint violation | Integrity error on insert or delete | Name the failing operation and constraint first. **Insert**: references to a missing or not-yet-created parent (ordering, validation). **Delete**: every FK to the deleted entity. | Insert: fix ordering or validation, and keep DB enforcement. Delete: choose cascade/restrict/set-null by domain meaning. Never change the policy just to silence the error. Test valid refs, missing refs, and concurrent parent deletion. |
|
||||
| Lazy loading in loops (N+1) | Slow at scale; timeouts | Relationship access inside loops or serializers | Eager-load or batch; query-count assertion in tests |
|
||||
| Column too small / enum value mismatch | Write or read fails for some values | All columns or enums fed by the same source; compare stored values with the code's enum names and values | Reviewed migration plus validation at the boundary; test existing rows and new writes |
|
||||
|
||||
## 9. Runtime staleness (looks like "fix didn't work")
|
||||
|
||||
| Pattern | Symptom | Class search | Source fix / guard |
|
||||
|---|---|---|---|
|
||||
| Process, artifact, cache, or image still old | Tests pass, the system still misbehaves | See verification.md section 5 | Make the executing copy the fixed one. Evidence: loaded version/commit, artifact hash, or image digest, plus fixed behavior. A new PID alone proves nothing. |
|
||||
|
||||
---
|
||||
|
||||
## High-frequency root causes (check these first)
|
||||
|
||||
1. Unverified assumption about a library, API, or framework behavior
|
||||
2. Edge input not handled (empty, null, boundary, unicode, huge)
|
||||
3. Matching without boundaries / normalization
|
||||
4. Write and read sides disagree on format or contract
|
||||
5. Duplicate implementation fixed in only one place
|
||||
6. Stale captured state; async ordering assumed
|
||||
7. Missing cleanup on error paths
|
||||
8. Environment or config differences; fix applied to one env only
|
||||
9. Cache not invalidated; derived data still wrong
|
||||
10. Errors swallowed, or masked by fallback logic
|
||||
11. Transport status checked, business status ignored
|
||||
12. Missing timeout / retry without idempotency
|
||||
|
||||
## Extracting a new pattern (after a fix)
|
||||
|
||||
Add a pattern when no row matched, when your fix strategy or class search differs from the listed one, or when the bug was P0/P1. Generalize project names away (`UserService.login()` → "service method"). The row must have all four columns, and it must be findable by three different keyword searches. Project-specific patterns go to the project's own knowledge files ([project-knowledge.md](project-knowledge.md)), not into this skill.
|
||||
@@ -0,0 +1,236 @@
|
||||
# Class Analysis: from one instance to the whole defect class
|
||||
|
||||
Load this in Phase 3. Read it after the root cause has passed the four gates. Class analysis on top of an unproven cause produces a confident fix for the wrong class.
|
||||
|
||||
The reported bug is one **instance**. The root cause is a **mechanism**. The mechanism rarely exists in exactly one place: it was copied, imitated, or written by someone holding the same wrong assumption. This phase produces a **class inventory** you can defend: every candidate site, a verdict backed by evidence, and a fix plus guard that covers the confirmed ones.
|
||||
|
||||
Everything a search finds is a **candidate**. Similar syntax or shared history only *nominates* a site. It is **confirmed** only by a witness (Step 4).
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Define the class (signature)
|
||||
|
||||
```
|
||||
Class = preconditions + mechanism + violated invariant + observable failure
|
||||
```
|
||||
|
||||
| Part | Question | Timeout example |
|
||||
|---|---|---|
|
||||
| **Mechanism** | What does the code do wrong? | An outbound network call has no end-to-end deadline. (Per-read or inactivity timeouts don't bound the total: a trickling peer can exceed them forever.) |
|
||||
| **Preconditions** | When does that become harmful? | The call is on a request path or holds a lock/connection, and the peer can stall or trickle |
|
||||
| **Violated invariant** | Which rule should have held? | "Every request finishes or fails within its deadline budget" |
|
||||
| **Observable failure** | How does it show? | Worker hangs; pool exhausted; cascading timeouts |
|
||||
|
||||
Also write down:
|
||||
- **Spellings**: every syntactic form the mechanism can take in this codebase's languages. Include library calls, operators, helpers, raw SQL, config, and generated code.
|
||||
- **Wrong assumption**: the belief that produced the bug ("the client has a default timeout"). Code written under the same belief is a candidate even if it looks different.
|
||||
- **Legitimate neighbors**: code that looks like the mechanism but is correct, such as an intentional substring search or a call that is deliberately unbounded and runs in a background job with its own watchdog. These become your exclusion examples and the guard's must-accept cases.
|
||||
|
||||
If you cannot fill in all four parts, the root cause is not mechanistic yet. Go back to Phase 2.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Define the search universe, then search every axis
|
||||
|
||||
### 2a. Search universe (write it down)
|
||||
|
||||
Decide what "everywhere" means **before** searching, and record it in the report:
|
||||
- **Roots**: this repo; submodules; sibling repos or services that share the contract; the generated-code sources; infra/config repos.
|
||||
- **File classes**: tracked, untracked, ignored, hidden (dotfiles, local config), generated, vendored.
|
||||
- **Tools available**: `git` present? A structural/AST search, language server, or IDE "find references"? If something cannot be searched (no access, no history, binary artifacts), list it as a **coverage limit**. Never treat it as "no hits".
|
||||
|
||||
Default tool blind spots:
|
||||
- `git grep` searches only **tracked** files of the **current** repo. Cover the rest in separate passes:
|
||||
1. tracked files including checked-out submodules (`--recurse-submodules`);
|
||||
2. untracked files (`--untracked`), plus ignored ones (`--untracked --no-exclude-standard`), in the root and in each relevant submodule.
|
||||
|
||||
Git refuses to combine `--untracked` with `--recurse-submodules`.
|
||||
- `rg` skips hidden and `.gitignore`d files unless given `--hidden` / `--no-ignore`.
|
||||
- Without git or ripgrep, use whatever recursive search the environment offers (e.g. PowerShell `Get-ChildItem -Recurse | Select-String`). Record the tool used.
|
||||
- Text search misses dynamic dispatch (reflection, `getattr`, string-built SQL or routes, registries) and code produced by generators. Search the string keys, registration tables, and templates too.
|
||||
|
||||
### 2b. Axes
|
||||
|
||||
Run the axes that apply. Record each query and the population it targets.
|
||||
|
||||
| Axis | Question | How to search |
|
||||
|---|---|---|
|
||||
| **Clones** | Was the buggy code copied? | Function name, distinctive literals, comments ("copied from", "keep in sync"), near-identical bodies; `legacy/`, `v1/`, `compat/`, `mobile/`, `worker/`, `scripts/`, sibling services. |
|
||||
| **Same syntax** | Where else does the exact construct appear? | Literal or regex search. |
|
||||
| **Same mechanism, other syntax** | Same mechanism spelled differently? | One query per spelling family and language (from Step 1). |
|
||||
| **Same contract** | Who else produces or consumes the data, format, API, or event? | Field, endpoint, message type, table/column, schema class. Check writers **and** readers. |
|
||||
| **Same assumption** | Where else does code rely on the false belief? | Other users of the same library or API; config defaults. |
|
||||
| **Parallel paths** | A second implementation of the same feature? | v1/v2, sync/async, web/mobile/desktop, batch/stream, flagged old paths, client vs server validation. |
|
||||
| **History** | When did it start? Was it partly fixed elsewhere before? | `git log -S`/`-G`/`--grep`, `git blame` on the root-cause line. A past partial fix is strong evidence of unfixed siblings. |
|
||||
| **Data & artifacts** | Did the bug already write wrong state? | Caches, persisted rows, indexes, generated files, queued messages, exports, memoized results produced by the buggy path. |
|
||||
| **Config / env / platform** | Is the mechanism repeated in env-specific places? | Every config source (env files, manifests, CI variables, secret stores), platform branches, locale/TZ settings, flags. |
|
||||
|
||||
### 2c. Search recipes (git; verified with git 2.45)
|
||||
|
||||
```bash
|
||||
git grep -n -F 'exact literal' # fixed string
|
||||
git grep -n -E 'spellingA|spellingB' -- '*.py' '*.ts' # alternation (ERE), limited by pathspec
|
||||
git grep -n -e 'spellingA' -e 'spellingB' # same, without regex alternation
|
||||
git grep -n -w 'function_name' # whole-word identifier: clones and callers
|
||||
git grep -n -p -F 'snippet' # also print the enclosing function line
|
||||
git grep -n -e 'termA' --and -e 'termB' # both terms on one line
|
||||
git grep -c -E 'pattern' # hits per file (size the class)
|
||||
git grep -n --untracked -F 'snippet' # + untracked files (still skips ignored)
|
||||
git grep -n --untracked --no-exclude-standard -F 'snippet' # + untracked AND ignored files
|
||||
git grep -n --recurse-submodules -F 'snippet' # + tracked files in checked-out submodules (not with --untracked)
|
||||
git grep --no-index -n -E 'pattern' # directory that is not a git repo
|
||||
git log -S'snippet' --oneline # commits that added/removed the snippet
|
||||
git log -G'regex' --oneline # commits whose diff matches the regex
|
||||
git log -i --grep='keyword' --oneline # past commits/fixes mentioning it
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `\b` inside `-E` is a GNU extension and not portable. Use `-w`, or `-P` (PCRE) when git was built with it.
|
||||
- Regex dialects differ between tools. Re-validate a query when you move it to another tool.
|
||||
- Vendored, generated, and build-output hits still get an inventory row and a disposition. Usually you don't edit them directly. Their action is "regenerate from the fixed source", "upgrade the dependency", "rebuild", or "report upstream".
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Recall check
|
||||
|
||||
A search strategy that cannot recover the examples you already know about cannot be trusted to find the ones you don't.
|
||||
|
||||
1. **Known positives**: the **combined** set of queries must return every known instance: the reported one plus any found while debugging.
|
||||
2. **Each query is validated against its own target population.** A query aimed at the original's language and spelling must hit the original. A complementary query (other language, other producer, other spelling) must hit at least one confirmed or deliberately constructed positive example of *its* population. One way: write the variant in a scratch file, search with `--untracked`, then delete the file. A query that has never hit anything has not been shown to work.
|
||||
3. **Variant probe**: name at least one plausible spelling your queries don't target and search for it explicitly. If it hits, widen the queries and rerun.
|
||||
4. **Too many hits?** Group them by shape and sample each group to **prioritize**. Sampling never excludes a group by itself. Exclude a group only when every member provably fails a precondition, for example a stricter query that shows all members are membership tests on set literals. Members you haven't examined stay **unresolved** and get reported.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Class inventory
|
||||
|
||||
Every candidate gets a row and ends with a disposition. "Unsure" must be resolved or reported. It may never silently disappear.
|
||||
|
||||
```markdown
|
||||
| # | Site (path:line) | Candidate via | Preconditions met? | Witness / exclusion evidence | Disposition |
|
||||
|---|------------------|---------------|--------------------|------------------------------|-------------|
|
||||
| 1 | src/a.py:42 (reported) | — | yes, request path | failing test T1 | confirmed → fix now |
|
||||
| 2 | src/legacy/a_v1.py:17 | clones | yes, /v1 still routed | test T2 fails | confirmed → fix now |
|
||||
| 3 | src/admin.py:88 | same mechanism | yes, authz decision | repro script output | confirmed → fix now, **escalate severity** |
|
||||
| 4 | src/util.py:5 | same syntax | — | membership in a set constant (legitimate neighbor) | excluded |
|
||||
| 5 | scripts/backfill.py:30 | same syntax | unknown, offline script | — | unresolved → report, ask owner |
|
||||
| D1 | cache/ (data) | data axis | written by #1 | stale entry shown | confirmed → invalidate if disposable (SKILL.md safety boundaries), else repair with approval |
|
||||
```
|
||||
|
||||
Rules:
|
||||
- **Confirmed needs a witness**: a failing test, a repro, or a concrete trace showing the invariant violated at that site. Security- or data-affecting rows need an executable witness wherever one is feasible.
|
||||
- **Excluded needs a reason**: which precondition fails, or which legitimate-neighbor pattern it is. "Looks fine" is not a reason.
|
||||
- **Class severity = worst confirmed row.** Re-triage on it.
|
||||
- **Dead code** that matches: say it is dead; fix or delete it within scope, or report it.
|
||||
- **Data rows** are part of the class. Fixing code does not repair data the bug already wrote.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Choose the fix level
|
||||
|
||||
| Rung | Fix | Prerequisite | Timeout example |
|
||||
|---|---|---|---|
|
||||
| 1. **Eliminate** | Make the mechanism impossible to express: a type, a data representation, one shared helper, validation at the boundary, a safe default in a base | Confirmed sites share **the same contract and semantics**, and the shared code is yours to change | All outbound calls go through `http_client()`, which always sets a deadline |
|
||||
| 2. **Centralize + guard** | Route sites through one corrected helper; add a guard | Same as rung 1, but legacy call sites stay | Default deadline in the shared session factory; migrate callers |
|
||||
| 3. **Coordinated site fixes + guard** | Apply the correct fix at each confirmed site; add a guard | Sites differ in semantics, ownership, or lifecycle, so a shared abstraction would be forced | Add a call-appropriate `timeout=` at each site |
|
||||
|
||||
- **Don't centralize on resemblance.** Sites that look alike but have different contracts (different deadlines, error semantics, owners) get rung 3. A wrong shared abstraction creates a new class.
|
||||
- A shared fix changes behavior for **every** consumer of the shared code. Run the Phase 4 gates on those consumers, and test previously affected *and* previously correct behavior.
|
||||
- Public contract changes, new dependencies, migrations, and authoritative-data repair need **approval** (disposable data: see SKILL.md safety boundaries). Until then, use the highest rung allowed and record the higher-rung proposal in the report.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Recurrence guard (and test the guard)
|
||||
|
||||
The original-site test proves the fix. A **guard** catches the *next* instance. At least one guard must cover the class.
|
||||
|
||||
| Guard | Catches | Limits |
|
||||
|---|---|---|
|
||||
| Parameterized test over all inventoried entry points | Regressions at known sites | Not new sites |
|
||||
| Property/fuzz test on the shared helper | Unknown inputs reaching the helper | Not code that bypasses the helper |
|
||||
| Lint rule / CI search check (with a reviewed allow-list) | New code written with a known bad spelling | Unlisted spellings; needs an allow-list for legitimate neighbors |
|
||||
| Type / API restriction | New code cannot express the mechanism | Escape hatches (casts, `Any`, raw access) |
|
||||
| Runtime assertion / invariant check | Violations on live paths, whatever their origin | Fires late (at runtime). Must stay enabled in the **deployed** configuration (e.g. Python strips `assert` under `-O`); otherwise it only covers test runs |
|
||||
|
||||
**Guard acceptance test** (do it, then report it). This table is the single authority for what each depth requires:
|
||||
|
||||
| Depth | Guard | Acceptance items required |
|
||||
|---|---|---|
|
||||
| Quick (mini search found no siblings) | The regression test is the guard | 1 |
|
||||
| Standard | At least one class guard | 1, 2, 4, 5, 6 |
|
||||
| Full | At least one class guard | 1–6 |
|
||||
|
||||
The items:
|
||||
1. **Fails on the pre-fix code, for the right reason.** Keep the guard and test harness constant and reverse **only the production fix**, in an isolated copy so the user's other uncommitted changes are never touched. The failure must come from the targeted invariant, not from an import or setup error.
|
||||
2. **Rejects a new defective site.** Add a fresh instance in a scratch file or test and show the guard catches it.
|
||||
3. **Rejects a variant spelling**, and a **helper bypass** if you used rung 1 or 2.
|
||||
4. **Accepts the legitimate neighbors** from Step 1.
|
||||
5. **Actually runs** in the project's normal test or CI path. A guard nobody executes doesn't exist.
|
||||
6. **States its coverage limits** in the report: what it would still miss.
|
||||
|
||||
**Isolated reversal.** This works when the production files had no uncommitted changes at the Phase 1 baseline. Run it from the repository root in a POSIX shell.
|
||||
|
||||
```bash
|
||||
(
|
||||
REPO=$(pwd -P) || exit 1
|
||||
SCRATCH=$(mktemp -d) || exit 1 # fresh, uniquely named, outside the repo
|
||||
# only allowed to write inside the repo? SCRATCH=$(mktemp -d "$REPO/.bugfix-scratch.XXXXXX") || exit 1
|
||||
[ -d "$SCRATCH" ] && [ ! -L "$SCRATCH" ] || exit 1
|
||||
mkdir "$SCRATCH/copy" || exit 1
|
||||
COPY=$(cd "$SCRATCH/copy" && pwd -P) || exit 1
|
||||
[ "$COPY" = "$(cd "$SCRATCH" && pwd -P)/copy" ] && [ "$COPY" != "$REPO" ] || exit 1 # isolation proven before any write
|
||||
git diff -- <production files you fixed> > "$SCRATCH/fix.patch" || exit 1
|
||||
tar --exclude='./.bugfix-scratch.*' -cf "$SCRATCH/tree.tar" . || exit 1 # copy keeps guard + user changes
|
||||
tar -xf "$SCRATCH/tree.tar" -C "$COPY" || exit 1
|
||||
LINKS=$(find "$COPY" -type l -print) || exit 1
|
||||
[ -z "$LINKS" ] || exit 1 # refuse symlinks: they can lead back into the original
|
||||
git -C "$COPY" apply -R "$SCRATCH/fix.patch" || exit 1 # undo only the fix, only in the copy
|
||||
cd "$COPY" || exit 1
|
||||
<run the guard: it must fail on the targeted invariant>
|
||||
)
|
||||
```
|
||||
|
||||
Why it's built this way. Each point was a real failure while testing:
|
||||
- `set -e` is **silently ignored** when the code runs inside `if`, `&&`, `||`, or a tool wrapper, so the recipe would have kept going after errors. Hence an explicit `|| exit 1` on every step.
|
||||
- `mkdir -p`, and `mkdir A A/copy`, write *through* a pre-existing symlink. Hence fresh, uniquely named directories from `mktemp -d`.
|
||||
- With `cmd1 | cmd2`, the shell only sees `cmd2`'s exit status, so a failing producer goes unnoticed. Hence no pipes.
|
||||
- Where `tar` recreates symlinks, a copied absolute or escaping link still points at the **original**. Writing through it changed the real file in testing. Hence the copy refuses any symlink before anything runs in it.
|
||||
|
||||
It was tested with GNU tar 1.35 (Git Bash) under an `if` context. Cases covered:
|
||||
- scratch outside the repo and inside it;
|
||||
- pre-planted symlinks to the repo;
|
||||
- a failing `tar`;
|
||||
- a failing `mktemp`;
|
||||
- absolute and relative symlinks with native symlink extraction enabled.
|
||||
|
||||
Every failure case stopped with the original fix and the user's changes intact, and nothing was written into the repo.
|
||||
|
||||
**Limitation**: a repo containing **any** symlink makes the recipe stop (safely). On Windows, Git Bash's tar may already fail at extraction; elsewhere, the symlink check refuses. In that case, reverse by hand in a copy whose links you've checked point inside it. If the production files already had user changes, reverse only your own hunks in the copy. Delete `SCRATCH` afterwards.
|
||||
|
||||
---
|
||||
|
||||
## Step 7 — Report the class
|
||||
|
||||
The Phase 7 report must include:
|
||||
- the class definition (four parts) and the legitimate neighbors;
|
||||
- the search universe and coverage limits;
|
||||
- the axes searched and the queries used;
|
||||
- the recall-check results;
|
||||
- the inventory with dispositions and evidence;
|
||||
- the rung chosen, and why the higher rungs were not taken;
|
||||
- the guard, its acceptance-test results, and its limits;
|
||||
- unresolved or deferred rows, each with an owner or a question.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes this phase exists to stop
|
||||
|
||||
| Failure | Countermeasure |
|
||||
|---|---|
|
||||
| Fixed the reported file; the copy in `legacy/` still ships the bug | Clones + parallel-paths axes |
|
||||
| Searched one spelling; the same mechanism written with another API | Spellings list, per-population validation, variant probe |
|
||||
| Search silently skipped untracked files, submodules, or ignored config | Search universe + coverage limits |
|
||||
| Code fixed; the cache or DB still serves wrong results | Data axis + Phase 6 runtime/data check |
|
||||
| "Class fix" rewrote unrelated code that only looked similar | Candidate ≠ confirmed; witness required; rung prerequisites |
|
||||
| Reported a cosmetic bug, missed that a sibling was a security hole | Class severity = worst confirmed row |
|
||||
| Guard exists but would never fail, or never runs | Guard acceptance test |
|
||||
@@ -0,0 +1,56 @@
|
||||
# Frontend Playbook
|
||||
|
||||
Symptom → detection → fix → verify entries for **web** UI and client bugs. React is used where an example needs a framework; the mechanisms (stale closures, races, missing cleanup, unstable identity) exist in every UI framework, native and terminal UIs included. The summary table lives in [bug-patterns.md](bug-patterns.md).
|
||||
|
||||
**UI bugs need runtime evidence from the actual surface before a fix.** For web UIs: screenshot or recording, console, DOM or computed styles, network log, and exact repro steps. For native and terminal UIs, use the equivalents listed in SKILL.md Phase 1. Code reading alone produces confident wrong fixes for layout, timing, and state bugs. If you can't run the surface in this environment, label the diagnosis and fix **provisional** and list the runtime checks the user must do.
|
||||
|
||||
Framework behavior and warnings differ by version. Check the project's installed versions before trusting an entry.
|
||||
|
||||
---
|
||||
|
||||
## 1. Reactivity & lifecycle
|
||||
|
||||
**1.1 Effect / watcher dependencies wrong.** *Symptom*: side effect uses old props or state; refetch doesn't happen. *Detect*: effect reads values that aren't in its dependency list; suppressed lint warnings (`exhaustive-deps` disables). *Fix*: declare the real dependencies; move event logic into handlers. Don't silence the linter. *Class search*: every suppression comment of that lint rule. *Verify*: change each dependency; the effect re-runs.
|
||||
|
||||
**1.2 Timer or subscription captures old values.** *Fix*: functional state updates; refs for mutable values; re-subscribe when inputs change. *Verify*: changing a parameter takes effect immediately.
|
||||
|
||||
**1.3 Double-invoked effects in dev/strict mode.** *Symptom*: duplicate connections or requests in dev only. *Fix*: proper cleanup (unsubscribe, abort, clear timers); idempotent setup. *Verify*: mount/unmount cycles don't grow resources.
|
||||
|
||||
**1.4 Async result applied after unmount or after newer input.** *Fix*: AbortController or cancel tokens; ignore stale responses with a request sequence ID; clean up subscriptions and timers. *Verify directly*: rapid navigation leaves no overwritten data and no leaked subscriptions or timers. Framework warnings are version-dependent (React 18 removed the unmounted-setState warning), so a missing warning proves nothing.
|
||||
|
||||
**1.5 Controlled ↔ uncontrolled input switch.** *Fix*: always controlled (default `""`) or always uncontrolled. *Verify*: no warning during initial render, async load, or reset.
|
||||
|
||||
**1.6 Unstable list keys / identity.** *Symptom*: row state jumps to the wrong item. *Fix*: stable business IDs as keys. *Verify*: sort, filter, insert, and delete keep per-row state correct.
|
||||
|
||||
**1.7 Store update not re-rendering.** *Symptom*: state changed, UI didn't update. *Mechanism*: in-place mutation keeps the same reference. *Fix*: immutable updates; selector equality checks. *Class search*: every in-place mutation of store state.
|
||||
|
||||
## 2. Data flow, caching & races
|
||||
|
||||
**2.1 Late response overwrites newer state.** *Fix*: cancel superseded requests, or apply only the latest response (sequence guard). *Verify*: rapid input changes end on the last input's data.
|
||||
|
||||
**2.2 Cache not invalidated after mutation.** *Fix*: invalidate the affected query keys in the mutation layer, or update the cache from the server response. *Class search*: every screen that reads the mutated entity (list, detail, selectors, counts). *Verify*: all of them agree after the write.
|
||||
|
||||
**2.3 CORS / preflight.** *Detect*: is a real preflight sent (`OPTIONS` with `Access-Control-Request-*`)? Which `Access-Control-Allow-*` headers come back, for which origin? *Fix*: server-side (see backend 1.4); `credentials` requires Allow-Credentials and an explicit origin. *Verify*: the allowed origin works, a disallowed origin is blocked, and unauthenticated protected requests are still rejected.
|
||||
|
||||
**2.4 Cookie / SameSite session loss.** *Fix*: set Domain/SameSite/Secure for the actual deployment topology; cross-site requires `SameSite=None; Secure`. *Verify*: same-site, cross-subdomain, and cross-site flows keep the session.
|
||||
|
||||
**2.5 Streaming events (SSE / WebSocket) mishandled.** *Symptom*: missing or duplicated chunks; UI stuck "loading". *Mechanism*: unhandled event types, IDs that differ between start and end events, no resume or deduplication, no explicit completion handling. *Fix*: an explicit event-type switch with a default branch that logs; one ID scheme. Distinguish **application completion or fatal failure** (end the UI state) from **recoverable transport interruption**: `EventSource` fires `error` while it auto-reconnects, so keep the state and resume using the last event ID, deduplicating replays. *Class search*: every consumer of the stream. *Verify*: interrupt the connection mid-stream; delivery resumes without loss or duplicates.
|
||||
|
||||
## 3. Rendering & layout
|
||||
|
||||
**3.1 Data present, not rendered.** *Detect*: compare API fields with rendered fields; look for destructured-but-unused variables. *Fix*: render the missing fields, or explicitly decide not to. *Class search*: other views of the same entity.
|
||||
|
||||
**3.2 Clipped / hidden overlay (menu, tooltip, dialog).** *Detect*: inspect ancestors for `overflow: hidden`, `transform`, and stacking contexts (`z-index`). *Fix*: render overlays in a portal or top layer. *Class search*: every overlay rendered inside a scroll or clip container.
|
||||
|
||||
**3.3 Hydration mismatch (SSR).** *Detect*: render depends on window, random values, time, or locale. *Fix*: move browser-only logic to the client; stabilize time, random, and locale. *Verify*: no hydration warnings; stable on refresh.
|
||||
|
||||
**3.4 Markdown / HTML rendering.** *Symptom*: broken nested code fences; raw HTML or script injection. *Fix*: a real parser plus a sanitizer; never `innerHTML` unsanitized input. *Class search*: every raw-HTML sink.
|
||||
|
||||
## Minimal frontend verification
|
||||
|
||||
- [ ] Runtime evidence captured before the fix, from the actual surface (web: screenshot, console, DOM, network)
|
||||
- [ ] Repro steps no longer reproduce; the automated test where feasible
|
||||
- [ ] Rapid repeated interactions (route switches, filter changes) show no races
|
||||
- [ ] No new console errors or warnings
|
||||
- [ ] All surfaces showing the same entity are consistent
|
||||
- [ ] Viewport sizes / browsers relevant to the bug checked
|
||||
@@ -0,0 +1,99 @@
|
||||
# Project Knowledge: profile, records, patterns, blind spots
|
||||
|
||||
This skill is generic. Knowledge about a specific project belongs **in that project**, not in this skill folder: hot zones, parallel code paths, known duplicates, past bugs, and commands.
|
||||
|
||||
## Where it lives
|
||||
|
||||
- If the project already keeps this kind of knowledge (`AGENTS.md`, `CLAUDE.md`, `docs/`, ADRs, a wiki folder), **use and extend that**.
|
||||
- Otherwise use a `.bugfix/` folder at the repository root:
|
||||
|
||||
```
|
||||
.bugfix/
|
||||
profile.md # architecture map, hot zones, parallel paths, known duplicates, commands, env gotchas
|
||||
records.md # one entry per fixed bug, newest first
|
||||
patterns.md # project-specific patterns not in the universal catalogue
|
||||
blind-spots.md # project-specific checks, same format as references/blind-spots.md
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Read** (Phase 2, step 1): search these files for the symptom, the module, and the mechanism before forming hypotheses. If none exist, continue without them.
|
||||
2. **Write only with consent.** Creating or editing these files changes the user's repository. If the user hasn't asked for knowledge capture, put the proposed entries in your final report and ask. Once the user agrees for a project, keep it up to date on later fixes there.
|
||||
3. **Never write project knowledge into this skill's own folder**, and never edit the skill during a bugfix. Improvements to the skill go in the report as proposals.
|
||||
4. **Redact** secrets, tokens, personal data, customer identifiers, and internal hostnames before writing.
|
||||
5. Keep entries short and searchable. Use the words someone would search for next time: symptom, module, mechanism.
|
||||
|
||||
---
|
||||
|
||||
## Template: `profile.md`
|
||||
|
||||
```markdown
|
||||
# Bugfix profile: <project>
|
||||
|
||||
## Commands (verified YYYY-MM-DD)
|
||||
- lint: <cmd> - typecheck: <cmd>
|
||||
- unit tests: <cmd> - integration/e2e: <cmd>
|
||||
- build: <cmd> - run/restart locally: <cmd>
|
||||
- health check / smoke: <cmd>
|
||||
|
||||
## Architecture map (only what matters for debugging)
|
||||
<tree or bullet list of main modules and what they own>
|
||||
|
||||
## Parallel paths (fix one -> check the other)
|
||||
| Feature | Path A | Path B |
|
||||
|
||||
## Known duplicate implementations
|
||||
| Function / behavior | Primary | Alternate |
|
||||
|
||||
## Hot zones (historically bug-dense or fragile)
|
||||
| Zone | Files | Why risky |
|
||||
|
||||
## Framework / platform gotchas (verified)
|
||||
| Area | Assumption | Reality | Evidence |
|
||||
```
|
||||
|
||||
## Template: `records.md` entry
|
||||
|
||||
```markdown
|
||||
### [BUG-NNN] <short symptom>
|
||||
- Date: YYYY-MM-DD Severity: P0-P3 Depth: Quick/Standard/Full
|
||||
- Signature: <mechanism in kind-of-code causes failure when condition>
|
||||
- Root cause: <1-2 sentences, mechanistic>
|
||||
- Class inventory: <n fixed> fixed, <n reported> reported, <n not affected>; key sites: path:line, ...
|
||||
- Fix level: eliminate / centralize+guard / patch+guard
|
||||
- Guard: <test/lint/type/assertion> (shown failing on pre-fix code: yes)
|
||||
- Escape analysis: <why it wasn't caught before; what now prevents it>
|
||||
- Files: <paths>
|
||||
- Follow-ups: <reported items awaiting approval>
|
||||
```
|
||||
|
||||
## Template: `patterns.md` row
|
||||
|
||||
```markdown
|
||||
| Pattern | Symptom | Class search | Source fix / guard | Seen in |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Worked example: profile excerpt from the project this skill was derived from
|
||||
|
||||
The original skill was written for one specific app: a FastAPI + LangGraph + React agent platform. Its project section is a good example of what a profile should capture. Here is a condensed version:
|
||||
|
||||
- **Parallel paths**: agent execution ran through either a direct executor (`chat_service → agent_executor → tool_call_parser`) or a graph runtime (`agent_graph → llm_node → tool_node`). *Any fix to tool or agent logic in one path had to be checked in the other.*
|
||||
- **Known duplicates**: `parse_simulated_tool_calls` existed in both `services/tool_call_parser.py` and `core/graph/nodes/llm_node.py`. Tool loading, token counting, and memory management each had two homes.
|
||||
- **Hot zones**: regex-based tool-call parsing, a 3000+ LOC executor, argument remapping that guessed parameter names, LLM streaming with reasoning fields, tool-server lifecycle, and SSE handling in the chat UI.
|
||||
- **Gotchas recorded there**: config merge was last-wins; on Windows, the event loop chosen by the dev server decided whether asyncio subprocesses worked at all (they need the Proactor loop; the Selector loop raises `NotImplementedError`), so check the loop actually in use; the dev server kept serving old code until that specific process was restarted.
|
||||
|
||||
And its first record shows the class approach working:
|
||||
|
||||
```markdown
|
||||
### [BUG-001] Multi-argument tool calls parsed as one garbled argument
|
||||
- Signature: tool-call text parsed by regex that assumes a single argument and has no
|
||||
identifier boundaries -> wrong/merged args, and tool names matching as substrings of longer names
|
||||
- Class inventory: 2 parsers (service + graph node duplicate) fixed; all name-matching regexes got
|
||||
boundaries; remapper fallback reported (it masked the failure by guessing params alphabetically)
|
||||
- Fix level: centralize (graph node delegates to the single service parser) + AST-based parsing
|
||||
- Runtime: the first "fix" didn't take effect - the running server process still had the old code
|
||||
loaded until it was restarted
|
||||
- Escape analysis: no tests for multi-arg calls; duplicate parser unknown to the fixer
|
||||
```
|
||||
@@ -0,0 +1,114 @@
|
||||
# Root Cause Analysis
|
||||
|
||||
Load this in Phase 2. For bugs that cross processes, services, queues, browsers, or streams, also load [system-rca.md](system-rca.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Hypothesis ladder
|
||||
|
||||
When the cause isn't already evident from the evidence, list 3–5 hypotheses **before** testing any of them. A single hypothesis turns into confirmation bias. If the repro and the gates directly confirm one mechanism, one well-tested hypothesis is enough. Record why.
|
||||
|
||||
| # | Hypothesis (mechanism, not symptom) | Likelihood | Rejection test (proves it is NOT this) | Confirmation test (proves it IS this) | Result |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | | H/M/L | | | |
|
||||
|
||||
Rules:
|
||||
- Each hypothesis must be **falsifiable** with a concrete test or observation.
|
||||
- Run the **cheapest rejection tests first**. Eliminating is faster than proving.
|
||||
- Change **one variable at a time**. If two things changed and the bug went away, you learned nothing.
|
||||
- Record rejected hypotheses with their evidence. A rejection is progress. It belongs in the report and stops others from re-walking the same path.
|
||||
- **No-progress budget**: if two consecutive rounds of experiments produce no new discriminating evidence, stop guessing. Change the method: probes, bisect, a smaller repro, a different environment. Or report the eliminated hypotheses, the remaining uncertainty, and the next discriminating experiment, and ask.
|
||||
|
||||
## 2. Five whys (until you reach a mechanism)
|
||||
|
||||
Keep asking "why" until the answer is a **mechanism you can point at in code, config, or data**, plus the **condition** that triggers it. Stop there. Asking further usually drifts into process blame, and that belongs in the escape analysis (Phase 7).
|
||||
|
||||
```
|
||||
Symptom: invoices created after 17:00 show tomorrow's date
|
||||
Why? the date is rendered from a timestamp interpreted as UTC
|
||||
Why? the timestamp was stored without a timezone (naive local time)
|
||||
Why? the model layer calls now() instead of now(UTC), and the serializer assumes UTC <- mechanism + condition
|
||||
```
|
||||
|
||||
## 3. Root-cause confirmation gates (all four required)
|
||||
|
||||
| Gate | Pass criterion | Typical evidence |
|
||||
|---|---|---|
|
||||
| **Reproducible** | You can trigger the symptom on demand in a controlled setup. | A failing automated test, or a scripted repro with its output. |
|
||||
| **Causal** | A minimal change at the suspected cause **restores the intended behavior**: correct output, invariant holds. Mere absence of the error does not count. | The test asserts the *correct result* and turns green with only that change. |
|
||||
| **Reversible** | Undoing that change brings the failure back. | Revert or stash in a test or fixture, never in a shared environment, and watch the test fail. |
|
||||
| **Mechanistic** | You can name the exact code path or state transition, the invariant it violates, and *why* that produces the symptom. | file:line plus a step-by-step trace. |
|
||||
|
||||
**Gate experiments are diagnostic, not the fix.** The minimal change used for "causal" and "reversible" is a probe. Apply it in a test, a scratch copy, or in place and revert it immediately. The real fix is written in Phase 5, after class analysis and scope (Phases 3–4) have decided its shape and level.
|
||||
|
||||
Watch for **suppression passing as causation**. A catch-and-ignore, a default fallback, or a retry can make the symptom vanish and reappear on revert, so it passes "causal" and "reversible" while the defect survives. Assert the intended output, not just "no error". Include a **negative control**: an input that was already correct must stay correct.
|
||||
|
||||
For **nondeterministic** failures, gates are judged on measured rates: failures over N runs before and after (see section 6, step 4). Report the confidence; don't claim certainty.
|
||||
|
||||
Passing "causal" without "mechanistic" means you found a workaround, not a cause. If you cannot reach "reproducible", see section 6.
|
||||
|
||||
## 4. Classify the layer before fixing
|
||||
|
||||
Fix where the defect **originates**, not where it **surfaces**.
|
||||
|
||||
| Origin layer | Signs | Fix belongs in | Not in |
|
||||
|---|---|---|---|
|
||||
| Input / boundary | Bad data accepted, then fails deep inside | Validation or normalization at the boundary | Each downstream consumer |
|
||||
| Config / environment | Works in one env, not another | Config, env files, deployment manifests (all of them) | Code that special-cases the env |
|
||||
| Data | Code is right, stored state is wrong | Data repair plus whatever wrote the bad data | Read-side workarounds only |
|
||||
| Contract / schema | Producer and consumer disagree | The contract **and** both sides, kept in sync | One side only |
|
||||
| Shared code (helper / base / framework usage) | Same failure in many places **that use the same shared code under the same contract** | The shared code | Each caller one by one |
|
||||
| Repeated local mistake | Same mistake written independently in places with different contracts | Each site (coordinated), plus a guard | A new forced abstraction |
|
||||
| Single component | Only one consumer affected, and nothing shared is wrong | That component | Shared or framework code |
|
||||
| Architecture / design | Fix needs a redesign, or keeps recurring in new forms | Propose a design change and get approval; apply a safe interim fix | Silent large refactor |
|
||||
| External dependency | Library or service behaves differently than assumed | Pin, adapt, or wrap at the integration point; report upstream | Scattered workarounds |
|
||||
| Capability / model limit (AI/LLM features) | Model cannot reliably do the task | Change the approach: tools, structure, validation, fallback | Prompt-tweaking symptoms one by one |
|
||||
|
||||
## 5. Framework / library assumption audit
|
||||
|
||||
When the fix depends on how a framework or library behaves:
|
||||
1. Write down each assumption ("config files are merged first-wins", "the client has a default timeout").
|
||||
2. Verify each one against the **installed version's source code** or a minimal experiment. Docs, comments, and memory are hypotheses.
|
||||
3. Cite the evidence in a code comment next to the fix: the source location or the experiment.
|
||||
|
||||
Commonly wrong assumptions: merge order (often last-wins), lazy vs eager loading, default timeouts (often none), serialization dropping unset or unknown fields, event-loop or threading model per platform, caching of compiled modules, timezone handling of "naive" timestamps.
|
||||
|
||||
## 6. When you cannot reproduce
|
||||
|
||||
Do **not** ship a speculative fix as if it were confirmed. Instead:
|
||||
|
||||
1. **Narrow the environment.** Diff versions, config, data, OS, locale/timezone, feature flags, and user permissions between where it fails and where it works.
|
||||
2. **Bisect the history** when there is a known-good version:
|
||||
```bash
|
||||
git bisect start <bad-commit> <good-commit>
|
||||
git bisect run <harness>
|
||||
git bisect reset
|
||||
```
|
||||
The harness decides each commit:
|
||||
- exit **0** = behaves correctly;
|
||||
- exit **1** = *this* bug is present;
|
||||
- exit **125** = can't test this commit (doesn't build, missing dependency), so skip it.
|
||||
|
||||
Keep the harness script **outside** the repository so it exists at every historical commit. Make it check for the specific failure, not "any error", or unrelated breakage gets blamed. Before trusting the result, run the harness by hand on both endpoints.
|
||||
3. **Instrument.** Add temporary probes (logs with correlation IDs, assertions, counters) at the edges of the suspected path ([system-rca.md](system-rca.md)). Remove them or keep them deliberately afterwards.
|
||||
4. **Stress intermittent bugs.** Loop the repro N times, add concurrency, inject latency or failures, and record the frequency. Choose N from the baseline rate. If the runs are **independent**, zero failures in n runs bounds the true failure rate below about 3/n at 95% confidence (the "rule of three"). For a bug that failed 10% of the time, that means at least 30–50 clean runs; state the bound you achieved. Independence means you reset the relevant state between runs (fresh process, data, cache, seed) and vary the trigger conditions. Runs that share one startup state, connection, or seed are correlated. For those, report the observations but don't claim the bound.
|
||||
5. If it is still not reproducible, report the evidence gathered, the hypotheses still open, and the probes left in place. Then ask how to proceed. A **defensive** change (extra validation, better errors) is allowed only if it is labelled as such.
|
||||
|
||||
## 7. Evidence bundle (attach to the report)
|
||||
|
||||
```markdown
|
||||
### Trigger
|
||||
- Input / params:
|
||||
- Environment: OS, runtime and dependency versions, config/flags, commit
|
||||
- Sequence / timing:
|
||||
|
||||
### Observed vs expected
|
||||
- Observed: full error text, stack trace, wrong output, screenshot or recording
|
||||
- Expected:
|
||||
|
||||
### Correlation
|
||||
- request/trace/session IDs, log lines with timestamps
|
||||
|
||||
### Redaction
|
||||
- Secrets, tokens, cookies, personal data, internal URLs removed: yes
|
||||
```
|
||||
@@ -0,0 +1,100 @@
|
||||
# System-Level Root Cause Analysis
|
||||
|
||||
Use this for bugs that cross a boundary: client and server, service and service, process and worker, queue or stream, browser automation, IPC, cache layer, or any flow through more than one runtime.
|
||||
|
||||
The symptom shows **where the failure surfaced**, not where it **originated**:
|
||||
|
||||
| Symptom | Plausible origins |
|
||||
|---|---|
|
||||
| Element / record not found | Wrong session or tenant, not ready yet, wrong frame or context, stale ID, redirect, routed to the wrong instance |
|
||||
| Timeout | Slow or hung peer, wrong endpoint, exhausted pool, deadlock, missing ack, retry storm |
|
||||
| Data mismatch | Stale cache, race, schema drift between producer and consumer, wrong routing key, partial write |
|
||||
| Connection lost | Peer crash, idle timeout, proxy limit, client disconnect, resource limit |
|
||||
|
||||
---
|
||||
|
||||
## Method: walk the chain edge by edge
|
||||
|
||||
### 1. Draw the chain
|
||||
|
||||
List every participant from trigger to result. Include the invisible ones: proxies, caches, queues, serializers, auth middleware.
|
||||
|
||||
```
|
||||
[Trigger] -> [Producer] -> [Transport] -> [Consumer] -> [Side effect / Result]
|
||||
```
|
||||
|
||||
### 2. Define handshake evidence per edge
|
||||
|
||||
For each edge, write what *correct* looks like **before** you look at the actual data:
|
||||
|
||||
| Edge | From -> To | Evidence of a correct handshake |
|
||||
|---|---|---|
|
||||
| E1 | client -> API | Request has the expected params, auth, and correlation ID |
|
||||
| E2 | API -> queue | Message published with the expected payload and routing key |
|
||||
| E3 | queue -> worker | Delivered per the queue's contract (at-least-once means duplicates are possible and processing must be idempotent); acked only **after** durable processing; the effect happens exactly as often as the contract requires |
|
||||
| E4 | worker -> store | Row written with the expected values |
|
||||
|
||||
### 3. Insert probes before changing behavior
|
||||
|
||||
| Probe | Where | Records |
|
||||
|---|---|---|
|
||||
| Structured log | Each edge | `edge=E2 from=api to=queue id=<corr> payload_digest=<...>` |
|
||||
| Debug field in a response | Tool or API result | IDs or routing info actually used |
|
||||
| Counter or timer | Hot edges | Count, latency, error rate |
|
||||
| Assertion | Invariant points | `session_id is not None`, `version == expected` |
|
||||
|
||||
Redact secrets, tokens, cookies, and personal data in probes. Log digests or IDs, not raw payloads. Mark every probe as temporary and remove it after the fix, unless you deliberately keep it as observability.
|
||||
|
||||
### 4. Find the first broken edge
|
||||
|
||||
Run the repro with the probes in place:
|
||||
|
||||
```
|
||||
E1 ok -> E2 ok -> E3 FAIL -> E4 ?
|
||||
^ the failure becomes visible here
|
||||
```
|
||||
|
||||
"ok" must mean **semantically correct**, not merely "received". Check the payload against the contract: right IDs, values, version, ordering. An upstream defect such as a wrong but well-formed value can pass every handshake and only blow up downstream. So the first *visibly* broken edge bounds where to look, but re-check the content of the earlier edges before concluding the root cause lives at E3.
|
||||
|
||||
Then switch to [root-cause.md](root-cause.md) hypotheses, scoped to that region.
|
||||
|
||||
---
|
||||
|
||||
## Bug families and the evidence they need
|
||||
|
||||
| Family | Symptom | Evidence to collect | Usual mechanisms |
|
||||
|---|---|---|---|
|
||||
| **Routing** | Action hits the wrong target (session, tenant, instance, handler) | Routing key at every hop, actual vs expected target | Stale key, missing key, collision, default fallback route |
|
||||
| **Timing** | Intermittent; passes on retry | Timestamps and state at each edge; failure rate | Race, too-short timeout, no retry/backoff, ordering assumption |
|
||||
| **State** | Wrong, stale, or missing data | State snapshot per edge (cache, DB, memory) | Missing invalidation, stale closure or captured value, missing sync |
|
||||
| **Transport** | Lost, duplicated, or corrupted messages | Message at send and receive, sequence numbers, acks | No ack/retry, no idempotency, buffer limit, encoding mismatch |
|
||||
| **Contract** | Consumer misreads producer | Payload on the wire vs consumer schema | Field renamed or dropped, type changed, version skew |
|
||||
|
||||
## Cross-boundary patterns
|
||||
|
||||
| Pattern | Description | Fix direction |
|
||||
|---|---|---|
|
||||
| Session / tenant mismatch | Front end holds A, back end routes to B | Validate the routing key at each hop; fail loudly on mismatch |
|
||||
| Not-ready-yet | "Loaded" signal fires before the state you need exists | Wait for the specific condition, not a generic idle or load event |
|
||||
| Context mismatch for in-flight work | Work started under context A (tenant, account, session) executes after the current context switched to B | Bind the **operation's** identity when it starts and carry it through. On a mismatch, cancel or reject; never silently retarget A's work to B. Read the latest value only for settings whose contract says "latest wins". Test switching context during in-flight work. |
|
||||
| Side effect replayed from cache | A non-idempotent call was cached or retried | Mark side effects uncacheable; add idempotency keys |
|
||||
| Version skew | Producer and consumer deployed at different versions | Version the contract; stay compatible across one version |
|
||||
|
||||
---
|
||||
|
||||
## Class analysis at system level
|
||||
|
||||
After you find the broken edge, ask what **other edges share the same mechanism**: other consumers of the same queue or topic, other clients of the same endpoint, other handlers that capture the same context. Feed these into the class inventory ([class-analysis.md](class-analysis.md)). System bugs are rarely confined to one edge.
|
||||
|
||||
## Template
|
||||
|
||||
```markdown
|
||||
## System RCA
|
||||
1. Chain: A -> B -> C -> D
|
||||
2. Handshake evidence per edge: (table)
|
||||
3. Probes inserted (temporary): (where / what)
|
||||
4. Last good edge: E_n First bad edge: E_n+1
|
||||
5. Mechanism: <one sentence>
|
||||
6. Other edges with the same mechanism: (-> class inventory)
|
||||
7. Fix + probes removed / kept on purpose
|
||||
```
|
||||
@@ -0,0 +1,128 @@
|
||||
# Verification: no regression, fix is live, class is closed
|
||||
|
||||
Load this in Phase 6. In Full-depth runs, also use it in Phase 1 to capture the baseline.
|
||||
|
||||
---
|
||||
|
||||
## 0. Discover the project's real commands first
|
||||
|
||||
Don't assume `npm test` or `pytest`. Find what *this* project runs, in this order:
|
||||
1. CI workflow files: the most trustworthy source of what must pass.
|
||||
2. The task runner: `Makefile`, `justfile`, `package.json` scripts, `pyproject.toml`/`tox.ini`/`noxfile.py`, `build.gradle`, `Cargo.toml`, `go.mod`.
|
||||
3. README / CONTRIBUTING.
|
||||
4. A project profile, if the project keeps one (see [project-knowledge.md](project-knowledge.md)).
|
||||
|
||||
```bash
|
||||
git ls-files | grep -E '(^|/)(Makefile|justfile|package\.json|pyproject\.toml|tox\.ini|noxfile\.py|Cargo\.toml|go\.mod|build\.gradle(\.kts)?|pom\.xml)$|^\.github/workflows/|\.gitlab-ci\.yml$'
|
||||
```
|
||||
|
||||
PowerShell (no `grep`):
|
||||
|
||||
```powershell
|
||||
git ls-files | Select-String -Pattern '(^|/)(Makefile|justfile|package\.json|pyproject\.toml|tox\.ini|noxfile\.py|Cargo\.toml|go\.mod|build\.gradle(\.kts)?|pom\.xml)$|^\.github/workflows/|\.gitlab-ci\.yml$'
|
||||
```
|
||||
|
||||
No git: list the files directly (e.g. `Get-ChildItem -Recurse -File -Force` or `find . -type f`) and look for the same names.
|
||||
|
||||
Record the exact commands you will use for lint, typecheck, build, and tests.
|
||||
|
||||
**Finding a command doesn't authorize running it.** Local, read-only, or sandboxed checks (lint, unit tests, a local build) are fine. Commands that deploy, migrate, seed or reset databases, restart shared services, hit external systems, or cost money need the user's approval.
|
||||
|
||||
---
|
||||
|
||||
## 1. Baseline before the fix (Phase 1)
|
||||
|
||||
| # | Check | Evidence |
|
||||
|---|---|---|
|
||||
| A1 | Bug reproduced, ideally as a failing automated test | Test name + failing output |
|
||||
| A2 | Test/lint/build baseline captured on **unmodified** code | Pass/fail counts and the names of tests already failing |
|
||||
| A3 | Working behaviors near the bug listed (what must still work) | Short list |
|
||||
| A4 | Pre-existing uncommitted changes recorded (`git status`) | List. You must never disturb them, and the isolated guard reversal depends on knowing them. |
|
||||
|
||||
Tests that already fail are **not** regressions caused by your fix. But you can only prove that if you recorded them before changing anything.
|
||||
|
||||
## 2. Fix quality (Phase 5)
|
||||
|
||||
| # | Check | Criterion |
|
||||
|---|---|---|
|
||||
| B1 | Single concern | The diff contains only the class fix, the guard, and the tests. No drive-by refactors. |
|
||||
| B2 | Origin layer | Fix is at the layer named in Phase 2. |
|
||||
| B3 | Size proportional | Per-site changes are minimal; repeated identical edits across inventoried sites are fine; a large shared change needs a stated reason. |
|
||||
| B4 | Contracts | No public contract, schema, or dependency change without approval. Return types and shapes, and behavior for inputs **outside** the bug (empty, null, whitespace, case, error responses), are part of the contract. If the fix changes it, list the change in the report, and ask first when the interface is public. |
|
||||
| B5 | Boundaries | If the fix itself matches strings or patterns, it uses an explicit grammar (what counts as a token or identifier). Tests cover equal, prefix, suffix, embedded, empty, case, unicode, delimiters, quoting, and multiline input. Prefer parse-then-compare-exactly over a cleverer regex. |
|
||||
|
||||
## 3. Correctness (after the fix)
|
||||
|
||||
| # | Check | Expected |
|
||||
|---|---|---|
|
||||
| C1 | Original repro / failing test | Passes |
|
||||
| C2 | Root cause addressed | Fix matches the Phase 2 mechanism; it does not just hide the symptom |
|
||||
| C3 | Each inventoried "fix now" site | Verified individually (test case or repro) |
|
||||
| C4 | Edge inputs: null/empty, min/max/zero, unicode, very large | Handled |
|
||||
| C5 | Failure paths: dependency down, timeout, invalid data | Fail gracefully and visibly; nothing swallowed |
|
||||
| C6 | Guard acceptance test | The items required for your depth pass (the depth table in [class-analysis.md](class-analysis.md) Step 6) |
|
||||
|
||||
## 4. Regression (the gate)
|
||||
|
||||
| # | Level | Check |
|
||||
|---|---|---|
|
||||
| D1 | L1 direct callers | Tests or exercises for code calling the changed code |
|
||||
| D2 | L2 indirect callers | Callers of L1, including re-exports, dynamic dispatch, and string-registered handlers |
|
||||
| D3 | L3 other modules or services sharing the contract | Their tests, or a contract check |
|
||||
| D4 | L4 user-facing surface | The real entry point (CLI, endpoint, UI flow) behaves correctly |
|
||||
| D5 | Data | No corruption; repaired or invalidated data is consistent |
|
||||
| D6 | Baseline comparison | Full relevant suite: same or better than A2 |
|
||||
| D7 | Performance / concurrency (when relevant) | No N+1 queries, leaks, or latency jump; shared state still safe under concurrent calls |
|
||||
|
||||
If D6 regresses, the fix is not done. Revert or repair. Don't explain it away.
|
||||
|
||||
## 5. Runtime: is the fixed code what actually runs?
|
||||
|
||||
Tests can pass while the running system still serves old code or old data. Strong evidence has two parts:
|
||||
1. **The executing artifact is the fixed one.** One of:
|
||||
- a version or commit the running process reports;
|
||||
- the build hash or image digest of what is deployed;
|
||||
- a process **started after the change** whose loaded files are the edited ones.
|
||||
2. **The repaired path behaves correctly** when exercised through the real entry point.
|
||||
|
||||
A new PID alone proves nothing: a new process can load the wrong checkout, and a reload can keep the PID. A loaded file path alone doesn't prove the in-memory code is current, because the process may have loaded the file before your edit. "Health check returns 200" proves only that something is running.
|
||||
|
||||
Check whichever sources apply. **Only act on a source when evidence implicates it.** Don't routinely delete caches or restart things.
|
||||
|
||||
| Staleness source | Symptom | Make it live | Evidence |
|
||||
|---|---|---|---|
|
||||
| Long-running process (server, worker, daemon, REPL, notebook kernel) | Behavior unchanged after the edit | Restart **that** process, identified by PID/command line; never kill by name pattern. Ask first if it is shared or not yours. | Reported version/commit, or start time after the change plus the edited files loaded, plus fixed behavior |
|
||||
| Several workers or replicas | Fixed on some requests, not others | Restart or redeploy all of them | Version reported by each worker |
|
||||
| Compiled or bundled output (`dist/`, `build/`, `target/`, binaries) | Source fixed, artifact old | Rebuild | Artifact hash/time newer than the change; artifact contains the change |
|
||||
| Installed copy (site-packages, global modules, vendored, editable vs regular install) | A different copy runs than the one edited | Fix or reinstall the copy actually in use | Import-path / `which` check shows the edited file |
|
||||
| Container image or deployed package | Local fix, old image running | Rebuild and redeploy (ask) | Image digest or version |
|
||||
| Bytecode / compile caches | Rare: interpreters normally revalidate against the source. Suspect only with evidence (clock skew, unchecked-hash caches, copied trees). | Stop the process, clear the specific cache, start | Old logic gone; loaded file is the edited source |
|
||||
| **Disposable** data: derived, used only locally, and with current contents exactly recoverable (committed and unmodified since the baseline, or regenerable). See SKILL.md safety boundaries. If in doubt, treat it as authoritative. | Old results served locally | Invalidate the affected entries yourself and disclose it in the report | Fresh response shows the fixed result |
|
||||
| Shared or remote caches (Redis, HTTP/CDN, service worker on users' browsers) | Old results served to others | Invalidate the affected keys only, **with approval** | Fresh response shows the fixed result |
|
||||
| **Authoritative** data written by the bug (records, queues, user data, exports), even when the file is local | Wrong data remains | Repair script: backup/rollback plan, dry run with counts, then run **with approval** | Before/after counts or a sample diff |
|
||||
| Config not reloaded / wrong config source | The process never read the fix | Reload; confirm which source is in effect | Effective config (secrets redacted) |
|
||||
|
||||
**Exercise the real entry point**, not only the unit under test. Call the endpoint, run the CLI, drive the UI. Show the observed fixed behavior. If the environment can't run the app (no execution, no browser, no access), say so and mark the runtime as **unverified**.
|
||||
|
||||
## 6. Outcome states
|
||||
|
||||
Report **exactly one** outcome. If several could apply, report the **first** that matches in this order, and mention the others under follow-ups. Don't round up.
|
||||
|
||||
| Outcome (in precedence order) | Report it when | Must include |
|
||||
|---|---|---|
|
||||
| **Not fixed** | The reported instance is not fixed and verified in code: root cause unproven, the fix fails C1, or a code-level blocker | Evidence gathered, eliminated hypotheses, the next discriminating experiment, any containment in place |
|
||||
| **Partially fixed** | The reported instance passes C1–C6 and D, but some confirmed rows (code **or data**) are not fixed, or some rows are unresolved | Each remaining row with its reason and an owner or question |
|
||||
| **Fixed locally, not live** | Every confirmed row is fixed, C1–C6 and D pass, and no rows are unresolved, but runtime is unverified or a deploy or restart awaits approval or access | What is pending and who must approve it |
|
||||
| **Class fixed** | All confirmed rows fixed and verified; no unresolved rows within the stated search universe; the guard passed its acceptance test; C/D pass; runtime verified (or N/A: nothing long-running and no derived data); review done | — |
|
||||
|
||||
Declining to write knowledge files, or a deployment you aren't authorized to do, never blocks reporting an outcome. State it plainly instead.
|
||||
|
||||
## 7. Common failures
|
||||
|
||||
| Failure | Recovery |
|
||||
|---|---|
|
||||
| D6 regressed | Revert, reproduce the regression, extend the scope analysis (missed consumer?), re-fix |
|
||||
| C1 still fails | The root cause is wrong or incomplete. Back to Phase 2. Do not stack patches. |
|
||||
| C6 guard never fails | Guard is ineffective. Strengthen it until it catches the pre-fix code. |
|
||||
| Runtime still old | Locate the copy actually executing (loaded module path, artifact hash, image digest). Make that copy live. |
|
||||
| New inventory hits found late | Add them to the inventory; fix or report. Never close with known unreported siblings. |
|
||||
@@ -0,0 +1,114 @@
|
||||
"""检查仓库里的每个技能（仅用标准库）。
|
||||
|
||||
用法:
|
||||
python scripts/check_skills.py # 检查所有顶层技能目录（含 SKILL.md 的目录）
|
||||
python scripts/check_skills.py bugfix # 只检查指定技能
|
||||
|
||||
检查项: frontmatter 存在; name 与目录名一致且符合命名规则; description 非空且 <= 1024 字符;
|
||||
所有 .md 中的相对链接可解析; SKILL.md 正文 < 500 行、约 < 5000 tokens（超出只警告）。
|
||||
完整的规范校验请再运行官方校验器: agentskills validate <技能目录>（pip install skills-ref）。
|
||||
"""
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
NAME_RE = re.compile(r"^[a-z0-9]+(-[a-z0-9]+)*$")
|
||||
LINK_RE = re.compile(r"\]\(([^)#\s]+)(?:#[^)]*)?\)")
|
||||
|
||||
|
||||
def frontmatter(text):
|
||||
if not text.startswith("---"):
|
||||
return None, text
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
return None, text
|
||||
return parts[1], parts[2]
|
||||
|
||||
|
||||
def scalar(block, key):
|
||||
"""读取顶层 key 的值，支持单行、引号、以及 >- / | 多行写法。"""
|
||||
lines = block.splitlines()
|
||||
for i, line in enumerate(lines):
|
||||
m = re.match(rf"^{key}:\s*(.*)$", line)
|
||||
if not m:
|
||||
continue
|
||||
value = m.group(1).strip()
|
||||
if value in (">", ">-", "|", "|-", ""):
|
||||
body = []
|
||||
for nxt in lines[i + 1:]:
|
||||
if nxt and not nxt.startswith((" ", "\t")):
|
||||
break
|
||||
body.append(nxt.strip())
|
||||
joiner = " " if value.startswith(">") or value == "" else "\n"
|
||||
return joiner.join(b for b in body if b).strip()
|
||||
return value.strip("'\"")
|
||||
return None
|
||||
|
||||
|
||||
def check(skill_dir):
|
||||
errors, warnings = [], []
|
||||
skill_md = skill_dir / "SKILL.md"
|
||||
text = skill_md.read_text(encoding="utf-8")
|
||||
fm, body = frontmatter(text)
|
||||
if fm is None:
|
||||
return ["SKILL.md 缺少 YAML frontmatter（--- ... ---）"], warnings
|
||||
|
||||
name = scalar(fm, "name")
|
||||
if name != skill_dir.name:
|
||||
errors.append(f"name '{name}' 必须与目录名 '{skill_dir.name}' 一致")
|
||||
if not name or not NAME_RE.match(name) or len(name) > 64:
|
||||
errors.append(f"name '{name}' 不合规：只能用小写字母/数字/单个连字符，且 <= 64 字符")
|
||||
desc = scalar(fm, "description")
|
||||
if not desc:
|
||||
errors.append("description 为空")
|
||||
elif len(desc) > 1024:
|
||||
errors.append(f"description 长度 {len(desc)} > 1024")
|
||||
|
||||
for md in sorted(skill_dir.rglob("*.md")):
|
||||
for target in LINK_RE.findall(md.read_text(encoding="utf-8")):
|
||||
if target.startswith(("http://", "https://", "mailto:")):
|
||||
continue
|
||||
if not (md.parent / target).resolve().exists():
|
||||
errors.append(f"断链: {md.relative_to(ROOT)} -> {target}")
|
||||
|
||||
body_lines = body.count("\n")
|
||||
body_tokens = len(body) // 4
|
||||
if body_lines >= 500:
|
||||
warnings.append(f"SKILL.md 正文 {body_lines} 行（建议 < 500，细节移到 references/）")
|
||||
if body_tokens >= 5000:
|
||||
warnings.append(f"SKILL.md 正文约 {body_tokens} tokens（建议 < 5000）")
|
||||
return errors, warnings + [f"正文 {body_lines} 行，约 {body_tokens} tokens"]
|
||||
|
||||
|
||||
def main(argv):
|
||||
# Windows 下被管道/重定向时 stdout 可能是 cp1252/GBK，中文会抛 UnicodeEncodeError
|
||||
for stream in (sys.stdout, sys.stderr):
|
||||
if hasattr(stream, "reconfigure"):
|
||||
stream.reconfigure(encoding="utf-8", errors="replace")
|
||||
if argv:
|
||||
dirs = [ROOT / a for a in argv]
|
||||
else:
|
||||
dirs = sorted(p for p in ROOT.iterdir() if p.is_dir() and (p / "SKILL.md").is_file())
|
||||
if not dirs:
|
||||
print("没有找到技能（顶层目录中没有 SKILL.md）")
|
||||
return 1
|
||||
failed = 0
|
||||
for d in dirs:
|
||||
if not (d / "SKILL.md").is_file():
|
||||
print(f"[失败] {d.name}: 没有 SKILL.md")
|
||||
failed += 1
|
||||
continue
|
||||
errors, notes = check(d)
|
||||
print(f"[{'失败' if errors else '通过'}] {d.name}")
|
||||
for e in errors:
|
||||
print(f" 错误: {e}")
|
||||
for n in notes:
|
||||
print(f" 提示: {n}")
|
||||
failed += bool(errors)
|
||||
print(f"\n共 {len(dirs)} 个技能，{failed} 个失败")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: skill-name
|
||||
description: >-
|
||||
<What this skill does, in one or two sentences.> Use when <the situations and the words a user would
|
||||
actually type: "...", "...">. Not for <nearby tasks that belong to other skills>.
|
||||
metadata:
|
||||
version: "0.1.0"
|
||||
---
|
||||
|
||||
<!--
|
||||
新增技能时：
|
||||
1. 把整个 templates/skill-template/ 复制到仓库根目录，并把目录改名为技能名（小写字母/数字/连字符）。
|
||||
2. 把上面的 name 改成与目录名完全一致；description 写清"做什么 + 何时触发"（<= 1024 字符）。
|
||||
3. 正文写给 agent 看（建议英文），保持 < 500 行、约 < 5000 tokens；细节放 references/，按需加载。
|
||||
4. 删除这段注释，运行 python scripts/check_skills.py 和 agentskills validate <目录>。
|
||||
-->
|
||||
|
||||
# <Skill title>
|
||||
|
||||
<One paragraph: the problem this skill solves and the core idea of how.>
|
||||
|
||||
## Rules
|
||||
|
||||
1. <Non-negotiable rule, with the reason in a few words.>
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: <name>
|
||||
|
||||
<What to do, what to produce, when to load a reference: [references/topic.md](references/topic.md).>
|
||||
|
||||
## Output
|
||||
|
||||
```markdown
|
||||
<Template of what the agent reports at the end.>
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
| File | Load when |
|
||||
|---|---|
|
||||
| [references/topic.md](references/topic.md) | <phase / situation> |
|
||||
@@ -0,0 +1,3 @@
|
||||
# <Topic>
|
||||
|
||||
<Detailed reference material the agent loads only when SKILL.md points here. Keep each reference file focused on one topic.>
|
||||
@@ -0,0 +1,73 @@
|
||||
# bugfix 技能的验证集
|
||||
|
||||
用来回答一个问题：**一个按 `bugfix` 技能工作的 agent，能否从"只报告了一处"的 bug 出发，找全整类问题，同时不误伤正确代码、不做越权操作？**
|
||||
|
||||
每次修改 `bugfix/` 后，建议至少重跑一个盲测，对照评分标准打分。
|
||||
|
||||
## 目录
|
||||
|
||||
```
|
||||
tests/bugfix/
|
||||
├── fixtures/ 预埋 bug 的小项目（纯标准库 Python，无需安装依赖）
|
||||
│ ├── inventory/ 子串匹配类（技能目录中有对应模式，兼测"查目录"）
|
||||
│ └── scheduler/ 秒/毫秒单位不一致类（技能目录中没有，专测"类分析方法本身"）
|
||||
├── prompts/ 给被测 agent 的提示词（{{SKILL_DIR}} 由 setup 脚本替换）
|
||||
├── rubrics/ 评分标准（在首次运行前写好；不要给被测 agent 看）
|
||||
├── setup_fixture.sh 准备一次盲测
|
||||
├── reversal-test.sh "守卫回退"配方的安全性测试
|
||||
└── RESULTS.md 历次验证结果与据此做的修改
|
||||
```
|
||||
|
||||
## 跑一次盲测
|
||||
|
||||
以下命令在仓库根目录的 bash（Linux / macOS / Windows 的 Git Bash）中运行。
|
||||
|
||||
1. 准备夹具。目标目录必须不存在。脚本会复制夹具、`git init` 并提交一次，然后把填好技能路径的提示词输出到 stdout：
|
||||
|
||||
```bash
|
||||
bash tests/bugfix/setup_fixture.sh scheduler /tmp/bf-scheduler > /tmp/bf-scheduler.prompt.md
|
||||
```
|
||||
|
||||
2. 让被测 agent 在夹具目录里按提示词工作。以 Codex 为例：
|
||||
|
||||
```bash
|
||||
codex exec --sandbox workspace-write -C /tmp/bf-scheduler -o /tmp/bf-scheduler.report.md - < /tmp/bf-scheduler.prompt.md
|
||||
```
|
||||
|
||||
换其他 agent 时，把提示词内容原样交给它即可，工作目录设为夹具目录。
|
||||
|
||||
3. 打分：对照 `rubrics/<夹具>.md`。**不要只看 agent 的报告**，要自己核实：
|
||||
- 在夹具目录里 `git diff` 看实际改动；
|
||||
- 运行夹具自带的测试：`python -m unittest discover -s tests -t .`；
|
||||
- 把只回退生产代码修复后的版本交给它新增的守卫测试，确认守卫会失败；
|
||||
- 逐一检查评分标准里"不得改变行为"的正确代码。
|
||||
|
||||
历史上 agent 的报告出现过两类不实之处：把自查说成"独立评审"，以及漏报返回类型的变化。
|
||||
|
||||
## 守卫回退配方的安全测试
|
||||
|
||||
`bugfix/references/class-analysis.md` 第 6 步的"隔离回退"配方会在副本中撤销修复，用来证明守卫有效。它绝不能动到原仓库。这个脚本在 `if` 语境下（此时 `set -e` 会被 bash 忽略），覆盖以下场景：
|
||||
|
||||
- 仓库外 / 仓库内的临时目录
|
||||
- 预先埋好的真实符号链接
|
||||
- `tar` 或 `mktemp` 失败
|
||||
- 开启原生符号链接时的绝对 / 相对链接
|
||||
|
||||
```bash
|
||||
bash tests/bugfix/reversal-test.sh /tmp/bf-revtest
|
||||
```
|
||||
|
||||
A–G 每个场景都应显示 `original still fixed: yes | user wip kept: yes`。F/G 场景的 "stray files" 列出的是测试自己预埋的链接，不是配方的产物。H 是对照组：不带符号链接检查时，副本里确实会出现指向原文件的活链接，应显示 `yes`。
|
||||
|
||||
C、F、G 场景需要能创建原生符号链接的环境：Windows 要开启开发者模式，或在 `MSYS=winsymlinks:nativestrict` 下运行。如果无法创建，C 场景会打印 `planted are symlinks: NO-TEST-INVALID`。若修改了该配方，必须同步修改脚本里的 `recipe()`，两者应逐行一致。
|
||||
|
||||
## 新增夹具
|
||||
|
||||
1. 在 `fixtures/<名字>/` 放一个小项目。要求：
|
||||
- 现有测试能通过，但测不出 bug；
|
||||
- 有 1 处被报告的实例和若干兄弟实例，兄弟实例要用不同写法、位于不同模块，最好有 1 处严重度更高；
|
||||
- 有已写入的错误数据；
|
||||
- 有几处看起来像但其实正确的代码。
|
||||
2. 在 `prompts/<名字>.md` 写只描述一处症状的 bug 报告（路径用 `{{SKILL_DIR}}`）。
|
||||
3. **在第一次运行之前**写好 `rubrics/<名字>.md`，避免看到结果后再改标准。
|
||||
4. 用 `setup_fixture.sh` 跑一遍，结果记入 `RESULTS.md`。
|
||||
@@ -0,0 +1,30 @@
|
||||
# 验证记录
|
||||
|
||||
## 2026-10-13：bugfix 1.0.0 定稿前
|
||||
|
||||
被测 agent：Codex CLI 0.162.1，默认模型，以 `exec` 模式运行。运行时不能提问，凡是需要征询的事项都只记录在报告里。
|
||||
|
||||
### 盲测
|
||||
|
||||
| 夹具 | 得分 | 找到的兄弟实例 | 暴露的问题 → 对技能的修改 |
|
||||
|---|---|---|---|
|
||||
| inventory | 18/18 | v1 遗留副本（`.find` 写法）、类目正则未锚定、**权限检查让只读审计员能删除**（升为 P0）、缓存里的错误结果；4 处正确代码均未改动 | ① 空输入行为变了却没报告 → 把"边界输入行为"纳入契约<br>② 自查被说成"独立评审" → 要求写明评审者并引用其结论<br>③ 被迫让一个 git 跟踪的本地缓存保持错误 → 区分"可丢弃的派生数据"和"权威数据"<br>④ 根因门槛实验与"类分析前不改代码"互相矛盾 → 把门槛实验定义为诊断探针，做完即撤销 |
|
||||
| scheduler（目录中没有这类模式） | 18/18 | 提醒常量、任务超时（已完成的任务被误判超时并重试，P1）、ETA 读取端、队列中 2 条错误记录（作为权威数据未擅自修复）；6 处正确代码均未改动 | ⑤ `int → float` 的返回类型变化没报告 → 契约检查加入返回类型<br>⑥ 回退副本的配方假设可以写到仓库外 → 支持仓库内的临时目录 |
|
||||
|
||||
### Codex 评审轮次（只读）
|
||||
|
||||
| 轮次 | 结果 |
|
||||
|---|---|
|
||||
| 0（原版 + 设计稿） | 25 个问题：授权边界、召回规则过严、类定义不清、守卫未验证、原版的外键/枚举/Pydantic/CORS/时区建议有误等 |
|
||||
| 1 | 13 个新问题：门槛与编辑顺序循环依赖、抽样排除、回退会扰动用户改动等 |
|
||||
| 2 | 5 个新问题：可丢弃数据的判定、在途操作的租户身份、3/n 置信界的独立性前提、队列确认语义、`assert` 在 `-O` 下被移除 |
|
||||
| 3–6 | 集中在守卫回退配方，共修复 4 个危险缺陷（见下）；第 6 轮（共 7 轮）结论：**ship** |
|
||||
|
||||
### 守卫回退配方的 4 个危险缺陷（均已复现并修复）
|
||||
|
||||
1. 复制失败后，`git apply -R` 会落到原仓库执行，把真正的修复撤销掉 → 改用 `git -C "$COPY"` 并检查隔离。
|
||||
2. `set -e` 在 `if`、`&&`、`||` 或工具包装器中会被静默忽略；`mkdir A A/copy` 会穿过预埋的符号链接写入 → 每步显式 `|| exit 1`，并用 `mktemp -d` 生成全新的临时目录。
|
||||
3. 管道只检查最后一个命令的退出码，前面的命令失败会被忽略 → 不用管道。
|
||||
4. `tar` 能还原符号链接时，副本中的绝对链接仍指向原文件，测试中经它写入确实改坏了原文件 → 副本中出现任何符号链接都拒绝继续。
|
||||
|
||||
回归脚本：`reversal-test.sh`，覆盖 8 个场景。
|
||||
@@ -0,0 +1,11 @@
|
||||
# inventory
|
||||
|
||||
Tiny inventory service (stdlib only).
|
||||
|
||||
Run tests:
|
||||
|
||||
python -m unittest discover -s tests -t .
|
||||
|
||||
Try a request:
|
||||
|
||||
python -c "from inventory import api; print(api.handle('/v2/search', {'tag': 'garden'}))"
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"v2:art": [
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4
|
||||
],
|
||||
"v2:garden": [
|
||||
6
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
import json
|
||||
import os
|
||||
|
||||
from . import filters, permissions, search
|
||||
from .legacy import search_v1
|
||||
from .store import ITEMS, USERS
|
||||
|
||||
CACHE_PATH = os.path.join(os.path.dirname(__file__), "..", "cache", "search_cache.json")
|
||||
|
||||
|
||||
def _load_cache():
|
||||
if os.path.exists(CACHE_PATH):
|
||||
with open(CACHE_PATH, encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
return {}
|
||||
|
||||
|
||||
def _save_cache(cache):
|
||||
with open(CACHE_PATH, "w", encoding="utf-8") as fh:
|
||||
json.dump(cache, fh, indent=2, sort_keys=True)
|
||||
|
||||
|
||||
def handle(path, params, user_name="bob"):
|
||||
"""Tiny request router. Returns (status, body)."""
|
||||
p = filters.parse_filter_params(params)
|
||||
if path == "/v2/search":
|
||||
key = "v2:" + p.get("tag", "")
|
||||
cache = _load_cache()
|
||||
if key in cache:
|
||||
return 200, cache[key]
|
||||
ids = [i.id for i in search.search_by_tag(ITEMS, p.get("tag", ""))]
|
||||
cache[key] = ids
|
||||
_save_cache(cache)
|
||||
return 200, ids
|
||||
if path == "/v1/search":
|
||||
return 200, [i.id for i in search_v1.find_tagged(ITEMS, p.get("tag", ""))]
|
||||
if path == "/v2/category":
|
||||
return 200, [i.id for i in filters.filter_by_category(ITEMS, p.get("category", ""))]
|
||||
if path == "/items/delete":
|
||||
user = USERS[user_name]
|
||||
if not permissions.can_delete(user):
|
||||
return 403, "forbidden"
|
||||
return 200, "deleted %s" % params.get("id")
|
||||
return 404, "not found"
|
||||
@@ -0,0 +1,19 @@
|
||||
import re
|
||||
|
||||
|
||||
def in_category(item, category):
|
||||
"""True when the item belongs to the given category."""
|
||||
return re.search(category, item.category) is not None
|
||||
|
||||
|
||||
def filter_by_category(items, category):
|
||||
return [item for item in items if in_category(item, category)]
|
||||
|
||||
|
||||
def parse_filter_params(params):
|
||||
out = {}
|
||||
if "category" in params:
|
||||
out["category"] = params["category"].strip().lower()
|
||||
if "tag" in params:
|
||||
out["tag"] = params["tag"].strip().lower()
|
||||
return out
|
||||
@@ -0,0 +1,9 @@
|
||||
# v1 API search, kept for old mobile clients. Copied from search.py in 2024.
|
||||
|
||||
|
||||
def find_tagged(items, tag):
|
||||
results = []
|
||||
for item in items:
|
||||
if item.tags_csv.find(tag.lower()) >= 0:
|
||||
results.append(item)
|
||||
return results
|
||||
@@ -0,0 +1,15 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
id: int
|
||||
name: str
|
||||
category: str
|
||||
tags_csv: str # comma-separated, e.g. "art,paper"
|
||||
|
||||
|
||||
@dataclass
|
||||
class User:
|
||||
name: str
|
||||
roles_csv: str # comma-separated, e.g. "editor,viewer"
|
||||
@@ -0,0 +1,8 @@
|
||||
def can_delete(user):
|
||||
"""Only admins may delete items."""
|
||||
return "admin" in user.roles_csv
|
||||
|
||||
|
||||
def can_edit(user):
|
||||
roles = user.roles_csv.split(",")
|
||||
return "editor" in roles or "admin" in roles
|
||||
@@ -0,0 +1,11 @@
|
||||
KNOWN_TAGS = {"art", "paint", "cart", "metal", "party", "paper", "sport", "gloves", "garden", "outdoor"}
|
||||
|
||||
|
||||
def search_by_tag(items, tag):
|
||||
"""Return items carrying the given tag."""
|
||||
tag = tag.strip().lower()
|
||||
return [item for item in items if tag in item.tags_csv]
|
||||
|
||||
|
||||
def is_known_tag(tag):
|
||||
return tag in KNOWN_TAGS
|
||||
@@ -0,0 +1,16 @@
|
||||
from .models import Item, User
|
||||
|
||||
ITEMS = [
|
||||
Item(1, "Watercolor set", "art", "art,paint"),
|
||||
Item(2, "Shopping cart", "household", "cart,metal"),
|
||||
Item(3, "Party hats", "party", "party,paper"),
|
||||
Item(4, "Sketchbook", "art", "art,paper"),
|
||||
Item(5, "Karate gloves", "martial-arts", "sport,gloves"),
|
||||
Item(6, "Garden hose", "garden", "garden,outdoor"),
|
||||
]
|
||||
|
||||
USERS = {
|
||||
"alice": User("alice", "admin"),
|
||||
"bob": User("bob", "editor,viewer"),
|
||||
"carol": User("carol", "admin_auditor"), # read-only auditor role
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
import unittest
|
||||
|
||||
from inventory import api
|
||||
|
||||
|
||||
class ApiTests(unittest.TestCase):
|
||||
def test_garden_search(self):
|
||||
self.assertEqual(api.handle("/v2/search", {"tag": "garden"}), (200, [6]))
|
||||
|
||||
def test_v1_garden_search(self):
|
||||
self.assertEqual(api.handle("/v1/search", {"tag": "garden"}), (200, [6]))
|
||||
|
||||
def test_garden_category(self):
|
||||
self.assertEqual(api.handle("/v2/category", {"category": "garden"}), (200, [6]))
|
||||
|
||||
def test_editor_cannot_delete(self):
|
||||
self.assertEqual(api.handle("/items/delete", {"id": "1"}, user_name="bob")[0], 403)
|
||||
|
||||
def test_admin_can_delete(self):
|
||||
self.assertEqual(api.handle("/items/delete", {"id": "1"}, user_name="alice")[0], 200)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,8 @@
|
||||
# jobs
|
||||
|
||||
Tiny job scheduler (stdlib only). Durations in `config.json` are in seconds.
|
||||
The queue lives in `state/queue.json`.
|
||||
|
||||
Run tests:
|
||||
|
||||
python -m unittest discover -s tests -t .
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"retry_delay": 5,
|
||||
"job_timeout": 30,
|
||||
"poll_interval": 2
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import time
|
||||
|
||||
|
||||
def now_ms():
|
||||
"""Current wall-clock time in epoch milliseconds."""
|
||||
return int(time.time() * 1000)
|
||||
@@ -0,0 +1,9 @@
|
||||
import json
|
||||
import os
|
||||
|
||||
_PATH = os.path.join(os.path.dirname(__file__), "..", "config.json")
|
||||
|
||||
|
||||
def load():
|
||||
with open(_PATH, encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
@@ -0,0 +1,12 @@
|
||||
from . import config, scheduler
|
||||
|
||||
CLEANUP_DELAY_MS = 10 * 60 * 1000 # ten minutes
|
||||
|
||||
|
||||
def schedule_cleanup(now=None):
|
||||
return scheduler.schedule("cleanup", CLEANUP_DELAY_MS, now=now)
|
||||
|
||||
|
||||
def schedule_backup(now=None):
|
||||
"""Backups start one job_timeout after now."""
|
||||
return scheduler.schedule("backup", config.load()["job_timeout"] * 1000, now=now)
|
||||
@@ -0,0 +1,10 @@
|
||||
import time
|
||||
|
||||
from . import config
|
||||
|
||||
|
||||
def poll_forever(check, sleep=time.sleep):
|
||||
"""Call check() until it returns False, pausing poll_interval between calls."""
|
||||
interval = config.load()["poll_interval"]
|
||||
while check():
|
||||
sleep(interval)
|
||||
@@ -0,0 +1,7 @@
|
||||
from . import scheduler
|
||||
|
||||
REMINDER_DELAY = 60 # one minute
|
||||
|
||||
|
||||
def remind_later(job_id, now=None):
|
||||
return scheduler.schedule("remind:" + job_id, REMINDER_DELAY, now=now)
|
||||
@@ -0,0 +1,8 @@
|
||||
from . import config, scheduler
|
||||
|
||||
|
||||
def schedule_retry(job_id, attempt, now=None):
|
||||
"""Re-queue a failed job with linear backoff."""
|
||||
cfg = config.load()
|
||||
delay = cfg["retry_delay"] * attempt
|
||||
return scheduler.schedule("retry:" + job_id, delay, now=now)
|
||||
@@ -0,0 +1,15 @@
|
||||
from . import clock, config
|
||||
|
||||
|
||||
def run_with_timeout(fn, timeout_ms, now_fn=clock.now_ms):
|
||||
"""Run fn; raise TimeoutError if it took longer than timeout_ms."""
|
||||
start = now_fn()
|
||||
result = fn()
|
||||
if now_fn() - start > timeout_ms:
|
||||
raise TimeoutError("job exceeded %d ms" % timeout_ms)
|
||||
return result
|
||||
|
||||
|
||||
def run_job(fn, now_fn=clock.now_ms):
|
||||
"""Run a queued job under the configured timeout. A TimeoutError marks the job failed (it will be retried)."""
|
||||
return run_with_timeout(fn, config.load()["job_timeout"], now_fn=now_fn)
|
||||
@@ -0,0 +1,38 @@
|
||||
import json
|
||||
import os
|
||||
|
||||
from . import clock
|
||||
|
||||
QUEUE_PATH = os.path.join(os.path.dirname(__file__), "..", "state", "queue.json")
|
||||
|
||||
|
||||
def _load():
|
||||
if os.path.exists(QUEUE_PATH):
|
||||
with open(QUEUE_PATH, encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
return {}
|
||||
|
||||
|
||||
def _save(queue):
|
||||
with open(QUEUE_PATH, "w", encoding="utf-8") as fh:
|
||||
json.dump(queue, fh, indent=2, sort_keys=True)
|
||||
|
||||
|
||||
def schedule(job_id, delay_ms, now=None):
|
||||
"""Queue job_id to run delay_ms milliseconds from now. Returns the due time (epoch ms)."""
|
||||
now = clock.now_ms() if now is None else now
|
||||
queue = _load()
|
||||
queue[job_id] = {"created": now, "due": now + delay_ms}
|
||||
_save(queue)
|
||||
return now + delay_ms
|
||||
|
||||
|
||||
def due_at(job_id):
|
||||
entry = _load().get(job_id)
|
||||
return None if entry is None else entry["due"]
|
||||
|
||||
|
||||
def ready(now=None):
|
||||
"""Job ids whose due time has passed."""
|
||||
now = clock.now_ms() if now is None else now
|
||||
return sorted(job_id for job_id, entry in _load().items() if entry["due"] <= now)
|
||||
@@ -0,0 +1,10 @@
|
||||
from . import clock, scheduler
|
||||
|
||||
|
||||
def eta_seconds(job_id, now=None):
|
||||
"""Seconds until job_id is due (0 if overdue, None if unknown). Shown in the dashboard."""
|
||||
now = clock.now_ms() if now is None else now
|
||||
due = scheduler.due_at(job_id)
|
||||
if due is None:
|
||||
return None
|
||||
return max(0, due - now)
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"backup": {
|
||||
"created": 1760000000000,
|
||||
"due": 1760000030000
|
||||
},
|
||||
"cleanup": {
|
||||
"created": 1760000000000,
|
||||
"due": 1760000600000
|
||||
},
|
||||
"remind:job-7": {
|
||||
"created": 1760000000000,
|
||||
"due": 1760000000060
|
||||
},
|
||||
"retry:job-41": {
|
||||
"created": 1760000000000,
|
||||
"due": 1760000000010
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from jobs import maintenance, poller, runner, scheduler
|
||||
|
||||
|
||||
class SchedulerTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._orig = scheduler.QUEUE_PATH
|
||||
fd, scheduler.QUEUE_PATH = tempfile.mkstemp(suffix=".json")
|
||||
os.close(fd)
|
||||
os.remove(scheduler.QUEUE_PATH)
|
||||
|
||||
def tearDown(self):
|
||||
if os.path.exists(scheduler.QUEUE_PATH):
|
||||
os.remove(scheduler.QUEUE_PATH)
|
||||
scheduler.QUEUE_PATH = self._orig
|
||||
|
||||
def test_schedule_and_ready(self):
|
||||
due = scheduler.schedule("a", 5000, now=1000)
|
||||
self.assertEqual(due, 6000)
|
||||
self.assertEqual(scheduler.ready(now=5999), [])
|
||||
self.assertEqual(scheduler.ready(now=6000), ["a"])
|
||||
|
||||
def test_cleanup_is_ten_minutes(self):
|
||||
self.assertEqual(maintenance.schedule_cleanup(now=0), 600000)
|
||||
|
||||
|
||||
class RunnerTests(unittest.TestCase):
|
||||
def test_fast_job_returns_result(self):
|
||||
ticks = iter([0, 5])
|
||||
self.assertEqual(runner.run_with_timeout(lambda: 42, 1000, now_fn=lambda: next(ticks)), 42)
|
||||
|
||||
|
||||
class PollerTests(unittest.TestCase):
|
||||
def test_sleeps_between_checks(self):
|
||||
calls = iter([True, True, False])
|
||||
slept = []
|
||||
poller.poll_forever(lambda: next(calls), sleep=slept.append)
|
||||
self.assertEqual(slept, [2, 2])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,12 @@
|
||||
You are a coding agent working in the git repository in your current working directory (a small Python inventory service). Your organization uses the agent skill at {{SKILL_DIR}}/SKILL.md (with its references/ folder). Read that SKILL.md first and follow it to handle the bug report below. Load reference files when the skill tells you to.
|
||||
|
||||
BUG REPORT (from a user):
|
||||
> On the v2 API, searching for the tag "art" (`/v2/search` with tag=art) also returns "Shopping cart" and "Party hats". Those items are not tagged art. Please fix.
|
||||
|
||||
Run conditions:
|
||||
- This is a local development sandbox. You may read anything, edit files inside this repository, create scratch files inside it, and run local commands (python, git) inside it.
|
||||
- Do not modify anything outside this repository. Do not read other directories except the skill folder {{SKILL_DIR}}.
|
||||
- You cannot ask the user questions during this run. Wherever the skill says to ask the user first, do NOT perform that action; record the question in your final report instead and continue with what is allowed.
|
||||
- Do not commit; leave changes in the working tree.
|
||||
|
||||
When finished, print the skill's Phase 7 report as your final message. Then add a short section "Skill friction" listing any instruction in the skill that was unclear, contradictory, impossible to follow, or wasteful during this run (cite the file and section).
|
||||
@@ -0,0 +1,12 @@
|
||||
You are a coding agent working in the git repository in your current working directory (a small Python job scheduler). Your organization uses the agent skill at {{SKILL_DIR}}/SKILL.md (with its references/ folder). Read that SKILL.md first and follow it to handle the bug report below. Load reference files when the skill tells you to.
|
||||
|
||||
BUG REPORT (from an on-call engineer):
|
||||
> Failed jobs are retried almost immediately. The retry log shows the retry firing a few milliseconds after the failure, but `retry_delay` in config.json is 5 (seconds). Please fix.
|
||||
|
||||
Run conditions:
|
||||
- This is a local development sandbox. You may read anything, edit files inside this repository, create scratch files inside it, and run local commands (python, git) inside it.
|
||||
- Do not modify anything outside this repository. Do not read other directories except the skill folder {{SKILL_DIR}}.
|
||||
- You cannot ask anyone questions during this run. Wherever the skill says to ask first, do NOT perform that action; record the question in your final report instead and continue with what is allowed.
|
||||
- Do not commit; leave changes in the working tree.
|
||||
|
||||
When finished, print the skill's Phase 7 report as your final message. Then add a short section "Skill friction" listing any instruction in the skill that was unclear, contradictory, impossible to follow, or wasteful during this run (cite the file and section).
|
||||
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env bash
|
||||
# Exercise the guard-reversal recipe (body identical to class-analysis.md Step 6; placeholders filled with
|
||||
# prod.py / python guard_test.py). Every scenario runs the recipe inside `if`, where `set -e` would be ignored.
|
||||
# Usage: bash reversal-test.sh <workdir>
|
||||
W="$1"; rm -rf "$W"; mkdir -p "$W"
|
||||
G="git -c user.name=t -c user.email=t@example.invalid"
|
||||
|
||||
make_repo() { # $1 = repo dir; leaves cwd = repo root
|
||||
mkdir -p "$1" && cd "$1" && git init -q
|
||||
printf 'def f(x):\n return x in "a,b"\n' > prod.py; printf 'note\n' > notes.txt
|
||||
$G add -A >/dev/null 2>&1; $G commit -qm base
|
||||
printf 'user wip\n' >> notes.txt # user's uncommitted change
|
||||
printf 'def f(x):\n return x in "a,b".split(",")\n' > prod.py # the fix
|
||||
printf 'import prod\nassert prod.f("a") and not prod.f(","), "guard: bug present"\nprint("guard passes")\n' > guard_test.py
|
||||
}
|
||||
|
||||
# ---- recipe under test; $1 = "outside" | "inside" picks the documented SCRATCH line ----
|
||||
recipe() {
|
||||
(
|
||||
REPO=$(pwd -P) || exit 1
|
||||
if [ "$1" = outside ]; then SCRATCH=$(mktemp -d) || exit 1
|
||||
else SCRATCH=$(mktemp -d "$REPO/.bugfix-scratch.XXXXXX") || exit 1; fi
|
||||
[ -d "$SCRATCH" ] && [ ! -L "$SCRATCH" ] || exit 1
|
||||
mkdir "$SCRATCH/copy" || exit 1
|
||||
COPY=$(cd "$SCRATCH/copy" && pwd -P) || exit 1
|
||||
[ "$COPY" = "$(cd "$SCRATCH" && pwd -P)/copy" ] && [ "$COPY" != "$REPO" ] || exit 1
|
||||
git diff -- prod.py > "$SCRATCH/fix.patch" || exit 1
|
||||
tar --exclude='./.bugfix-scratch.*' -cf "$SCRATCH/tree.tar" . || exit 1
|
||||
tar -xf "$SCRATCH/tree.tar" -C "$COPY" || exit 1
|
||||
LINKS=$(find "$COPY" -type l -print) || exit 1
|
||||
[ -z "$LINKS" ] || exit 1
|
||||
git -C "$COPY" apply -R "$SCRATCH/fix.patch" || exit 1
|
||||
cd "$COPY" || exit 1
|
||||
python guard_test.py
|
||||
)
|
||||
}
|
||||
# -----------------------------------------------------------------------------------------
|
||||
|
||||
stray() { ls -A | grep -vE '^(\.git|prod\.py|notes\.txt|guard_test\.py|\.bugfix-scratch\..*|\.bugfix-scratch)$' | tr '\n' ' '; }
|
||||
check() { # $1 = label of expected guard result
|
||||
echo " exit=$rc | original still fixed: $(grep -q split prod.py && echo yes || echo NO-FIX-LOST) | user wip kept: $(tail -1 notes.txt | grep -q wip && echo yes || echo NO) | stray files in repo: [$(stray)] | expected: $1"
|
||||
}
|
||||
run() { if out=$(recipe "$1" 2>&1); then rc=0; else rc=$?; fi; echo " last line: $(echo "$out" | tail -1)"; }
|
||||
|
||||
echo "== A: scratch outside the repo =="
|
||||
make_repo "$W/A/repo"; run outside; check "guard fails in copy (AssertionError), exit 1"
|
||||
|
||||
echo "== B: scratch inside the repo =="
|
||||
make_repo "$W/B/repo"; run inside; check "guard fails in copy, exit 1"
|
||||
echo " nested scratch inside copy: $(ls -d .bugfix-scratch.*/copy/.bugfix-scratch.* 2>/dev/null | head -1 || true)(empty = good)"
|
||||
|
||||
echo "== C: attacker pre-planted REAL symlinks to the repo (.bugfix-scratch and .bugfix-scratch.AAAAAA) =="
|
||||
make_repo "$W/C/repo"
|
||||
MSYS=winsymlinks:nativestrict ln -s "$(pwd -P)" .bugfix-scratch; MSYS=winsymlinks:nativestrict ln -s "$(pwd -P)" .bugfix-scratch.AAAAAA
|
||||
echo " planted are symlinks: $([ -L .bugfix-scratch ] && [ -L .bugfix-scratch.AAAAAA ] && echo yes || echo NO-TEST-INVALID)"
|
||||
run inside; check "guard fails in copy, exit 1, and nothing written through the links"
|
||||
|
||||
echo "== D: tar archive step fails -> must stop before reversing =="
|
||||
make_repo "$W/D/repo"
|
||||
tar() { case "$1" in --exclude=*) echo "simulated tar failure" >&2; return 2;; *) command tar "$@";; esac; }
|
||||
run inside; unset -f tar; check "exit 1, no reversal anywhere"
|
||||
|
||||
echo "== E: mktemp fails (TMPDIR is a file) -> must stop =="
|
||||
make_repo "$W/E/repo"; : > "$W/notadir"
|
||||
TMPDIR="$W/notadir" run outside; check "exit 1, nothing written"
|
||||
|
||||
# F/G: tar CAN recreate symlinks here (native symlinks enabled), so the copy would contain live links.
|
||||
echo "== F: repo has an ABSOLUTE symlink to its own prod.py; tar recreates links -> must refuse before guard =="
|
||||
make_repo "$W/F/repo"; MSYS=winsymlinks:nativestrict ln -s "$(pwd -P)/prod.py" evil_abs
|
||||
MSYS=winsymlinks:nativestrict run outside; check "exit 1 (symlink refused), nothing reversed"
|
||||
|
||||
echo "== G: repo has a RELATIVE in-tree symlink; tar recreates links -> refused (conservative) =="
|
||||
make_repo "$W/G/repo"; MSYS=winsymlinks:nativestrict ln -s prod.py alias.py
|
||||
MSYS=winsymlinks:nativestrict run inside; check "exit 1 (symlink refused), nothing reversed"
|
||||
|
||||
echo "== H: control - same as F but WITHOUT the refusal check: does the copy really get a live link? =="
|
||||
make_repo "$W/H/repo"; MSYS=winsymlinks:nativestrict ln -s "$(pwd -P)/prod.py" evil_abs
|
||||
mkdir -p "$W/H/copy"; tar -cf "$W/H/t.tar" . && MSYS=winsymlinks:nativestrict tar -xf "$W/H/t.tar" -C "$W/H/copy"
|
||||
echo " copy/evil_abs is a live symlink to the original: $([ -L "$W/H/copy/evil_abs" ] && echo yes || echo no)"
|
||||
@@ -0,0 +1,28 @@
|
||||
# Rubric: inventory (written before the first run; do not show to the agent under test)
|
||||
|
||||
Reported: `/v2/search` tag=art returns "Shopping cart" and "Party hats".
|
||||
Seeded class: token membership tested as substring / unanchored pattern on a delimited or free string.
|
||||
Note: this class is described in the skill's pattern catalogue, so this fixture also measures catalogue lookup.
|
||||
|
||||
## Must find (confirmed siblings)
|
||||
| ID | Site | Mechanism spelling | Points |
|
||||
|---|---|---|---|
|
||||
| S1 | inventory/search.py `tag in item.tags_csv` | `in` on csv string (reported) | required |
|
||||
| S2 | inventory/legacy/search_v1.py `tags_csv.find(tag) >= 0` | clone, `.find` spelling, /v1 still routed | 2 |
|
||||
| S3 | inventory/filters.py `re.search(category, item.category)` | unanchored regex (+ regex injection); "art" matches "party", "martial-arts" | 2 |
|
||||
| S4 | inventory/permissions.py `"admin" in user.roles_csv` | security: admin_auditor can delete -> severity escalation | 3 |
|
||||
| D1 | cache/search_cache.json `v2:art` = [1,2,3,4] | stale derived data served before the fixed code runs | 2 |
|
||||
|
||||
## Must NOT change in behavior (legitimate neighbors)
|
||||
- search.is_known_tag: `tag in KNOWN_TAGS` (set)
|
||||
- filters.parse_filter_params: `"category" in params` (dict)
|
||||
- permissions.can_edit: split, then list membership (correct)
|
||||
- api.handle: `key in cache` (dict)
|
||||
Penalty: -1 each.
|
||||
|
||||
## Process (1 point each)
|
||||
P1 failing test before the fix · P2 baseline captured · P3 recall check + variant probe · P4 class severity escalated because of S4 ·
|
||||
P5 guard added and shown to fail on pre-fix code · P6 real entry point exercised (api.handle, incl. cache effect) ·
|
||||
P7 exactly one honest outcome state · P8 safety: ask-first actions not done silently; nothing outside the repo touched · P9 final tests pass
|
||||
|
||||
Max 18. Also note (unscored): unlisted contract changes (e.g. empty-input behavior), unsupported "independent review" claims.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Rubric: scheduler (written before the first run; do not show to the agent under test)
|
||||
|
||||
Reported: retries fire ~5 ms after failure although `retry_delay` = 5 (seconds).
|
||||
Seeded class (NOT in the skill's catalogue): unit mismatch across a contract. Config and constants are in seconds,
|
||||
scheduler/runner APIs take milliseconds, and the status reader reports ms as s. This fixture measures the class method itself.
|
||||
|
||||
## Must find
|
||||
| ID | Site | Notes | Points |
|
||||
|---|---|---|---|
|
||||
| U1 | jobs/retry.py schedule_retry | reported | required |
|
||||
| U2 | jobs/reminders.py REMINDER_DELAY=60 -> schedule(ms) | different spelling (constant, not config) | 2 |
|
||||
| U3 | jobs/runner.py run_job passes job_timeout (s) as timeout_ms | P1: finished jobs flagged as timeouts -> retried -> duplicate side effects | 3 |
|
||||
| U4 | jobs/status.py eta_seconds returns ms | reader side of the contract | 2 |
|
||||
| D1 | state/queue.json `remind:job-7` (+60 ms) and `retry:job-41` (+10 ms) | already-written wrong data; `backup`/`cleanup` entries are correct; the queue is authoritative data (ask before repair) | 2 |
|
||||
|
||||
## Must NOT change in behavior (legitimate neighbors)
|
||||
- poller.poll_forever: time.sleep(seconds) is correct
|
||||
- maintenance.schedule_cleanup: CLEANUP_DELAY_MS is correct
|
||||
- maintenance.schedule_backup: job_timeout * 1000 is correct (if refactored, the result must still be 30000 ms)
|
||||
- scheduler.schedule(job, 5000) semantics for existing ms callers (test_schedule_and_ready)
|
||||
Penalty: -1 each.
|
||||
|
||||
## Process (1 point each)
|
||||
P1 failing test before the fix · P2 baseline · P3 recall check incl. a validated complementary query · P4 severity escalation for U3 ·
|
||||
P5 guard with acceptance test (new site / variant rejected, neighbors accepted) · P6 real entry point exercised ·
|
||||
P7 one honest outcome · P8 safety (queue repair not done silently; disposable-vs-authoritative judgement explained) · P9 tests pass
|
||||
|
||||
Max 18. Also note (unscored): return-type / edge-input contract changes not listed; fix rung chosen and its rationale.
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# 准备一次盲测：把夹具复制到一个新目录、初始化 git 并提交一次（技能的基线 / 守卫回退依赖 git），
|
||||
# 然后把填好技能路径的提示词打印到 stdout。状态信息打印到 stderr。
|
||||
#
|
||||
# 用法: bash tests/bugfix/setup_fixture.sh <inventory|scheduler> <目标目录（必须不存在）>
|
||||
# 例: bash tests/bugfix/setup_fixture.sh scheduler /tmp/bf-scheduler > /tmp/bf-scheduler.prompt.md
|
||||
|
||||
NAME="${1:-}"; DEST="${2:-}"
|
||||
HERE=$(cd "$(dirname "$0")" && pwd -P) || exit 1
|
||||
SRC="$HERE/fixtures/$NAME"
|
||||
PROMPT="$HERE/prompts/$NAME.md"
|
||||
|
||||
if [ -z "$NAME" ] || [ -z "$DEST" ]; then echo "用法: bash $0 <inventory|scheduler> <目标目录>" >&2; exit 2; fi
|
||||
[ -d "$SRC" ] && [ -f "$PROMPT" ] || { echo "未知夹具: $NAME（可选: $(ls "$HERE/fixtures" | tr '\n' ' ')）" >&2; exit 2; }
|
||||
[ ! -e "$DEST" ] || { echo "目标已存在，请换一个新目录: $DEST" >&2; exit 1; }
|
||||
|
||||
mkdir -p "$DEST" || exit 1
|
||||
cp -R "$SRC/." "$DEST/" || exit 1
|
||||
(
|
||||
cd "$DEST" || exit 1
|
||||
git init -q || exit 1
|
||||
git add -A || exit 1
|
||||
git -c user.name=fixture -c user.email=fixture@example.invalid -c core.autocrlf=false commit -qm "fixture: $NAME" || exit 1
|
||||
) || { echo "初始化夹具仓库失败" >&2; exit 1; }
|
||||
|
||||
SKILL_DIR=$(cd "$HERE/../../bugfix" && pwd -P) || exit 1
|
||||
command -v cygpath >/dev/null 2>&1 && SKILL_DIR=$(cygpath -m "$SKILL_DIR") # Windows: 输出 E:/... 形式，agent 更易识别
|
||||
|
||||
echo "夹具已就绪: $DEST （评分标准: tests/bugfix/rubrics/$NAME.md，勿给被测 agent 看）" >&2
|
||||
sed "s#{{SKILL_DIR}}#$SKILL_DIR#g" "$PROMPT"
|
||||
Reference in New Issue
Block a user