Esc
↑↓ to move↵ to openEsc to close

DocsArchitecture

Repo miner

How the training-data miner works: architecture, every stage, every column, every flag, and why.

Complete description of pipelines/ingestion/repo_miner/: what it is, why it is built the way it is, how a run works step by step, every module, every output column, every file it writes, every flag and every tunable.

  • Written against miner version 7.0.0 (version.py). Contract versions: mined_dataset 1.3, mined_snapshot 1.4, mined_commit_features 1.1, mined_features 1.0.
  • Where this document and the code disagree, the code is right. The short operator guide is README.md in this directory; this file is the long form, and does not replace it.
  • Column lists were checked against contracts/mined_dataset.schema.json, mined_snapshot.schema.json and mined_commit_features.schema.json (section 14). The contracts, not this file, are the source of truth for names and types.
  • British spelling, as in the ADRs.

Contents

  1. What the miner is and why it exists
  2. Design principles and the reason for each
  3. Architecture
  4. How a run works, step by step
  5. Feature families: how each is computed and why
  6. Labels (targets)
  7. The LLM subsystem
  8. The GitHub API subsystem
  9. The git subsystem
  10. SZZ fix tracing
  11. PR snapshots
  12. Scaling, resuming and repairing
  13. Observability and operator tools
  14. Schema reference: every column
  15. Files the miner reads and writes
  16. Command-line reference
  17. Environment variables
  18. Tunables (Settings)
  19. Tests
  20. Known limitations and sharp edges
  21. Version history
  22. Glossary

1. What the miner is and why it exists

1.1 What it is

The repo miner turns a cloned GitHub repository plus the GitHub API into a training table: one row per closed pull request, with

  • every feature the Riffle scorer would have seen at the moment the PR opened, and
  • the outcomes that happened afterwards (did it merge, was it reworked, was it reverted, did CI break, was it hotfixed, why was it closed).

It also produces two derived tables for training on PRs while they are open (snapshots and per-head-commit features).

It is a Python package of plain scripts (no installable package). git and the standard library are enough for the core; everything else (lizard, semgrep, pandas/pyarrow, an LLM) is an optional stage that degrades to null columns.

1.2 Where it sits in Riffle

Riffle is a self-hosted GitHub App that ranks pull requests by learned risk. Training data has to come from somewhere. The decision (ADR 0009 note, pipelines/ingestion/README.md) is that the miner's own rows are the primary training data; other datasets (On the Shoulders of Giants, ApacheJIT) are secondary and AIDev is not ingested because the miner produces the same labels from scratch.

text
repos file ──► build_dataset.py ──► mine_repo.py (one process per repo)
                                        │
                                        ├─► dataset.parquet            one row per closed PR
                                        ├─► snapshots_raw/<repo>/      raw capture (clones are deleted)
                                        └─► ci_cache/, health/, logs/  caches, checks, traces
                                  (batch end)
                                        ├─► dataset_snapshots.parquet        one row per PR snapshot
                                        └─► dataset_commit_features.parquet  one row per head commit
                                  (several PCs)  merge_shards.py ──► *_merged.* tables
                                  (training)     pipelines/training/frames.py joins the three tables

1.3 Two targets from one run (ADR 0002 and ADR 0009)

TargetQuestionRows it trains onKnown at
label_bad (ADR 0002)Did a merged PR break: revert OR failed required CI OR hotfix?merged PRs only7 to 90 days after merge
label_merged (ADR 0009)Will the PR merge at all, and if not, why?every closed PRat close

They are never combined into one label. Rationale (ADR 0009): a rejected PR is not a broken change, and the two have different maturity. A combined risk, if wanted, is P(merge) × P(bad | merged).

Further derived targets sit on every row (outcome_3class, nowhere_reason) so every consumer trains on the same definitions (proposal Part 4).

1.4 Why a miner at all, and why scratch-mined

  • The scorer scores a PR when it opens. A model has to learn from PRs whose future is known, so every historical PR must be reconstructed as it was when it opened and paired with how it ended.
  • Third-party datasets do not carry the GitHub timeline signals (reviews, force-pushes, duplicate marks, close reasons) the merge model needs, nor the point-in-time discipline of ADR 0010. Mining from scratch gives one consistent definition of every feature and label, and lets mining and serving share extractor code later.

2. Design principles and the reason for each

PrincipleWhat it means in codeWhy
Point-in-time, no leakageDiff features come from the PR as it stood at created_at + 300 s (ADR 0010). History/blame walk only commits before the base commit. Repo-wide aggregates (merge rates, similar PRs, scrutiny) count a past PR only if it closed before this PR opened, and an outcome only if it happened before then.The scorer sees only the past. A feature that peeks (final size after rework, today's PR title, labels added after a failure) makes offline scores overstate production.
Features at open, labels from what landedmine_pr uses two ranges: the at-open range for features and the final range (what actually merged) for post-merge labels.Rework after open is part of what the targets measure, so it must not leak into the inputs; revert/hotfix/CI/SZZ are about what landed.
Deterministic firstRegex, parsers and linters are used wherever an exact answer exists (secrets, off-by-one-ish edits, migrations, manifests). The LLM only answers semantic questions.Cheap, reproducible, auditable; the LLM is never the verdict and is never on the scoring path.
Missing is null, never 0A tool that is absent or failed, a git call that timed out, a network call that failed or was cut short, too little history: the column is null (XGBoost/LightGBM treat it as missing). 0 means "measured, found none".A silent 0 is a lie the model will learn. Matches the contract rule null = not declared/not enough history.
Never write a row without its LLM answerA failed LLM call is retried per PR (3×, waits 60/120/180 s). If it still fails the repo stops (exit code 3) and --out is not written. Rows out of LLM scope keep null LLM columns on purpose.Half-populated LLM columns would be a hidden bias (failure correlates with diff content).
Every row has every columnmined_dataset.schema.json lists the columns; a stage that is off writes null, not "missing".One schema for every run mode. tests/test_dataset_contract.py enforces it both ways.
Resumable, crash-safeRows stream to <out>.partial.jsonl; PR selection is pinned; SZZ blames are stored per (fix commit, file); LLM answers are reused on re-mine.A big repo takes hours; stops (ctrl+c, quota, crash, reboot) must lose nothing.
Output identical regardless of parallelismPRs mine in worker threads but are consumed in PR order; a failed LLM answer is retried in PR order, so which PR stops a repo never depends on which worker finished first.Reproducible datasets and a stop rule that does not depend on timing.
Graceful degradationlizard/semgrep/oasdiff/buf/graphql-inspector/network each degrade to null. A repo with no token still mines (with fewer columns).Different machines have different toolchains; a missing tool must not abort an overnight run.
ObservableRun log, structured trace, progress events, health check, TUI, web view.A run takes hours or days; a stall must be explainable without guessing.
Contract-boundAdding a column means adding it to the contract in the same PR. Version bumps: MAJOR = output columns/meaning change (retrain-level), MINOR = new features/flags with unchanged existing output, PATCH = fixes that leave output identical.Training, serving and the feature store must agree on what a field means.

3. Architecture

3.1 Process and thread model

DiagramDiagram
Mermaid source
flowchart TD
  tui["tui.py (optional)<br/>Textual UI + web view (dashboard.py)"] -->|"subprocess, RIFFLE_EVENTS file"| bd
  cli["command line"] --> bd
  bd["build_dataset.py<br/>batch driver: repos file, shards, clone, retries"] -->|"subprocess per repo<br/>--append --out ..."| mr
  mr["mine_repo.py<br/>one repo"] --> pool["ThreadPoolExecutor<br/>(workers) mine_pr per PR"]
  mr --> szz["labels.build_szz_index<br/>ProcessPoolExecutor (blame workers)"]
  pool --> llm["claude -p subprocess<br/>or OpenAI-compatible API<br/>(llm_concurrency semaphore)"]
  pool --> gh["github_api.py (urllib)<br/>REST + GraphQL"]
  pool --> git["git subprocesses (gitio.py)"]
  mr --> out[("dataset.parquet<br/>+ partial.jsonl checkpoint")]
  mr --> raw[("snapshots_raw/owner_name/")]
  bd -->|"batch end"| asm["snapshots.py assemble_files"]
  asm --> st[("*_snapshots.parquet<br/>*_commit_features.parquet")]
  • build_dataset.py runs each repo as a separate mine_repo.py subprocess. One process per repo bounds memory (a Kubernetes-sized history is large) and means a crash or Stop only affects that repo; the checkpoint makes a retry resume.
  • mine_repo.py mines PRs in a thread pool (work is mostly git subprocesses, so threads are fine). SZZ blame uses a process pool because it is CPU/blame heavy.
  • LLM calls are bounded by a global semaphore (llm_concurrency, default 4) across all workers.
  • Global state (per process = per repo): the git numstat cache, blame LRU cache (64 entries), lizard complexity caches, LLM usage counters, quota gate, the run trace.

3.2 Module map (every file)

FileRole
version.py__version__ (semver rules in the docstring). Written to every row as miner_version.
config.py.env auto-loader, Settings dataclass (all tunables), path rules (PATH_RULES, test/doc/generated patterns), language map, word-boundary fix/revert/hotfix text matchers, subsystem_of, RAM-aware default worker count.
gitio.pyAll git access: numstat history cache, blame, diffs, merge-base, commit parents, ref fetching, landed-commit index, first-parent chains.
github_api.pyREST/GraphQL client (stdlib urllib): PR lists, window selection, timelines, review events, force-pushes (GraphQL), check conclusions, required checks, CI history map, user account age. Rate-limit aware.
mine_repo.pyPer-repo orchestrator and CLI. mine_pr (one PR → one row), selection pinning, at-open snapshot resolution, concurrency index, writer.
build_dataset.pyBatch driver: repos file parsing, domain tags, sharding, clone, retries, skips, audit/re-mine, snapshot assembly, shard manifest.
aggregate.pyCollapses per-file dicts into fixed max_/mean_/sum_/min_/median_ columns (AGG_SPEC).
labels.pyBreakage labels (label_revert, label_ci_fail, label_hotfix, label_szz_bug, label_bad, observability columns) and the SZZ index builder.
merge_labels.pyMerge-outcome labels (ADR 0009), close-reason rules and LLM classifier prompt, at-open metadata rebuilt from the timeline, outcome_3class/nowhere_reason.
flaky.pyFlaky-check detection; corrects label_ci_fail/label_bad after a repo is mined.
settings_schema.pyThe lead's shared settings as data (riffle-data/server/static/settings-schema.json), built from the parser and join.SHAPING, for the admin form; a test keeps the committed copy equal.
maintainers.py"Who is a maintainer" evidence from the repo's own timelines (point-in-time).
scope.pyWhich PRs are "training PRs" (outside non-bot contributors plus a stable 5% sample).
features/diff_stats.pyKamei size/diffusion metrics, diff shape, path flags.
features/history.pySource 2: per-file and per-author history index at the base commit; hotspots, ownership, co-change, experience, percentiles.
features/line_history.pySource 3: blame of the exact lines the PR modifies.
features/scanners.pySource 4d: lizard complexity delta, semgrep new findings.
features/deps_cve.pySource 4e: dependency changes, OSV vulnerabilities, GHSA cross-check.
features/deterministic.pySource 5: regex/parser checks on added lines.
features/api_contract.pySource 5: oasdiff / buf / graphql-inspector verdict on spec files.
features/llm_flags.pySource 6: 23 LLM flags, backends, diff slimming, secret redaction, quota gate, usage counters, reuse store. There is no circuit breaker in the module: the per-PR retry in mine_repo is what stops a run.
features/merge_signals.pyAIDev-catalog merge-model signals: body structure, agent detection, repo-level prior-only rates, text similarity.
features/repo_local.pyThe nine repo-local columns of mined_features.schema.json.
snapshot_capture.pyCapture step of PR snapshots (runs during mining).
setup_cache.pyPer-repo cache of the author index, concurrency counts and SZZ relevant set (ci_cache/<owner>_<name>.setup.json), keyed on the PR list, HEAD and merge style (4.3).
snapshots.pyAssembly step of PR snapshots (offline, from captures).
consistency.pyDoes the snapshot table still match its dataset?
merge_shards.pyJoins the tables of several PCs, refusing mismatched pieces (including a manifest that lists failed repos, unless --allow-incomplete).
naming.pyOutput/log naming for sharded runs, shard manifest path, snapshot table names.
health.pyPer-repo health summary and flags.
sync.pySends a PC's logs, health, datasets and shard manifest to the team store (ADR 0015); incremental, bounded, never fatal.
runlog.pyStructured run trace (*.trace.jsonl), stage/step/span timing, heartbeat, stall detection.
trace_report.pySummarises a trace file.
progress.pyProgress events (RIFFLE_EVENTS) for the TUI.
tui.pyTerminal UI (Textual) driving build_dataset.py.
dashboard.pyRead-only LAN web view of the TUI, "big picture" mode.
doctor.pyEnvironment checks and fixes (run on every TUI launch).
deps.pyInstalls Python dependencies; manages the private .venv on PEP 668 systems.
scout.pyMetadata-only ranking of candidate repos (no clone).
llm_compare.pyCompares two LLM models on the same diffs, per flag agreement.
audit_export.pyStratified CSV for the human label audit.
repos.txt, repos_web_ts.txt, repos_vite.txtRepository lists (see 15.1).
.env.example, requirements.txtConfig template and optional dependencies.
tests/See section 19.

3.3 Data flow of one row

DiagramDiagram
Mermaid source
flowchart LR
  sel["PR selection<br/>(API list / git)"] --> snap["at-open snapshot<br/>(base, head, kind)"]
  snap --> diff["diff_stats / deterministic /<br/>scanners / deps / api_contract"]
  snap --> hist["history index at base<br/>+ blame of modified lines"]
  hist --> agg["aggregate.py<br/>max/mean/sum/min"]
  tl["PR timeline"] --> meta["at-open title, milestone, labels"]
  tl --> ml["merge-outcome labels"]
  fin["final range (what landed)"] --> lab["revert / hotfix / CI / SZZ labels"]
  diff --> row(("row"))
  agg --> row
  meta --> row
  ml --> row
  lab --> row
  row --> llm["LLM flags (resolved in PR order)"]
  llm --> ckpt[("checkpoint")]
  ckpt --> repo["repo-wide pass:<br/>flaky, repo-local, merge signals"]
  repo --> final[("dataset row")]

4. How a run works, step by step

4.1 The batch: build_dataset.py

python build_dataset.py reads a repos file and mines every repo into one growing dataset, de-duplicated by (repo, pr_number).

  1. Output naming. With --shard K/N the output is tagged (dataset.parquet → dataset_pc1of3.parquet, run log run-<time>-pc1of3.log). --out-prefix / --out-postfix add text (placeholders {pc} {n} {host} {date}; a postfix replaces the automatic tag); --no-auto-name turns the tag off. A name that already carries its tag is left alone, so a rerun resumes the same file. If an untagged file exists but is not used, the run says so, because resume only looks at the tagged file. Why: several PCs writing the same dataset.parquet get mixed up when copied together.
  2. Trace. runlog.configure points RIFFLE_TRACE at logs/<run-stem>.trace.jsonl (inherited by every child miner).
  3. Audit-only and shard-plan-only modes print and exit (see 12). Before anything is cloned, validate_options(args) runs once in main and refuses a bad combination: --full-repo with a window, cap or lead-in, --window-from without --window-to or a bad date, a negative or out-of-range number, a malformed --shard (--shard-plan below 1 exits too), a bad --remine-repo name. --max-prs 0 means no cap, on both sides of the pipe.
  4. Read the repos file. Each line is a GitHub URL; text after # is a comment but carries two tags: [~N commits] (approximate commit count, used as the repo's weight) and [domain: NAME] (ADR 0011). --only-domain NAME keeps matching repos only. A GitHub URL is parsed with github.com[:/]owner/name(.git), taking the first two path segments only. A repo name must match ^[A-Za-z0-9._-]+$ (no . or .., no leading -), so a line cannot point the clone outside --clone-dir; a repo listed twice (case-insensitive) is mined once, with a warning. A [domain: X] tag is case-insensitive and lowercased, and a malformed tag warns. --only-domain with no match says 0 repos tagged X.
  5. Shard. balanced_shards gives each PC an equal number of repos (±1) with balanced total weight: heaviest repo first, each to the lightest PC that still has room; unknown weights count as the median; deterministic, so every PC computes the same split. split_id is the first 8 hex of a SHA-256 over the ordered repo list, weights and PC count; PCs that print the same split ID have disjoint shares of the same list.
  6. Test mode (--test): first repo only (default pallets/flask), --test-prs PRs (10), fresh output test_output/test_dataset.jsonl (also saved as parquet to check that path), clone kept. It always mines the fixed window config.TEST_WINDOW_FROM..TEST_WINDOW_TO (2020-01-01 to 2025-06-30) with --min-age-days 0 and no --cap-on or --max-unmerged, whatever the form or flags say, so every PC and every run picks the same PRs whatever day it runs. --test-prs then caps merged and unmerged PRs together (the newest contiguous block, mode A), so the output holds about that many rows. Remaining sources of drift: a PR created in the window that was still open on the first run and closes later, and a stale reused clone (merge-landing detection reads it).
  7. Re-mine. --auto-remine (off by default) audits the output and adds flagged repos in this PC's share; each --remine-repo must be in this PC's share (else the run stops with nothing changed). _drop_repo keeps the repo's earlier LLM answers (ci_cache/<owner>_<name>.llm_reuse.json, keyed by base and head SHA), rewrites the dataset atomically without the repo, removes its SZZ cache, partial files, blame store and pinned selection, and removes its checkpoint lines. A clean repo write likewise removes only that repo's lines (case-insensitive, atomic) from the shared out.partial.jsonl, deleting the file only when it is empty.
  8. Skip finished repos (rows already in --out; --remine disables). A repo only lands in --out after a clean complete mine, so presence means done.
  9. For each repo:
    • --skip-big N skips repos whose [~N commits] tag is larger; the TUI can also request a skip through the file named in RIFFLE_CONTROL. A skip never writes rows, so a later run mines the repo.
    • Clone (full history, never --depth) into <clone-dir>/<owner>_<name> with git clone --progress -- https://github.com/{owner}/{name} <target>, retried (--repo-retries, default 2 extra) with waits of min(120, 15 × attempt) s. The target's real path must stay inside --clone-dir, else the repo is refused and reported as (unsafe path). A clone that prints nothing for --clone-stall-s (default 600, RIFFLE_CLONE_STALL_S, 0 = never) is killed and retried. An existing clone is removed unless --skip-existing.
    • Mine by running mine_repo.py as a subprocess with --append and the pass-through flags. A retry resumes from the checkpoint. Exit code 3 (LLM could not answer) aborts the whole batch without retry. The miner runs in its own session (start_new_session=True), and a Stop or skip signals the whole process group: SIGINT first so the miner keeps its checkpoint, then SIGKILL after 30 s, and KeyboardInterrupt is re-raised (a Stop during a skip stops the batch). Workers end at their next step, and a Stop also reaches git: gitio.STOP is checked before every git subprocess starts and raises GitStopped (not a GitError, so code that tolerates git failures cannot swallow it), and a row from a PR whose git calls were refused during a Stop is not written. Their children are killed with their process groups. If the batch process itself is SIGKILLed, the miner is not signalled automatically.
    • After a successful mine the clone is deleted (one clone on disk at a time) unless --keep-clones. A failed or stopped repo keeps its clone so a rerun resumes without re-cloning. ci_cache/, checkpoints and snapshots_raw/ are kept.
  10. Batch end: save test parquet (test mode); assemble snapshots (section 11; failure never fails the batch); write the shard manifest (<out>.shard.json) for sharded runs; emit batch_done; print skips and failures.
  11. Stop (ctrl+c): one [stop] line, child miner stopped with its checkpoint kept, exit 130.

The lead's pasted repo list is filtered line by line (join.clean_repos_text); apply_split raises if no line is valid.

4.2 One repo: mine_repo.py, stage by stage

The run log gets [stage] X done in 2m57s for each stage and the trace gets start/done events with durations.

#Stage (runlog name)What happensWhyAPI cost
0setupFind default branch (origin/HEAD, else main/master/HEAD). snapshot = now. obs_cutoff = newest commit time on the default branch in this clone, capped at now. Configure blame (-w -M, plus the repo's own .git-blame-ignore-revs if present). Build GitHubAPI unless --no-network.obs_cutoff says how far forward outcomes can be seen in this clone; reverts after it are not there. Blame flags and ignore-revs stop formatting sweeps being blamed as the author.none
1fetching PRsSelect PRs (4.3): merged, unmerged, stale-open, lead-in, block start. Resume loads the pinned selection instead.Defines which rows exist.1 call per 100 closed PRs, plus lead-in and open-PR pages
2required checksRead the default branch's required status checks from branch protection.Distinguishes gating CI from auxiliary checks precisely. 403/404 is normal for repos you don't own and falls back to a name denylist.1
3merge styleDecide whether the repo is merge-commit, squash, or rebase style (4.4).Picks the right diff base for 1-parent merges.1 (+1 per multi-commit PR in rebase repos)
4timelinesRead every PR's issue timeline up front (parallel, up to 8; prints [timeline] n/N read and PRs per hour every 1,000 PRs or 5 minutes, since a quota sleep looks like a hang), build maintainer evidence, set _author_maintainer on every PR. Pinned selection also pins the evidence.Merge labels, at-open metadata, lead-in signals and maintainer detection all need timelines; reading once avoids refetching.1+ per PR (100 events per page, 2000 cap); a big repo is bound by the 5,000 an hour REST quota (next.js, 34,555 PRs: 6h42m, with about 50 minutes of sleep each hour)
5author indexauthor_email → sorted times of that author's merged PRs, from one local git show per PR; kept in the setup cache (below), so a restart on the same PRs and history skips it.Point-in-time author_prs_merged_repo.none
6open snapshotsFetch heads missing from the clone (refs/pull/N/head), ask GraphQL for force-pushes (25 PRs per call), fetch replaced heads by SHA, resolve pr["_open"] (4.5). The per-PR snapshots run in slices of 2,000 and log progress. Git runs with gc.auto=0 and maintenance.auto=false, so a fetch of thousands of PR heads never starts an automatic repack of the clone.ADR 0010: features from the PR as it opened.1 GraphQL call per 25 PRs
7concurrencyFor each PR count other PRs open at its opening that touch the same files / same base-side lines; kept in the setup cache unless a diff could not be read.Integration-risk features; point-in-time by construction.none (git diffs in parallel)
8ci historyBuild `{commit: failuresuccess}from the Actions workflow-runs list (100 runs/call, cached toci_cache/<owner>_<name>.json`; the cache is versioned, expires after 7 days and is rebuilt when it was built before the newest PR, see 8.2).Historical CI-fail-rate features (file/author). Far cheaper than one check-runs call per commit.
9loading historyParse the whole repo's git log --all --numstat -M -z once (streamed) into an in-memory cache, and build the per-repo history index once, answered by bisect.Every history feature, SZZ and path_size_pctile reuse it; otherwise ~1.5 minutes would be silently spent inside the first PR on a Kubernetes-sized repo.none
10szzBuild or reuse the SZZ index (section 10). The set of commits it may skip blames for (relevant, every commit of every merged PR's range, needed for the cache check) comes from the setup cache.label_szz_bug, days_to_szz_fix.none
11mining PRsMine PRs in worker threads, consume in order, resolve LLM answers, write checkpoint and capture records.The main work.per PR, see 4.6
12repo-wide labelsflaky.apply, repo_local.compute_repo, merge_signals.compute_repo (4.7).These need every PR's outcome.none

Afterwards main() writes the dataset (_write), runs the health check, deletes the one-shot LLM reuse file, prints LLM usage, and only after a clean complete write removes the checkpoint and the pinned selection.

4.3 Choosing which PRs become rows

There are four selection modes (plus --full-repo, below). Drafts are always left out (never offered for review).

A. Date-window mode (--window-from/--window-to, set both; the recommended mode for dataset runs). GitHubAPI.window_prs:

  • Pages through closed PRs sorted by creation, newest first, 100 per page.
  • Keeps every closed PR created in the window, merged and unmerged together, so every repo covers the same span and keeps its real merged/unmerged mix.
  • A PR is dropped if its outcome (merge or close time) is younger than --min-age-days (default 90), so labels are mature (30-day revert window, 90-day SZZ window).
  • --max-prs N keeps the newest contiguous block of that many PRs by creation time. With --cap-on training only training PRs count toward N (outside non-bot contributors plus the 5% sample, scope.py), so --max-prs 10000 --cap-on training yields about 10,000 PRs to train on. Maintainer and bot PRs inside the block are still mined (history needs them) but do not count. Maintainers confirmed from history are known only after timelines are read, so the training count can come out slightly below the cap.
  • History lead-in: closed PRs created in the --lead-in-days (default 180) before the block are collected as metadata only (the PR list plus one timeline call each). They are never mined into rows; they give the block's first PRs their merge-signal history (repo_outside_merge_rate, first-response times, agent load, similar PRs). Breakage history (similar_pr_*, path_failure_mode_*) needs mined outcomes, so it still starts empty.
  • The line [window] ... reports merged, unmerged, training PRs, block start and lead-in.

--full-repo. Mode A with the window fixed to config.FULL_REPO_FROM..FULL_REPO_TO (2000-01-01 to 9999-12-31), max_prs unset and lead_in_days 0: every closed, non-draft PR the repo has, merged and unmerged, minus those younger than --min-age-days. The dates are fixed, not today's, so a resumed run's pinned selection still matches. Rejects --window-from/--window-to, --max-prs, --lead-in-days, --cap-on (and --test in build_dataset.py).

B. Legacy mode (no window). merged_prs: fetches closed PRs by creation order, collects merged ones, sorts by merged_at, keeps the newest --max-prs. Unmerged PRs are taken from the same span as the merged ones: closed between the oldest and the newest kept merge (from 6.11.0; before that only the lower creation bound applied, so a cap on merged PRs left the unmerged ones unbounded and skewed the merged share). Open PRs with no response are bounded by the newest kept merge as well. A small over-fetch covers created-vs-merged reordering at the boundary.

C. Landed-commit mode. Some repos never use GitHub's merge button (merge scripts, bots), so their PRs are only "closed". On the first page, if under half of closed PRs merged through GitHub and at least half of the closed-unmerged PRs link to a commit on the default branch, the miner switches to landed-commit mode: gitio.landed_commit_index maps PR numbers to commits on the default branch (recognised links: Pull Request resolved: <url> (PyTorch), PR-URL: <url> (Node.js), Closes/Resolves #N, subject ending (#N), Merge pull request #N). Such rows have merge_via = landed_commit. In this mode closed-unmerged PRs get no rows because "not merged" is ambiguous. Why both conditions: a low GitHub-merge share alone also describes a repo flooded with rejected AI PRs. Settings.merge_signal (auto|github|landed) forces a mode but has no CLI flag.

D. Git-primary discovery (--pr-discovery git). Enumerate merged PRs from the default branch's commit messages (same link patterns); the API only enriches each PR (fetch_pr: created_at, base ref, head, labels, association, draft, milestone, authoritative merged_at). No closed-PR pagination; works offline for discovery. PRs without a recognisable reference are missed; no unmerged rows. Used by the offline contract test.

Unmerged rows. The unmerged PRs' commits are fetched from refs/pull/N/head without creating local refs, so git log --all scans (history cache, revert/hotfix search) never see unmerged work as history or as revert candidates. The diff base is the merge-base of the base commit GitHub recorded (or the default branch when it is gone) and the head. PRs whose head cannot be fetched are dropped with a warning, not counted as gone (a row with null features for such a PR was considered and left for later, see 20). A git fetch that fails for any reason other than "ref missing" is retried and then raises FetchError, so a failed head is unobserved, not gone.

Stale-open rows. PRs still open after abandon_days (60) with no human non-author comment or review are rows too: unmerged, abandoned, closed_at null, close_reason = abandoned, close_reason_source = rules. Open PRs that did get a response have no outcome yet and are left out. Only open PRs created in the mined block's creation window are considered; not in landed-commit mode.

Pinned selection (v5.1.0). The first run of a repo writes ci_cache/<owner>_<name>.selection.json: the merged, unmerged, stale-open and lead-in PRs, lead-in sidecars, block start, landed flag, incomplete flag, and the maintainer evidence, plus the selection parameters (window_from/to, max_prs, min_age_days, lead_in_days, merge_outcome, max_unmerged, merge_signal, abandon_days, major version). A resume with the same parameters mines exactly the same PRs even if more PRs merged since, and skips the PR list, lead-in and timeline fetches. A run with different parameters makes a new selection. Removed after a clean write and by --remine-repo. Why: new merges would otherwise shift the selection, orphan the checkpoint's rows and change the SZZ relevant set.

Setup cache (v7.2.2). author index, concurrency and the SZZ relevant set each walk every PR through git (on n8n, 23k PRs: about 2,700 s, 1,100 s and, for relevant, most of a 79-minute szz stage that logged from cache). Before 7.2.2 a restart paid all three again, so an interrupted run spent about 2.5 h on setup before it mined a PR, even with every other cache warm. setup_cache.py keeps their results in ci_cache/<owner>_<name>.setup.json, one section each, stored under a key that hashes the file version, the section name, the clone's HEAD, whether the clone is shallow, the merge style the base resolver chose, and for every PR the fields these stages read (number, created, merged and closed times, merge, head and base SHAs, unmerged and stale-open flags, the at-open head and base). Any difference is a miss and the stage runs as before; a reused result equals a computed one. concurrency is kept only when every diff was read (stats["errors"] == 0), so a transient git failure is not frozen in. An unreadable or foreign file is a miss, a failed write a warning. The file is removed after a clean repo write and by --remine-repo. A reuse prints [setup-cache] <stage>: reused. Not cached: fetching PRs (500 to 650 s on n8n, not traced), loading history (an in-memory git log, about 95 s) and reviews prefetch (depends on which PRs remain).

Truncation. If a PR-list page fails the list is marked incomplete and a warning says label coverage is partial.

4.4 The diff range, merge styles and the at-open snapshot

There are two ranges per PR (ADR 0010).

Final range (_pr_diff_range): what landed, immune to a stale API base.sha (which would pull in unrelated commits that landed on main between open and merge).

Merge shapeBaseHead
2-parent merge commitmerge-base(p1, p2)p2 (equivalently git diff p1...p2)
1-parent squash / normalmerge commit's parentmerge commit
1-parent rebase mergehead~N (N = PR commit count)merge commit
landed-commit (git base known)parent of the oldest linked commitnewest linked commit
root commit (no parents)API base.shamerge
fast-forwarded PR whose head is itself a merge commitmerge-base(base, head)head
closed-unmergedmerge-base(base, head)PR head

For a fast-forwarded PR whose head is a merge commit, the author is taken from the first-parent chain, not from the merge itself. A commit's landing is found from landed_commit_index and the first hit that is a merge or carries (#N) wins; reverts are skipped, and the API's merge_commit_sha is preferred when there is one.

Merge style detection (_make_base_resolver, once per repo): one call to repo settings for allowed merge methods plus a local parent-count sniff over up to 50 PRs. If at least half are 2-parent merges the repo is merge-dominant. Per-PR commit-count calls are paid only when the repo is plausibly rebase-style (not merge-dominant, rebase allowed, API on). In repos allowing both squash and rebase, a cheap local author-overlap check (_looks_rebased: a majority of head~1..head~(n-1) share the PR author) confirms each multi-commit PR. Squash and merge-commit repos cost nothing extra. The printed [merge-style] line states the verdict.

At-open snapshot (_open_snapshot): the PR as it stood when it opened.

  • Head: the newest commit on the PR branch's first-parent line committed at or before created_at + rework_grace_s (300 s, the same grace label_rework uses). The walk stops at GitHub's recorded base.sha, so commits landed on the base branch with their original dates cannot be mistaken for the PR's.
  • Force-push after open: GraphQL HeadRefForcePushedEvent.beforeCommit is the head the push replaced (the REST timeline does not keep it). The walk starts from the beforeCommit of the first force-push after the grace period, fetched by SHA (GitHub still serves it).
  • Base: merge-base of that head and base.sha. History features use this base, never after open. base_behind_by = base-branch commits the fork point was behind at open.
  • open_snapshot says how it was found:
    • exact: no force-push after open (known from GraphQL),
    • recovered: walked from a replaced head,
    • approx: force-push history unreadable (no token, failed batch),
    • final: not recoverable; diff features use the final range and may leak. Training drops or masks diff features on final rows (frames.py nulls the diff features). A PR whose at-open diff turns out empty is downgraded to final.
  • Commit times are commit dates (push time is not exposed by the APIs): a commit made before open but pushed after it counts as part of the PR at open.

4.5 mine_pr: one PR becomes one row

Run inside a worker thread; every step is timed (runlog.step). Returns None (a "no-rows" skip) when there is no diff, a missing commit or a shallow clone.

  1. Ranges. Final range, then at-open range (pr["_open"]) else final with open_snapshot = final. If the at-open diff is empty, fall back to final.
  2. Identity. author_email is the mailmap-canonical git author (%aE) of the commit that is the PR's own head (_feat_head: p2 of a 2-parent merge, the merge commit for a squash, the PR head for unmerged), not the maintainer who merged. This makes EXP, REXP and every author feature about one person. Merge commits (more than one parent) never count as anyone's work: they are left out of the author totals, EXP, REXP, SEXP, author_last, n_commits_total, repo_revert_base_rate and the percentile populations. Their CI verdict still goes to the merger and to the files of the first-parent diff.
  3. Timeline (pr["_events"], or fetched if missing). Used for merge labels and to rebuild the title, milestone and labels as at open + grace (merge_labels.at_open). Why: maintainers retitle rejected PRs ("AI spam") and milestone accepted ones; in one pilot repo 210 of 312 titles had changed.
  4. diff_stats (Source 4a/4b) → size, spread, path flags.
  5. History index at the base commit (history.build_repo_index, restricted to the changed paths), then per file per_file_features + line_history.per_file_line_features.
  6. LT, la_norm, ld_norm, AGE; aggregate.aggregate_per_file → max_/mean_/... columns; cochange_features; test_expectation_features; author_features; repo_revert_base_rate; FIX; author_prs_merged_repo; bot/agent flags; EXP/REXP/SEXP; percentile features; base_behind_by; concurrency; hotspot membership.
  7. deterministic, scanners, deps_cve, api_contract on the at-open diff.
  8. Repo-local: path_size_pctile now; the sidecar for the repo-wide pass; review events (2 API calls per PR, unless --no-review-history).
  9. LLM fetch (llm_flags.fetch) in the worker; resolved later in PR order.
  10. PR metadata (_pr_metadata, with title/milestone from the at-open rebuild), merge signals (body features, agent name, branch prefix, account age, repo files at base), declared_hotfix.
  11. Breakage labels (labels.build_labels) on the final range, or all post-merge columns null for unmerged PRs (observed_through null, mined_at set).
  12. Merge-outcome labels (merge_labels.build_merge_labels), with one check-runs call on the head commit for non-stale unmerged PRs and a classifier call when applicable; or the 13 columns set to null with --no-merge-outcome.
  13. Provenance columns (miner_version, miner_commit, llm_model, llm_prompt_version, domain).
  14. Snapshot capture for rows in LLM scope (section 11).

Private keys carried on the row until written: __llm__ (the fetch result), __capture__, _repo_local, __merge_signals__, __ci__. They are popped before the dataset is written (the checkpoint keeps the sidecars so a resumed run can still do the repo-wide pass).

Big PRs. A PR touching more than --max-files-deep files (default 300, RIFFLE_MAX_FILES_DEEP, 0 = off) gets per-file features for the N files with the largest churn only; the rest are null, and so are LT, la_norm, ld_norm and every sum_* aggregate on that PR (max_, min_ and mean_ cover the measured files). The cap is part of the checkpoint stamp (4.8).

Failure handling per PR: a GitError is a git-error skip; any other exception is an exc:<Type> skip with a warning; LLMUnavailable stops the repo. Skips are counted and printed ([skips] N/M PRs produced no row).

4.6 The ordered consumer: parallel yet reproducible

  • --workers N (default min(6, CPUs) lowered to fit free RAM: (free RAM − 1.5 GB) // 0.8 GB; RIFFLE_WORKERS overrides; the miner says when it lowered it). One October 2026 run peaked at 5.5 GB with 6 workers on a large repo.
  • A window of max(2 × workers, 2 × llm_concurrency) futures is kept in flight; results are consumed in PR order.
  • For each result the consumer (main thread): prints [pr i/N] #num (Xs), handles skips, pops __llm__, retries a failed LLM answer in place (_retry_llm; later PRs queue behind it so the miner pauses), resolves the flags, then commits: capture record and index entry written, row appended, checkpoint line written and flushed.
  • Interrupts: ctrl+c shuts the pool down without waiting; finished PRs are in the checkpoint; SZZ blame workers ignore SIGINT so there are no per-worker tracebacks.

4.7 The repo-wide pass

Needs every PR's outcome, so it runs once all rows exist (and is skipped if the LLM stopped the repo):

  1. flaky.apply corrects label_ci_fail, ci_fail_flaky_only and label_bad (6.4).
  2. repo_local.compute_repo fills the PR-history columns (similar_pr_*, path_failure_mode_*, path_detection_lag_days, revealed_path_scrutiny, release_cycle_position).
  3. merge_signals.compute_repo fills the eight prior-only repo-level columns using the lead-in sidecars too.
  4. Sidecars are stripped; any missing merge-signal or repo-local column is set to null (similar_pr_count defaults to 0).

4.8 Writing and de-duplication

  • --out may be .parquet (default; falls back to .jsonl with a warning when pandas/pyarrow are missing, before the append merge so every repo appends to the same file), .jsonl or .csv.
  • --append merges into an existing file and de-duplicates by (repo, pr_number) keeping the newest extraction. If the existing file cannot be read, the run writes <out>.recover-<time>.jsonl and leaves --out unchanged (never clobbers the accumulated dataset).
  • Row order: merged PRs newest-merged first, then unmerged PRs.
  • Writes are atomic. The final dataset is written to a temporary file and moved with os.replace. A _write in append mode with no rows leaves the dataset untouched. If the parquet write fails, the fallback merges the existing parquet and the existing .jsonl alternative, or writes a recover file and keeps the checkpoint.
  • The checkpoint (<out>.partial.jsonl) is read tolerantly: torn, blank or non-object lines are skipped, the tail is healed before appending, and rows are de-duplicated by pr_number, the last winning. The batch's per-repo row count (_checkpointed, used by the store status) reads only each line's leading repo field, caches by file size and mtime plus a head and tail fingerprint, and parses only appended lines; a rewrite is detected by the fingerprint and re-read in full (a rewrite that changes only the middle of the file while size and mtime stay the same is not detected).
  • Checkpoint stamp. Each line carries _ckpt: whether the LLM was on and its model, CI history, SZZ, the observation cutoff day, and the flags that shape the data (scanners, api_contract, review_history, merge_outcome, declared_labels, deps_cve, github_api, pr_discovery, llm_scope, llm_sample_pct, close_reason_llm, llm_max_diff_chars, domain, max_files_per_pr_deep). The miner's version is not in it: a new miner never throws earlier work away, and an older row's miner key is ignored. On resume, a row whose stamp differs from the current run's is mined again, and the log names the keys that differ. Rows with no stamp (written by older versions) are accepted. _ckpt is stripped from the output. Rows written with --no-network carry repo = owner/name.
  • snapshots_raw/<repo>/capture.jsonl and pr_index.jsonl are trimmed atomically on resume and are idempotent per PR; a resume with --snapshots after a --no-snapshots run re-mines the in-scope PRs missing from the index.

4.9 After mining

  • Snapshot assembly (section 11) runs at batch end.
  • Shard manifest (dataset_pc1of3.shard.json): split ID, PC, N, miner version, host, finish time, repos, mined-OK count, failures, file names.
  • Admin (admin.py, ADR 0015): the lead's list, revoke NAME, revoke-split ID and whoami over the store's /v1/admin/* endpoints, with RIFFLE_ADMIN_TOKEN from .env.
  • Joining a split (join.py, ADR 0015): --shard-plan N --issue-tokens asks the store (admin token) for one write token per PC, bound to the plan's split ID and optionally expiring (--token-days), and prints a join code per PC (riffle1. plus base64 JSON: store URL, token, split, PC K of N, miner version, and the plan's flags without the plan-only ones). join.py CODE calls the store's /v1/whoami, writes RIFFLE_SYNC_URL and RIFFLE_SYNC_TOKEN to .env and prints build_dataset.py <flags> --shard K/N. At the start of a run _sync_begin re-checks the token and turns sync off, with the reason, when it is rejected or its split is not the split these flags compute. doctor.py has a Team store line.
  • Lead settings after joining. The lead changes a split's shared flags with the store's settings action (POST /v1/admin/split-settings). Tokens and join codes are not re-issued: each PC reads the current flags with its existing token (/v1/split) when a batch starts, and runs them, keeping its machine flags (shard, workers, clone dir). Each finished repo records the settings version it was mined under in <out>.settings.json. A repo mined under other settings, or with no record, is reported and its rows are kept; re-mine one with --remine-repo to use the current settings. A test run never reads the store.
  • Version check. Every dataset row carries miner_version. A repo whose rows have no version, or one older than 7.0.0, is stale. --audit lists stale repos with the blame check and adds them to its re-mine list. --auto-remine re-mines them and is off by default (--auto-remine turns it on; the TUI checkbox "Auto re-mine stale repos" is off by default and sends the matching flag). It is a lead setting: it is in SHAPING, so the lead's flags carry it to joined PCs, and a joined TUI reads the lead's current flags when it restarts. Three guards: a repo whose rows come from a newer miner than the one running is never re-mined there; a repo automatically re-mined under this miner version is recorded in <out>.auto_remined.json and is not re-mined again by that version, so a repo still flagged after its re-mine is reported, not looped; and a test run never re-mines. Every batch first prints which stale repos are in its share. The audit and these checks read six columns of --out once (_audit_frame), not the whole dataset several times.
  • Version report. _audit reads --out, and a repo is not in --out until it finishes, so it cannot see work in progress. _redo_report (now a notice, nothing is redone) reads the checkpoint (<out>.partial.jsonl: rows whose _ckpt stamp names another miner major.minor are kept), the first bytes of each ci_cache/<owner>_<name>.selection.json (a miner_major other than this miner's: the pinned PR list is kept) and the count of tl: entries in ci_cache/<owner>_<name>.api.sqlite. Every batch prints a [version] block for the repos it will mine; --audit prints it for every checkpoint and selection. It only reports. A repo is mined again only by --remine-repo (the TUI's "Re-mine repos" box).
  • Central sync (sync.py, ADR 0015), when RIFFLE_SYNC_URL and RIFFLE_SYNC_TOKEN are set: after each repo, on a Stop and at batch end, build_dataset.py PUTs the run log and trace, health/*.json, the dataset, its snapshot and commit-feature tables, the .partial.jsonl checkpoint and the shard manifest under <miner_version>/<split_id>/, then a run.json with the counts and reason. The final batch done is sent only when every file was sent or skipped; otherwise the reason is batch incomplete, with files_failed and a new files_deferred list, and the store rewrites such a run to batch incomplete (a run only counts as finished when its status says so). run.json carries each repo's state (pending, mining, done, failed, skipped), its rows (a done repo's from the dataset, an unfinished one's from its own lines in the shared checkpoint, which also holds the rows a failed or stopped repo keeps for the rerun) and, once selected, its PR count, so the store can show done and remaining per PC; remaining covers started repos only, since an unstarted repo has no selection yet. While a repo mines, a status-only run.json goes out at once when the run starts and when each repo starts, and then every 3 minutes while a repo mines, so the store shows a PC within seconds of Start and its progress within a few minutes. Unchanged files (size and mtime in .sync_state.json) are skipped, each PUT carries a SHA-256, and a failure or an unreachable store is a warning that ends that sync, never the run. RIFFLE_SYNC_URL must be https (plain http only for localhost, 127.0.0.1 and ::1), and a cross-origin redirect is refused so the Authorization header never follows it. .sync_state.json is locked (a .sync_state.json.lock file is created next to it), merges only the keys that changed, and skips a second file with the same base name with a warning; health JSON is written atomically. A wiped store is not detected (it needs a listing API on the store; deferred). ci_cache/, snapshots_raw/ and clones stay local. --no-sync turns it off.
  • merge_shards.py joins PCs (section 12).
  • Training reads the three tables with pipelines/training/frames.py: joins at load time (it raises if commit rows reference PRs missing from the dataset, and handles empty tables), splits by PR or repo from a stable hash (the repo name lower-cased; connected components grouped by repo and PR and by a shared head_sha or snapshot_head_sha across the three tables, so no commit straddles train and test), masks the complete list of 125 diff-derived columns on open_snapshot == final (read from x-riffle-diff-derived in the schema plus an explicit list: la_norm, LT, AGE, size percentiles, hotspot, blame and history columns, the diff-only LLM flags), drops near_decided snapshots, rescales snapshot weights per PR to 1, never offers outcome/id/metadata columns as inputs (x-riffle-non-input, ids, snapshot bookkeeping, merge_via, open_snapshot and the five provenance columns), and refuses snapshots that no longer match their dataset.

5. Feature families: how each is computed and why

All "per file" values are for the PR's changed files. The base commit is the at-open fork point. "Before base" means commits with committer time before the base commit's time (git --before).

5.1 Source 4a/4b: diff size, spread and paths (features/diff_stats.py)

Computed from git diff --numstat -M -z base...head (and --name-status -z, with -c core.quotePath=false on every git call, so a path with non-ASCII, quoted or backslash characters is read as it is and still gets its lines of code, blame and SZZ). The patch is read by a hunk-count parser, so a content line that starts -- or ++ is kept. One numstat, name-status and patch is taken per (base, head).

  • Generated/vendored files are removed from the risk counts but their share is kept (frac_generated_lines).
  • LA / LD: lines added / deleted in non-generated files. NF: non-generated files. ND: distinct directories. NS: distinct subsystems (config.subsystem_of: the top directory, or the first two components when the top is a generic container such as src, lib, pkg, packages, apps, internal, cmd, crates, ... so NS and SEXP do not collapse to 1). These are the Kamei (JIT defect prediction) size/diffusion metrics.
  • Entropy: normalised change entropy over per-file (added + deleted) lines, -Σ p·log2 p / log2(files), 0 for a single file. Only files with changed lines count (a pure rename or a mode change does not).
  • frac_test_lines, lang_mix (distinct recognised languages), n_files_added/removed/renamed (git diff --name-status -M), patch_missing_files (files git printed -\t- for: binary or too large; 0\t0 rows such as pure renames are not counted).
  • Path flags (touches_*) are any() over non-generated changed files using config.PATH_RULES (migrations, lockfiles, dependency manifests, CI config, infra, auth-like paths with token boundaries so author doesn't match, CODEOWNERS/riffle.yml). test_only / docs_only: all changed files are tests / docs.

Why: path flags carry "this kind of change is usually riskier" priors the model should not have to learn from text.

5.2 Source 2: repository history at the base (features/history.py)

The index is built once per repo and answered by bisect for each PR, so a PR no longer re-walks history (values are unchanged except for the last digit of a float in the decay and REXP sums, whose order of summation changed). build_repo_index makes one pass over all commits before the base (from the cached git log --all --numstat -M), keeping per-file aggregates only for the PR's changed paths (a vendor bump touching 5k files would otherwise create 25M co-change pairs) but computing repo-wide author aggregates and the repo-wide hotspot ranking.

Per-file (collapsed by aggregate.py, see the table in 5.2.1):

  • file_revert_density, file_fix_density: share of the file's prior commits whose subject is a revert / a fix (word-boundary matching, section 5.10). revert_decay_density, fix_decay_density: sums of 0.5^(age_days / 90) over such commits (half-life 90 days), so recent ones dominate. days_since_last_revert/fix.
  • file_churn_90d, file_churn_365d, recent_merge_burst_7d: prior commits touching the file in the window.
  • hotfix_count_365d: hotfix-worded commits touching the file in the last year.
  • test_cochange_ratio: share of the file's prior commits where a test file changed too (for non-test files).
  • file_post_merge_ci_fail_rate: share of the file's commits with a CI verdict whose CI failed (from the CI history map; null when no verdicts).
  • file_age_days: days since the file's first commit before base. NUC = number of prior commits touching it; NDEV = distinct prior human authors (bots are not counted).
  • Renames. Base-side lookups use the file's old path, and a file's history is followed across renames, so a renamed file keeps its churn, fixes and owners.
  • file_complexity_ccn: max function cyclomatic complexity of the file at base (lizard; 0.0 for a file with no functions; null for an unsupported language or lizard missing).
  • Ownership by lines (not commits): the file is blamed at base and each surviving line's author counted. file_top_author_share (largest author's share), file_minor_contributor_count (authors owning under 5% of lines), abandoned_lines_share (share of lines whose author has been inactive more than 180 days as of base). Blame is cached and shared with Source 3. Bot commits do not own lines; line ages are clamped to 0.
  • Hotspot = churn_365d × max CCN, ranked across every file in the repo at base with recent churn and a lizard-supported language (not just the changed ones), so "touches a top-20 hotspot" is a real repo-relative signal. file_hotspot_score, file_hotspot_rank (1 = hottest; null if unranked), touches_topk_hotspot, n_topk_hotspots_touched, min_hotspot_rank_touched. Only files with recent churn can score, which keeps this cheap on large repos. Complexity is cached per blob and per content hash.
  • _loc (file lines at base) feeds LT; _last_change_days feeds AGE. Both are internal and not emitted per file.

PR-level:

  • LT = total lines of the changed files at base; la_norm / ld_norm = LA/LT, LD/LT (0.0 when LT is 0). AGE = mean days since last change across changed files that have history (null when none).
  • cochange_degree = max over changed files of the number of "strong partners" (files that changed together in at least 3 prior commits); cochange_partners_missing = max over changed files of strong partners not in this PR ("you usually change X with Y").
  • expected_test_cochange_missing: the PR changes a source file whose test_cochange_ratio ≥ 0.5 but contains no test file.
  • Author behaviour (author keyed by canonical email): author_revert_rate is smoothed toward the repo base rate, (reverts + α·base_rate)/(total + α) with α = 5 (empirical Bayes), where reverts are the author's commits that are reverts; author_file_familiarity = share of touched paths the author touched before; author_days_since_last_change; author_post_merge_ci_fail_rate (same smoothing, null with no CI history for the author); repo_revert_base_rate = reverts / all commits before base (merge commits excluded).
  • Kamei experience: EXP = author's prior commits; REXP = recency-weighted, Σ 1/(age_years + 1); SEXP = author's prior commits in the touched subsystems.
  • Repo-relative percentiles: pr_size_pctile_repo, nf_pctile_repo, entropy_pctile_repo = fraction of prior non-empty commits strictly smaller. The population is per commit (a point-in-time proxy for PR size), skips merge commits and generated files.
  • FIX: the PR title at open plus the head commit's subject as at open name a bug/fix as a whole word.
  • author_prs_merged_repo: how many PRs the same git author had merged before this PR opened (bisect over the author's sorted merged times). null without a PR list; 0 for an author with none.
  • concurrent_open_prs_same_files / _same_hunks: other PRs that opened at or before this one and were still open at this one's opening, touching a common file / overlapping base-side lines (git diff -U0). Computed over the fetched PR set, so a small --max-prs undercounts. Unmerged PRs count (closer to what was actually open); stale-open PRs count as open "forever".

5.2.1 Aggregation (aggregate.py)

A PR touches 1..N files but the model needs fixed columns. Each per-file feature is collapsed by AGG_SPEC; default max if unspecified:

Per-file featureEmitted columns
file_revert_densitymax_, mean_
revert_decay_densitymax_
days_since_last_revertmin_
file_fix_densitymax_, mean_
fix_decay_densitymax_
days_since_last_fixmin_
file_churn_90d, file_churn_365d, recent_merge_burst_7dmax_
hotfix_count_365dmax_, sum_
test_cochange_ratio, file_post_merge_ci_fail_ratemax_
file_age_daysmean_
NUC, NDEVmax_
file_complexity_ccnmax_, sum_
file_hotspot_scoremax_
file_hotspot_rankmin_ (the hottest file touched)
file_top_author_share, file_minor_contributor_count, abandoned_lines_sharemax_
mod_line_age_days_minmin_
mod_line_age_days_medianmedian_
mod_lines_distinct_authors, mod_lines_prior_fix_sharemax_
mod_lines_self_authored_sharemean_
hunk_fix_proximitymax_, mean_

Nulls are ignored in aggregation; if all are null the aggregate is null. This is why a column name such as median_mod_line_age_days_median looks odd: it is median_ of the per-file mod_line_age_days_median. Note min_hotspot_rank_touched (from _hotspot_touch) and min_file_hotspot_rank (from aggregation) are two columns for the same idea.

5.3 Source 3: line-level blame (features/line_history.py)

For each changed file, the lines the PR deletes or modifies (base-side line numbers from a -U0 diff) are looked up in the base-commit blame:

  • mod_line_age_days_min / _median: age of those lines.
  • mod_lines_distinct_authors: how many people wrote them. mod_lines_self_authored_share: share written by the PR author.
  • mod_lines_prior_fix_share: share whose last-touching commit's subject is a fix.
  • hunk_fix_proximity: recency-decayed (half-life 90 days) count of base lines owned by fix commits that lie within ±5 lines (hunk_proximity_k) of any modified line: "you are editing next to old bug fixes".
  • Files with no modified lines or no blame (a PR that only adds files) get null ages, 0 counts/shares. A blame that failed or timed out is null, not 0. A renamed file is blamed at the base under its old path.

These are the only history features anchored to specific lines. They are novel as model inputs: validate with an ablation before claiming lift.

5.4 Source 4d: scanners (features/scanners.py)

  • lizard (ccn_delta, max_ccn_changed_func, n_funcs_modified): every changed file in a supported language is analysed at base and head; functions are matched by long_name; a function whose CCN changed (a new function compares against 0) counts as modified; ccn_delta is the sum of differences, max_ccn_changed_func the largest head CCN among them. Deleted functions count in ccn_delta and n_funcs_modified. A renamed file is read at the old path for its base content. null when scanners are off, lizard is missing, or lizard hit a parse error (all three CCN columns are then null).
  • semgrep (semgrep_findings_new, semgrep_high_conf_security_new, semgrep_cwe_top25_new): the changed files are exported at head and base to temp dirs and scanned. A finding is new when the multiset of (rule, path relative to the exported tree, normalised matched code) at head has more of it than the base scan; no line number is part of the key, so moving code is not a new finding. The base scan is skipped when head has no findings (80% of PRs in the vite pilot; halves semgrep starts). "High-confidence security" = metadata category: security and confidence: HIGH. The Top-25 flag is true if any new finding's CWE is in a fixed Top-25 set. A semgrep start costs about 4 s, the PR's biggest cost; batching several PRs per process was tried and measured no faster.
    • A scan that did not run is null, never zero. A missing tool, a return code other than 0 or 1, bad JSON or a timeout makes all three semgrep columns null (before 7.0.0 a missing tool gave 0, and a failed base scan counted every head finding as new).
    • Rules and egress. RIFFLE_SEMGREP_RULES (a file or a directory) pins local rules; without it the scan uses --config auto, which fetches the rules from semgrep.dev, so that request leaves the machine. --metrics=off --disable-version-check is always passed. RIFFLE_SEMGREP_PARALLEL (default 2) limits concurrent semgrep processes, each with --jobs 1.
    • Export safety. .semgrepignore, .gitignore and .ignore are never exported, and every exported path is checked with a real-path containment test.

5.5 Source 4e: dependencies and CVEs (features/deps_cve.py)

Only when a manifest or lockfile changes.

  • Ecosystem is inferred from the filename (npm, PyPI, Go, crates.io, RubyGems, Maven, Packagist, NuGet by extension, Pub, Hex). Both versions are parsed best-effort per ecosystem (pom.xml resolves ${property} versions), and each dependency is classified added / removed / upgraded / downgraded by semver comparison. Parsers cover manifests and lockfiles: package-lock.json (v1, v2 and v3), yarn.lock, poetry.lock, Pipfile.lock, composer.lock, plus the manifest formats above. Counts are kept per (ecosystem, directory), and a lockfile overrides the manifest's range. PyPI names are PEP 503-normalised and extras are read; go.sum and /go.mod suffixes are stripped.
  • Concrete versions only. A range, workspace: or a git URL is unresolved: it cannot be looked up, so the CVE columns are null.
  • n_deps_added, n_deps_upgraded, n_deps_downgraded, n_major_bumps (upgrades whose major changed) need no network. They are null when any touched manifest is unparsed or unreadable (pnpm lockfiles, pyproject.toml, Gradle, a git read that failed), instead of counting only the files that were understood.
  • Point-in-time CVEs. deps_cve.compute takes as_of: the PR's open time (for a snapshot, the time the head was first seen). OSV and GHSA records published after as_of, or withdrawn, are dropped, so a vulnerability disclosed after the PR opened cannot appear. An undatable record, or an as_of that cannot be read, makes all CVE columns null.
  • With the network: OSV.dev is queried for every before/after (ecosystem, name, version). osv_vulns_introduced = IDs present at head and not at base; osv_vulns_fixed the reverse. vuln_has_fix_available is False when no introduced record lists a fix and null when any does: OSV does not date the release of a fix, so "a fix exists" cannot be shown as of as_of. ghsa_max_severity_introduced (LOW/MODERATE/HIGH/CRITICAL, from GHSA-sourced OSV records, or a CVSS score fallback) and ghsa_malware_dep (a MAL- ID or alias, or type: malware) come from the introduced records.
  • osv_ghsa_disagreement: for versions newly present at head, does GHSA's GraphQL verdict (vulnerable range match) differ from OSV's? null with no token, and null when GHSA gave an error reply, a hasNextPage, a version range that could not be parsed, or more than 100 vulnerabilities.
  • Unobserved is not clean: if any OSV query fails after retries (429/5xx back-off up to 4 attempts, Retry-After honoured), or the reply is paged, all CVE fields stay null rather than presenting an undercount as a reassuring number. The OSV cache holds successful answers only.
  • Caveat: with no manifest change, the CVE fields are also null (the counts are 0), so for the CVE columns null means both "no dependency change" and "unobserved".

5.6 Source 5: deterministic checks (features/deterministic.py)

Regex over the added lines of the diff (and removed lines for observability). Secrets, unused imports and comparison edits are scored on real source only (tests, docs and generated files excluded), so a test fixture does not inflate hardcoded_secret_added. Env keys are scored on all files because config files legitimately carry them.

ColumnRule
hardcoded_secret_addedcount of added source lines matching: api_key/secret/password/passwd/token assigned a quoted string of 8+ chars; AWS key id AKIA...; GitHub tokens gh[pousr]_...; PEM private key header
new_env_or_config_keyadded lines using os.environ, process.env., getenv(, System.getenv(
observability_deltaremoved log/metric lines minus added ones (positive = net removal)
bound_or_comparison_editadded source lines containing a comparison, ==, !=, +1, -1 or [i±1]: the cheap "off-by-one-ish edit" signal
unused_import_or_dead_code_addedadded import/from lines whose first symbol never appears in the other added lines (very rough, Python/JS)
dependency_major_bump_countpackages whose major version rose between removed and added manifest/lockfile lines (coarse name/version parse; a bare "version": "1.2.3" line is now attributed to the package named on the nearest preceding line, and skipped when there is none: it used to be read as a package called version, so every lockfile entry collapsed into one)
migration_reversibilitydestructive if an added line in a migration path has DROP TABLE/COLUMN, TRUNCATE, DELETE FROM; reversible if a migration changed with none; null otherwise

Note dependency_major_bump_count (regex) and n_major_bumps (parsed manifests, Source 4e) are two different measurements.

5.7 Source 5: API contract change (features/api_contract.py)

For changed spec files (.proto, .graphql/.gql, and .yaml/.yml/.json whose first 2,000 characters contain openapi: or swagger:): OpenAPI → oasdiff breaking, protobuf → buf breaking, GraphQL → graphql-inspector diff. A removed spec is breaking, a brand-new one additive. graphql-inspector exit 0 is additive; exit 1 is breaking only when it prints Detected N breaking changes, otherwise (unparsable schema, crash) the verdict is null. The PR value is the worst across spec files: null (no spec changed, or no tool installed / the tool failed to give a verdict), additive, breaking. A PR is null whenever any changed spec is unobserved, unless another spec is already breaking. oasdiff with a non-zero return code or unparseable output gives no verdict; buf is run with --error-format json. Each tool is optional.

5.8 Source 6: LLM flags

See section 7.

5.9 Merge-model signals (features/merge_signals.py)

The AIDev feature catalog's merge-model features the miner's own data can supply.

Per PR:

  • has_linked_issue (fixes/closes/resolves #N, owner/repo#N, or an issue URL in the body), body_headings (Markdown headings), body_checklist_items (- [ ] / - [x]), body_has_test_plan ("Test plan", "How to test", "Testing" heading), agent_boilerplate (an agent watermark: "Generated with Claude Code", "Co-authored-by: Claude/Copilot", "This PR was generated by", "Devin run", "OpenAI Codex", "Cursor agent", ...).
  • agent_name: the first matching agent among claude_code, codex, copilot, cursor, devin, jules, sweep, tested against the author login, head branch, git email and body. Login patterns are anchored (jules[bot], google-labs-jules[bot]), a branch prefix needs its /, and cursoragent, devin-ai and sweep-ai need an exact match, so julesb, claude-docs and cursor-fix are not agents. head_branch_agent_prefix: branch starts claude|codex|copilot|cursor|devin|jules|sweep followed by / or -.
  • author_account_age_days: open time minus the account's creation time (one cached /users call per author; static so it is point-in-time safe).
  • has_contributing, has_agents_md: files at the base commit (two ls-tree calls): a contributing guide at the root, .github/ or docs/; an agent instructions file (AGENTS.md, CLAUDE.md, .cursorrules, .cursor, .github/copilot-instructions.md).
  • The body is read at mining time (edits are not in the timeline), so body features can see edits made after open (ADR 0010 open item: body at open from GraphQL userContentEdits). The timeline has no edit history, so as-of-open can only be null; this is deferred, not fixed in 7.0.0 (body_len, has_linked_issue, body_*, agent_boilerplate, author_association).

Per repo, prior-only (a PR counts only if it closed strictly before this one opened, within the last 365 days; merge_signal_min_prs = 5; null below that). Implemented by merge_signals.History, indexed so it can be recomputed as of any moment (also used for snapshots):

  • repo_outside_merge_rate (+ repo_outside_closed_count): share of outsiders' closed PRs that merged.
  • agent_recent_merge_rate (+ agent_recent_closed_count): same, for the same agent (agent rows only).
  • repo_median_first_response_hours: median hours from open to the first human non-author comment/review over prior PRs that got one (only responses before the moment count).
  • repo_agent_pr_load_30d: agent PRs opened in the previous 30 days.
  • max_sim_open_prs: highest Jaccard word-set similarity (title at open + body, stop-words removed) to a PR open at that moment. max_sim_recent_merged: against PRs merged in the previous 90 days.
  • The merge rates need unmerged rows, so they stay null with --no-merge-outcome.
  • "Outsider" = author_association not OWNER/MEMBER/COLLABORATOR and not a maintainer confirmed from history.

5.10 Text matchers (config.py)

Substring matching ("fix" in the subject) wrongly fires on prefix, suffix, fixture, debug, dispatch, patch release. These use whole-token matches with common inflections; underscores and digits count as boundaries (bugfix_test matches, debugger does not).

  • Fix: bug(s|fix(es|ed)), fix(es|ed|ing), defect(s), patch(es|ed|ing), hotfix(es|ed).
  • Revert: a subject starting revert, revert ", revert:; a conventional-commit prefix then revert (fix: revert "x", revert(server): x, as squash-merge repos title reverts); or "this reverts commit".
  • Hotfix: hotfix/hot-fix/hot fix, rollback/rolled back/roll back, quickfix/quick fix, or urgent|emergency|critical followed by fix|patch|bug|issue|regression|security. Cherry-picks and backports are deliberately not hotfixes (they port a change to another branch).

5.11 Repo-local mined features (features/repo_local.py)

The nine columns of contracts/mined_features.schema.json, capturing how this repository behaves around the paths a PR touches. Point-in-time rule (contract): a past PR contributes only if it merged before this PR opened and its labels were mature (merged ≤ opened − maturity_days (30)); its outcome is what was observable at open (an outcome event after open does not count); only reviews submitted before open count. History is only merged PRs mined in the same run, so with a small --max-prs the oldest rows have little history and come back null. Unmerged rows get these columns from the history but never count as past PRs. Thresholds mirror rules.mining defaults.

ColumnDefinition
similar_pr_countpast mature PRs (within 365 days) with file-set Jaccard overlap ≥ 0.2, capped at the 5 nearest (ties broken by recency); 0, not null, when none (the contract defines 0 as "none"; unchanged in 7.0.0)
similar_pr_bad_ratewith at least 3 neighbours: (bad + (k − n)·prior)/k, k = 5, missing neighbours filled with the prior; the prior is the repo's own bad-PR rate among past PRs that were mature at the PR's open time T (0.0 when none were); null otherwise
path_failure_mode_revert/_ci/_hotfixwith at least 2 bad past changes on these paths in the last 365 days: share of each primary failure mode (the earliest outcome event)
path_detection_lag_daysmedian days from merge to that outcome event over those bad changes
revealed_path_scrutinymean per-file review "scrutiny" on the touched files divided by the repo mean (score = comments + 2 × changes-requested + review rounds, bots excluded, 180-day lookback, at least 5 prior PRs per file and for the window); needs review events (2 API calls/PR)
path_size_pctilepercentile of this PR's LA+LD among prior commits (365 days, before base) touching the same files, weighted by this PR's lines per file, ties count half; null with fewer than 10 distinct prior commits
release_cycle_positiondays since the previous release tag divided by the median gap between releases; tags matching ^v?\d+\.\d+(\.\d+)?$; null with fewer than 3 prior releases

The contract's evidence block (similar PRs, top scrutiny path, top failure mode, size context) is for the narrator at scoring time and is not written to the training dataset.

5.12 Bots, agents and "outside contributors"

  • author_is_bot = automation only (Dependabot, Renovate, github-actions, other [bot] accounts) from git identity, GitHub user type or login. ai_agent_authored = AI coding agents only (same signals as agent_name: account, branch prefix, commit trailer/email, body boilerplate). An agent PR is never a bot PR, so agent PRs stay in --llm-scope training.
  • author_association is today's status and reports private org members as CONTRIBUTOR. So "outside contributor" for scope, snapshots, outsider rates and health is association in {CONTRIBUTOR, FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, NONE} (or unknown) and not author_confirmed_maintainer.
  • author_confirmed_maintainer (maintainers.py, proposal 2.2): every timeline is read up front; anyone who merged or closed someone else's PR (a human, not a bot) is a maintainer from that moment; asked point-in-time for the PR author. It catches people who triage; an employee who never merges or closes others' PRs still counts as outside. Saved as snapshots_raw/<repo>/maintainers.json, also used for snapshot actor roles. null when no timelines were read.
  • Training PRs (scope.is_training): outside, non-bot PRs plus a fixed llm_sample_pct (5%) sample of the rest (a SHA-1 of repo#number mod 100, the same PRs on every run). An unknown association counts as outside, so git-only runs keep their LLM columns.

6. Labels (targets)

Labels are outcomes. They are never model inputs, and they are written beside the features so training can choose.

6.1 Breakage labels (labels.py, ADR 0002)

For each merged PR the miner records when each outcome happened, not only whether it happened inside a fixed window. Booleans are convenience values at the default windows; any other window is a training-time comparison on the days_to_* columns.

ColumnDefinition
days_to_revertDays from merge to the first commit that reverts this PR. Searched from merge to merge + label_lookahead_days (90) on the default-branch tip (git log --all only as a fallback), so a commit on an unmerged branch is not a revert. Matching is strict: the revert must reference the merge commit's SHA (7-char or full), have this PR's number as #N in its subject (as squash-merge reverts do, fix: revert "x (#N)") or Reverts owner/repo#N in its body, or quote this PR's exact title as Revert "<title>". The PR's own merge commit and own commits never count, and a PR number must be lower than the revert's. The old loose rule (title anywhere in the revert body) made short titles like "Fix typo" match unrelated reverts. A revert merges after the PR it undoes, so a #N in its message can only point at an earlier PR; the same order is required of a hotfix. null = not observed. Needs a merge commit SHA.
days_to_hotfixDays from merge to the first hotfix-worded commit (5.10) touching a significant shared file with this PR. Files that co-change in nearly every PR (generated files, docs, lockfiles, CHANGELOG) do not count, so a hotfix that merely bumps a changelog or lockfile is not pinned to every PR that touched them. Like reverts, it never matches the PR's own merge commit or commits.
days_to_szz_fixDays from merge to the earliest later fix commit that repaired a line this PR introduced (SZZ, section 10), bounded by szz_maturity_days (90) so older PRs don't accumulate an unbounded label rate. A fix inside the same PR, or dated before the merge, doesn't count. null when SZZ did not run.
label_revertdays_to_revert ≤ revert_window_days (30)
label_hotfixdays_to_hotfix ≤ hotfix_window_days (7)
label_ci_failThe required/gating build check failed on the merge commit; only the commit's push-triggered workflow runs (and other CI apps' checks) count (6.2). Tri-state: true failed, false passed, null no CI verdict observed (no CI on push, retention expired, unreadable). null is not a pass; use ci_observed. Failures that come only from flaky checks are corrected to false by flaky.py (6.4).
label_ci_fail_optionalThe same for auxiliary checks (CodeQL, Scorecards, coverage, ...). Diagnostic only; never in the target.
label_badThe v1 training target: label_revert OR label_hotfix OR (label_ci_fail is true). Only observed positives count; an unobserved CI verdict does not force false. SZZ is not included (it tops out near 0.6 precision and catches normal iteration). Union it yourself if wanted: `y = label_bad
label_szz_bugSZZ found a later fix of a line this PR introduced (days_to_szz_fix not null). Noisier, kept separate. null when SZZ did not run (for example --no-szz), not false.
szz_window_closedOptional (contract 1.3). The 90-day SZZ window had fully elapsed by observed_through, so a false label_szz_bug is mature. null when the SZZ label was not computed or szz_maturity_days is 0. Never a model input.
ci_observedA required-CI verdict existed on the merge commit.
revert_window_closed / hotfix_window_closedThe 30 d / 7 d window had fully elapsed by observed_through.
label_bad_observedci_observed and both windows closed: every constituent of label_bad was fully observed. frames.py builds the breakage frame from rows with this true.
observed_throughHow far forward outcomes could be seen: the newest commit time on the default branch in this clone, capped at now. A revert after it is not in the clone, so a row is only mature up to here, not up to now.
mined_atWhen the row was extracted (UTC). Not a label or a feature. Maturity is decided from it at training time.

Maturity (ADR 0002). 30 days, or 90 for SZZ; younger PRs are dropped from training. --min-age-days 90 skips PRs merged in the last 90 days at mining time so a capped selection picks PRs old enough (busy repos merge 100 PRs in days, so without it most rows would be too young). In code the mined_at − merged_at check is done at training time:

python
age = (pd.to_datetime(df.mined_at) - pd.to_datetime(df.merged_at)).dt.days
m = df[df.label_bad | (age >= 30)]       # positive, or old enough to know it is negative

Unobserved sources (ADR 0002): labels.unobserved = downweight (contract default): an observed source that fired still makes the row positive; otherwise the row counts as clean at weight 0.5 per missing source. CI history on public repos is capped at about 90 days by GitHub.

Post-merge columns are null on unmerged rows: days_to_revert/hotfix/szz_fix, label_revert/ci_fail/ci_fail_optional/hotfix/szz_bug/bad, ci_observed, revert_window_closed, hotfix_window_closed, szz_window_closed, label_bad_observed, ci_fail_flaky_only, plus observed_through.

Declared hotfix (declared_hotfix) is a feature, not a label: was a label containing hotfix or rollback on the PR at open (+120 s grace), rebuilt from labeled/unlabeled timeline events? An author/triager prior set before review. It replaces the earlier label_is_hotfix, dropped because the PR list returns labels as of mining time, so reactive tags added after a PR went bad (regression, reverted, security, a later hotfix) would leak the outcome. Also dropped: n_commits (list endpoint never returns it) and is_draft (constant for merged PRs).

6.2 How CI is read (github_api.check_conclusion)

  • Reads the Checks API (all pages, up to 2,000 runs) with filter=all, and the legacy commit-status API (Jenkins, Buildkite, CircleCI, Travis), merging them: a repo can gate on either, and reading one would miss failures. A failed or truncated page of either, or of the workflow-runs read below, leaves the verdict null rather than judging from the part that arrived.
  • Each check's verdict is its latest run. A check whose earlier run failed but latest passed is a rerun-pass (input to flaky detection).
  • failure, timed_out, startup_failure, action_required are failures; success, neutral, skipped are non-failures. Reduction: any failure → failure; some ran and none failed → success; else null.
  • Required vs optional. If branch protection's required checks are readable they are authoritative (a check is required if its name equals, contains or is contained in a required context). Otherwise a name denylist of auxiliary checks is used: security scanners (CodeQL, Snyk, Sonar, Semgrep, Trivy, ...), coverage reporters, deploy previews (Netlify, Vercel, ...), housekeeping bots (labeler, stale, greeting, ...), sign-off/PR-metadata checks (CLA, DCO, commitlint, semantic PR title, ...), docs builders. The list deliberately excludes ambiguous checks that are often genuinely gating (lint, test, build, typecheck, visual regression, bundlesize, pre-commit): marking one optional would hide a real build failure.
  • Which runs count (6.3.0). A check run attaches to whichever commit was the default branch's head when an unrelated event fired (an issue labelled or evaluated, a schedule, a /command comment), so a merge commit's check list mixes real CI with automation. For the merge commit, check_conclusion(sha, events=("push",)) asks the Actions runs API for the workflow runs on that sha (/actions/runs?head_sha=), keeps the check suites started by push, and drops GitHub Actions check runs from other suites. Check runs from other apps (CircleCI, Buildkite) have no workflow event and always count. A commit left with no counted run has no verdict (ci_observed false). If the runs API is unreadable the filter is skipped, and the verdict is then reduced from every check run: a failure stays failure, but a would-be pass is null, because the runs that were not filtered out may have been automation. It costs one extra API call per merged PR. The close-reason lookup for unmerged PRs (check_conclusion(final_head)) keeps counting every run, because CI there runs on pull_request. Checked on 14 sampled vite positives: 2 became passes, 12 stayed failures. preview joined the optional-check denylist.
  • The historical CI map (stage 8, cache details in 8.2) uses the Actions workflow-runs list, excludes pull_request-event runs (the PR's own pre-merge iteration would leak the outcome into a post-merge signal), and excludes optional workflows so the file/author CI-fail rates match label_ci_fail. Only GitHub Actions is covered there; commits with no runs are absent (feature null).

6.3 Merge-outcome labels (merge_labels.py, ADR 0009)

From the PR timeline (reviews, comments, commits, force-pushes, duplicate marks, labels, closes) plus the CI verdict on the head commit for unmerged PRs. All outcomes, never inputs; null = not observed. Events after the close (plus 1 minute for the closing comment and label) are ignored. The rules are pure functions of the event list.

ColumnRowsRule
label_mergedallmerged_at present (state is not used: closed covers merged PRs too)
label_reworkmergeda human non-author CHANGES_REQUESTED review, or a commit or force-push more than 5 minutes (rework_grace_s = 300) after open. A commit's time is its committer date, which a rebase rewrites, so a rebase after open counts too (it comes with a force-push). "Update branch" merges from the base (commits with more than one parent) are not rework. Bot and own change requests are not rework.
post_open_commits, post_open_force_pushes, changes_requested_reviewsmergedthe counts behind label_rework (up to merge)
label_duplicateunmergeda marked_as_duplicate event, a duplicate label, or a comment/review matching "duplicate of", "dupe of", "superseded by", "supersedes", "already fixed/addressed/done/implemented/merged in/by/via", "closing in favour of", "in favour of #N", "replaced by #N" (authors close as superseded too)
label_unwantedunmergeda maintainer (OWNER/MEMBER/COLLABORATOR, not the author, not a bot) comment saying out of scope, won't fix, "not something we want", "we won't accept/merge", "not planned", "doesn't fit", "not interested", "decline this PR", or a close label matching unwanted_labels (substring match: wontfix, won't fix, invalid, not planned, out of scope, declined, rejected, spam; "rejected AI" counts as "rejected")
label_abandonedunmergedno human non-author comment or review before close, or closed by a stale bot (stale[bot], stale-bot, stalebot), a stale label, or a bot saying stale/inactive; also every stale-open row
label_ci_technicalunmergedrequired checks failing on the head commit at close, with no maintainer objection and not a duplicate; null when the CI verdict wasn't observed
close_reasonunmergedwith the LLM: the classifier's pick (see below). Without it, or for a PR with no human activity: the first rule that fires, in the order duplicate, unwanted, ci_technical, abandoned, else other.
close_reason_sourceunmergedllm or rules
created_at, closed_atallfor time splits (closed_at null for stale-open rows)
outcome_3classalllands_clean (merged, no rework), lands_after_rework, goes_nowhere (unmerged or abandoned); null when rework wasn't observable
nowhere_reasonunmergedabandoned, duplicate, unwanted, ci_technical, or unknown (the timeline was read and no rule/classifier found a reason; other maps to unknown); null on merged rows and when the timeline was unreadable

A partial timeline (any failed page, or past 2,000 events) is treated as unreadable (null labels beyond label_merged): the lost part would be the newest events (the close), so a partial timeline would mislabel the PR rather than leave it unobserved.

The close-reason classifier. Asked only for unmerged PRs in LLM scope. With --close-reason-llm other_only (default) it is asked only when the rules find no reason (result other); always asks for every unmerged PR with human activity. Input (close_transcript): the title as of the close and the timeline up to the close, with each person's role (author, maintainer, bot, other), comments and reviews (1,500 chars each, newest entries kept up to 12,000; every line of a comment body or title is quoted with a | prefix, so a body cannot forge a role line), label events, renames, closer, force-pushes. If nothing human is in it (nobody responded) the rules decide and no call is made; a person other than the author closing it counts as human even without a comment (a maintainer closing and retitling a PR "AI spam" is a rejection, not abandonment). One of duplicate | unwanted | ci_technical | abandoned | other is chosen from a JSON-schema-constrained answer. The classifier's answer decides close_reason and label_unwanted and adds to the duplicate/abandoned/CI-technical flags. It is monotone: it may raise a false or null flag to true, never lower a flag the rules set to true. close_reason_source is llm whenever the LLM's reason is the one used. It is retried like the flags; if it still has no answer the repo stops rather than label the PR by rules alone. It runs only on the miner's training rows, never on the scoring path.

Rejected alternatives (ADR 0009): one multiclass target; folding not-merged into label_bad; using state; fetching unmerged branches as local refs; training a supervised unwanted classifier first (no labelled set yet); treating every silent close as abandoned.

6.4 Flaky-check correction (flaky.py, rules.mining.flaky_detection)

After the repo is mined, a check is flaky at a merge if it had at least 3 rerun-passes (flaky_min_rerun_passes) in the 90 days (flaky_lookback_days) before that merge, counted over the repo's mined merge commits. A merge-commit CI failure that comes only from checks flaky at merge time is not counted: label_ci_fail becomes false, ci_fail_flaky_only becomes true, and label_bad is recomputed as label_revert OR label_hotfix. Point-in-time: only rerun-passes strictly before a PR's merge count. ci_fail_flaky_only is false when CI was observed and not flaky-only, null when CI wasn't observed and on unmerged rows.

6.5 What is not a label

days_to_*, label_*, post_open_*, changes_requested_reviews, close_reason*, outcome_3class, nowhere_reason, ci_observed, windows, observed_through, mined_at, created_at, closed_at, merged_at, snapshot_weight, near_decided are outcomes, filters or weights. pipelines/training/frames.py reads the contract and never offers outcome, id or metadata columns as model inputs.


7. The LLM subsystem (features/llm_flags.py)

7.1 What the LLM is for, and is not for

  • 23 semantic flags that a parser cannot answer ("did this PR weaken an authorisation check?"), plus the close-reason classifier.
  • The LLM reads only the diff. Never the PR title or description: they can mislead it, and they carry the outcome.
  • Flags are features for the model, never the verdict, and the LLM is never on the scoring path (project invariant).
  • With the LLM off, every flag is null.
  • Data leaves the machine. The LLM is on by default, and the diff of each in-scope PR is sent to the backend (Anthropic via Claude Code, or the api endpoint). Obvious secrets are redacted first (below), but the code is not. --no-llm sends nothing.

7.2 The 23 flags (FLAG_SCHEMA)

FlagValues
llm_touches_authn_authzbool
authz_check_changenone, added, removed, weakened, moved
changes_public_api_signaturebool
error_handling_changenone, added, removed, modified
exception_swallowedbool
db_schema_or_query_changenone, schema, query, both
transaction_boundary_changenone, added, removed, widened, narrowed
input_validation_changenone, added, weakened, modified
config_or_feature_flag_changenone, config, feature_flag, both
feature_flag_guardingn_a, guarded, partial, unguarded
security_sensitive_opsbool
resource_lifecycle_changenone, acquire_no_release, release_removed, modified
null_handling_changenone, added, removed, new_nullable_deref
numeric_money_precisionbool
time_handling_changebool
retry_timeout_idempotency_changebool
refactor_vs_behaviorpure_refactor, behavior_change, mixed
change_intentfeat, fix, refactor, perf, docs, test, build_ci, chore
unrelated_change_shareinteger 0 to 3
tests_added_for_behavior_changeyes, no, n_a
test_code_alignmentnone, unrelated, partial, aligned
satd_addedinteger ≥ 0 (self-admitted technical debt added)
semantic_risk_1to5integer 1 to 5

The system prompt tells the model: judge strictly from the diff, never speculate about code you cannot see, never invent line references, return exactly the schema's keys, and when a flag does not clearly apply use the most neutral value (none/n_a, false, lowest number). _validate coerces any non-conforming value to null. The same schema is passed as a JSON Schema to claude -p --json-schema.

7.3 Which rows are sent, and what is sent

The flags are features, so only rows the model trains on need them. Order of checks (_llm_skip_reason):

  1. Scope (--llm-scope training, default): outside contributors who are not bots plus a stable 5% sample of the rest (--llm-sample-pct). --llm-scope all sends every row. Out-of-scope rows keep null LLM columns, and snapshots/capture use the same scope.
  2. Trivial diffs (llm_skip_trivial): docs-only, tests-only, lockfile-or-generated-only diffs are not sent (their flags would all be the neutral value). Resolves to null.
  3. Reuse (re-mine): an earlier answer for the same (base_sha, head_sha) is reused, no call. It passes through _validate again and is skipped when its recorded prompt_version or model differs from the current ones (records from older versions without those fields are still reused; _save_llm_reuse writes them).
  4. The diff sent: git diff -M -U1 base...head (1 line of context), secrets redacted (_redact_secrets: quoted and unquoted key = value and JSON key-values for secret|password|token|api_key, Authorization and Bearer values, private-key blocks, ghp_, github_pat, sk-, AKIA, xox and AIza tokens, and credentials in URLs → <redacted>), lockfiles and generated/vendored/minified/snapshot/.map/.svg files cut to one line each (# path: reason, +N -M lines (diff omitted)), and the rest capped at --llm-max-diff-chars (25,000) keeping whole files where possible and naming what it dropped (a single oversized file is truncated and marked). When the cap dropped or cut code the model would have read, llm_diff_truncated is true (optional contract column, never an input).
  5. An empty diff, or one with no code left after slimming (an .svg-only change), is status empty: flags null, no call, not a failure.
  6. A reply with zero valid flags is a failure. _validate rejects booleans where a number is expected, negatives, NaN and infinity.

Fetch statuses: off, skipped, empty, ok, fail.

7.4 Backends

  • claude_cli (default): headless Claude Code, claude -p --output-format json --model <m> --setting-sources "" --tools "" --strict-mcp-config --disable-slash-commands --no-session-persistence --system-prompt ... --json-schema ..., run from an empty temp dir with no tools, MCP, settings or slash commands, so no CLAUDE.md, hook or plugin leaks into the judgement. Uses the local Claude Code login (no API key); model sonnet unless LLM_MODEL. Timeout 180 s; two quick retries with waits of 10 s × attempt; returns None on failure. llm_model records the model id the CLI reports (modelUsage), else the alias. Token counts and total_cost_usd are accumulated per run (on a Pro/Max login that is Claude Code's estimate of API pricing, not a charge).
  • api (LLM_BACKEND=api): any OpenAI-compatible endpoint with the openai SDK and LLM_API_KEY; default LLM_BASE_URL Anthropic's endpoint, default model claude-sonnet-5; llm_model is the model the response names (resp.model), else the alias. Transient statuses (408, 409, 425, 429, 5xx, including 529) and timeout/connection/rate-limit errors retry up to 6 attempts with back-off and Retry-After; permanent errors (bad key/request/unknown model) fail fast so a misconfiguration cannot hang the run. No temperature is sent (newer Claude models reject it).
  • --llm is on by default and is checked before any repo is mined: when no backend is usable (unavailable_reason: no claude on PATH, or api without a key/SDK) both build_dataset.py and mine_repo.py stop with an error, so no dataset is silently missing its LLM columns. --no-llm is the explicit way to mine without them.
  • --llm-backend (LLM_BACKEND) and --llm-diff-context (LLM_DIFF_CONTEXT, default 1) are build_dataset.py flags that reach the miner as environment variables.
  • --llm-test makes one call on a known-risky sample diff (auth check removed, exception swallowed) and reports whether the answer looks sane.

7.5 Never a row without its answer

  • A failed call for a PR in scope is retried in place up to llm_row_retries (3) with waits of llm_retry_backoff_s × attempt (60, 120, 180 s); later PRs queue behind it, so the whole miner pauses.
  • If it still fails, the repo stops: rows already done stay in the checkpoint, --out is not written, the miner exits 3, and build_dataset.py aborts the batch. A rerun resumes from that PR.
  • There is no separate circuit breaker (7.0.0 removed the dead one in resolve and the llm_breaker_threshold setting): the per-PR retry in mine_repo is the stop. llm_breaker_tripped stays as the flag that records it.
  • The close-reason classifier follows the same rule.

7.6 Plan limits (headless Claude Code)

  • At most every 5 minutes (llm_quota_poll_s) the miner reads plan usage with claude -p /usage (a local command: no model call, no tokens, no quota). RIFFLE_NO_PLAN_USAGE=1 turns this off.
  • At 95% of a limit (llm_quota_pause_pct), or when a call is refused for a limit (message matched by regex, with the reset time parsed), every LLM caller pauses until the reset (one shared deadline across threads) and then continues; PRs wait for their answers. Pause time is counted as quota_wait, not work.
  • A reset more than 12 h away (a weekly cap, llm_quota_max_wait_h) is not waited for: calls fail and the run stops. The total time spent waiting is capped by the same setting, and a limit message in non-JSON CLI output is recognised too.
  • --no-llm-quota-wait turns the whole mechanism off. Unreadable usage (no subscription, API backend) never blocks.

7.7 Provenance

Every row carries llm_model and llm_prompt_version: a SHA-1 (12 hex) of the flag prompt, schema prompt, JSON schema, the close-reason classifier prompt and its schema, and (from 7.0.0) the slimming rules, the redaction patterns, the diff prefix, llm_max_diff_chars, llm_diff_context, the backend and merge_labels.TRANSCRIPT_FORMAT, so the value changes for every dataset at 7.0.0. Rows mined with different prompts, diff settings or models can be told apart. Both are null when the LLM was off.

7.8 Choosing one model: llm_compare.py

Before a big run, asks two models (default sonnet and haiku) about the same ~300 in-scope diffs, sent exactly as the miner sends them (same prompt, schema, quota pause), and reports per-flag agreement so one model is picked for the whole dataset. --dry-run shows the calls first; answers are saved as they arrive, so it resumes. A failed call is retried 3 times and not stored, a null association counts as outside, the sample key is always owner/name, and agreement() adds a macro mean.


8. The GitHub API subsystem (github_api.py)

Dependency-free (urllib). Auth via GITHUB_TOKEN (without one: 60 requests/hour and no labels).

8.1 What is called

CallUsed forVolume
GET /repos/{o}/{r}/pulls?state=closed&sort=created&direction=descPR selection1 per 100 PRs
GET .../pulls?state=openstale-open rowspages
GET .../pulls/{n}rebase commit count; git-primary enrichmentper PR where needed
GET .../issues/{n}/timelinemerge labels, at-open metadata, maintainer evidence1+ per PR (100/page, 2000 cap); cached 1 hour while the PR is open, 14 days once closed
GET .../pulls/{n}/reviews, .../commentsrevealed_path_scrutiny2 per PR (10 pages max each); None and not cached on a failed page or at the 1,000-item cap
POST /graphql (HeadRefForcePushedEvent)force-pushes for the at-open head1 per 25 PRs (separate rate budget)
GET .../commits/{sha}/check-runs, /statusCI verdict (merge commit; head commit of unmerged PRs)per PR
GET .../actions/runshistorical CI map≤ 400 pages once, cached
GET .../branches/{b}/protection/required_status_checksrequired checks1
GET /repos/{o}/{r}allowed merge methods1
GET /users/{login}account age1 per distinct author, cached
OSV POST /v1/query, GHSA GraphQLdependenciesper dependency change

8.2 Rate limits and errors (_get)

  • Secondary limit: Retry-After (seconds or an HTTP date), or the secondary-limit message in the body: sleep that long (capped at 300 s) and retry, up to 10 waits in a row on one request.
  • Primary limit (403/429, X-RateLimit-Remaining: 0): wait for the reset (capped at 3,600 s per wait) and retry, with a rate_limit event for the TUI countdown. One process-wide gate pauses every thread while any one of them waits; its state file .rate_wait lives under ci_cache_dir. The GraphQL block time is counted under a lock.
  • A 403 with no rate-limit signal is a real permission error: returned as 403, reported as unobserved, not retried.
  • 404 → unobserved. 5xx and network errors (including IncompleteRead, connection resets, cut bodies) back off min(60, 2^n) s and retry up to 8 times, then count as unobserved. Other 4xx are permanent for that request.
  • The CI history pre-pass uses a persistent getter that waits 60 s on a limit indefinitely, because it is long-running and must survive rate-limit windows.
  • CI-history cache. ci_cache/<owner>_<name>.json is versioned (v2, as {"v", "built", "map"}): a bare-dict file from an older version is discarded, the map expires after 7 days (CI_MAP_TTL_S) and is rebuilt when it was built before the newest PR (covers_until), and the file is written atomically.
  • API cache (api_cache.py): rows live at most 14 days; an open PR's timeline only 1 hour (OPEN_TIMELINE_TTL_S); expired rows are purged when the cache is opened. RIFFLE_NO_API_CACHE=1 turns it off.
  • Unobserved never becomes a value. A check-run, status or workflow-run page that failed or was truncated gives a null check_conclusion; the review read gives None (not cached) on any failed page or at the 1,000-item cap; labels_at and force-push reads give None on failure or truncation.
  • A failed list page flags the PR list as truncated.
  • Quotas are per account. With several PCs use a token from a different account on each.
  • Unobserved never becomes a value: any failed call leaves the dependent column null (or the list flagged incomplete).

9. The git subsystem (gitio.py)

Raw git -C <repo> via subprocess (minimal dependencies, explicit behaviour); every call is timed and a call over 30 s logs a WARN. Every call passes -c core.quotePath=false, and numstat, name-status and renames use -z, so any path is read as it is. A call has a timeout: RIFFLE_GIT_TIMEOUT_S (300), RIFFLE_GIT_BLAME_TIMEOUT_S (120) for a blame, RIFFLE_GIT_LONG_TIMEOUT_S (3600) for the whole-history log; a file read at a commit is capped at RIFFLE_MAX_BLOB_BYTES (4 MiB). A timeout is unobserved (null), never 0. A blame that times out is not retried for the same (repo, path) in that process.

  • History cache. _numstat_cache parses one streamed git log --all --numstat -M --format=... into {sha: (Commit, rows)} per repo per process. Streaming matters: the output is hundreds of MB on a large repo. iter_commits_with_files(repo, rev, before) lists reachable SHAs with a cheap rev-list --before and looks them up, so each PR costs a rev-list, not a log. SZZ's head_commits_with_files reuses it (no second full log). Falls back to a direct log if any SHA is missing from the cache.
  • Identity is the mailmap-canonical author (%aE, %aN) so one person's two emails merge; times are Unix epoch (%ct), which are immune to the corrupt timezone strings (+518:00) in some old histories.
  • Renames. With -z the old and new paths come as separate fields, so no brace or arrow parsing is needed (the old a/{b => c}/d and old => new forms are gone). A renamed file's base-side lookups use the old path.
  • Blame. git blame --line-porcelain with -w (ignore whitespace) and -M (detect moved lines), plus --ignore-revs-file when the repo ships .git-blame-ignore-revs (the path is made absolute: blame runs as git -C <repo>, which resolves a relative path against the repo; a relative path once made every blame fail silently). -C (cross-file copies) is off: very slow on large histories. Whole-file blame is cached (LRU 64) and shared between ownership (Source 2) and line features (Source 3). SZZ uses blame_lines: only the requested ranges (-L), padded by 20 lines so -M doesn't mis-credit a short moved block to the mover (±20 lines matched full-file blame on every line in the flask/itsdangerous fix history). A path that does not exist at the commit is an expected empty result (a PR that adds the file), not a failure. deleted_or_modified_lines_many still returns [] for a renamed file on its own; the rename is handled by the caller (mine_repo._szz_fix_bugs).
  • Blame failures are counted (calls, fails, absent) and warned once per process: they used to vanish into an empty result, silently emptying line features and SZZ labels. --audit reports blame coverage per repo.
  • Default branch names containing / are kept whole (release/1.x was truncated at the slash).
  • Fetching without refs. fetch_pr_heads (refs/pull/N/head) and fetch_shas fetch objects with --no-tags and no local refs, chunked at 200, splitting failed batches in halves (log n rounds per bad ref). Only a "ref missing" failure is final; any other failure is retried and then raises FetchError, so a failed head is unobserved, not gone. Nothing points at them, so git log --all never sees unmerged or force-pushed-over commits.
  • first_parent_chain (limit 1,000), count_before, commits_in_range, existing_commits (cat-file --batch-check), merge_base, commit_parents, landed_commit_index, rename_pairs (loose 30% similarity, includes merge commits: a superset of renames blame follows), deleted_or_modified_lines_many (one git diff per fix commit, not per file), changed_lines_by_file (-U0 hunks for concurrency).

10. SZZ fix tracing

Idea. For every fix commit, blame the lines it deletes or modifies at the fix's parent; the commit that last touched each such line is bug-introducing. The index is {bug_introducing_sha: [(fix_sha, fix_time), ...]}. A merged PR gets days_to_szz_fix if any of its commits is bug-introducing for a later fix that is not part of the PR itself.

Renames (SZZ_VERSION 3). A renamed file is blamed at the fix's parent under its old name, so a fix to a renamed file reaches the commit that wrote the line. The version bump makes every cached SZZ index and blame store rebuild.

Properties and limits. Noisy by nature (about 0.6 precision), so it is an additional, softer label kept out of label_bad. Blind spot: only deleted/modified lines are blamed, so a fix that only adds lines (a missing null-check) traces to nothing. Fix commits are recognised by subject (5.10), skipping root commits.

Making it cheap without changing any label (tested in test_szz_equivalence.py, checked on five repos):

  1. Window. Only fix commits dated between the oldest mined merge and the newest merge + szz_maturity_days (90) are blamed: a fix outside that range can't count for any mined PR. (pallets/click, 60 PRs: 20 fix commits instead of 672.) The cache records its window and is reused only when it covers the one needed.
  2. Relevant paths. Only files whose history (renames followed forward, found with a looser threshold so none are missed) the mined PRs' commits touched are blamed: no other blame can land on a commit any label reads. (Click: 9 fix commits; a 500-commit window ran 1.3 to 3.9× faster.)
  3. Ranged blame (-L with padding) per fix commit instead of whole-file blame; one git diff per fix commit instead of one per file; HEAD's commits from the history cache; tasks fed to the process pool a few at a time (workers × 4 in flight).
  4. Per (fix commit, file) blame store ci_cache/<owner>_<name>.szz.json.blames.jsonl: a crash resumes by blaming only missing pairs, and a later run with another window or PR selection on the same history reuses every pair. Its first line is a fingerprint (SZZ version, HEAD, blame flags, ignore-revs file content); a store made under other settings is never reused. It is kept after the run; --remine-repo clears it. A fresh clone with new commits changes HEAD, so the store starts over; keep --skip-existing/"Reuse clones" on.
  5. The final index is assembled in log order so keys and list order equal the serial algorithm's regardless of how workers finished.

Blame is about 90% of SZZ time, so the trims above are a few percent each. SZZ workers ignore SIGINT. If any blames failed, a warning says SZZ labels are undercounted; if at least 100 fix commits trace to zero bugs it warns that blame is almost certainly failing, not that the repo is clean.

Note: label_szz_bug and days_to_szz_fix are null when SZZ did not run (7.0.0; they used to be false), and the optional szz_window_closed says whether a false is mature.


11. PR snapshots

11.1 Why

The at-open row only shows PRs as they opened. Snapshots are rows describing a PR at a later moment while it was open, labelled with how it finally ended. They teach the model what "changes requested, author pushed a fix" or "7 days without a maintainer reply" mean, so re-scoring a live PR as events arrive gives a better answer. It is a two-step design because one part cannot wait.

11.2 Step 1, capture, during mining (snapshot_capture.py)

Clones are deleted after mining and force-pushed commits expire on GitHub, so the miner stores, for each in-scope PR (--llm-scope rule, --no-snapshots turns it off), under snapshots_raw/<owner>_<name>/ (--snapshot-dir, build_dataset --snapshot-root):

  • capture.jsonl: one record per PR with pr (trimmed PR object), events (trimmed timeline: bodies cut to 4,000 characters, users reduced to login/type), pushes (force-pushes, or null if unreadable), head_line (each moment the head changed while open), heads (up to --snapshot-max-heads, default 5, with the code features of fork-point...head), open_head, open_snapshot, and the optional top-level merge_outcome (assembly passes it as with_unmerged; an old capture without it defaults to True). A review record keeps its review id, and a dismissed review also carries dismissed_review {review_id, state} (optional; the dismissal fix needs the PR captured again).
  • pr_index.jsonl: each mined PR's merge-signal inputs (the sidecar), plus the lead-in PRs, so assembly can recompute repo-level signals as of any moment.
  • maintainers.json: the maintainer evidence.

Head over time (HeadLine): the head at moment t follows ADR 0010's rule: the newest commit, on the first-parent line that was live at t, committed at or before t. The live line starts at the head the first force-push after t replaced, or at the PR's final head. Commit time stands in for push time (GitHub doesn't record push time for ordinary pushes).

pick_heads: up to the cap of distinct heads, always the first (at open) and last, the rest spread evenly in time. The at-open head's features are the mined row's, not repeated. Each other head gets diff_stats, deterministic, scanners, deps_cve, api_contract computed against its own fork point (code_features); deps_cve is as of the time the head was first seen. A head that failed has error and null features.

Not recomputed per head: git-history and blame features (they describe the repo at the fork point and change little while a PR is open) and the LLM flags (the at-open ones), so snapshots add no LLM calls.

Cost is logged per repo: [snapshots] captured N PRs · H heads/PR · capture took S s, P% of PR mining time (~X% slower than without).

11.3 Step 2, assembly, offline (snapshots.py)

No GitHub calls and no clones, so the schedule can change without re-mining (about 15 s for 1,200 PRs). A batch with --snapshots runs it itself when it ends (it needs every repo's finished at-open rows); --no-assemble-snapshots skips it; a failure never fails the batch and prints the rebuild command:

bash
python snapshots.py --dataset dataset.parquet --raw-root snapshots_raw \
    --out snapshots.parquet --commit-features-out commit_features.parquet

Outputs: <out>_snapshots.parquet and <out>_commit_features.parquet. A PR captured twice (two folders for the same repo, e.g. from different machines) is built once from the folder that sorts last, so keys stay unique; within a folder, the last sidecar and capture per PR win, and History.at excludes the PR itself and same-number duplicates. Both tables are written through .tmp and a rename, even when empty (an empty table has the contract's columns). capture.jsonl is parsed once and torn lines are skipped.

Roles. Each timeline event gets a role: author (commits on the PR branch, or the PR author), bot, maintainer (association OWNER/MEMBER/COLLABORATOR or confirmed from history at that time), else other. Bots count for nothing.

Open periods. Snapshots exist only while the PR is open and ready: the clock starts at open + grace, or at ready-for-review for a PR opened as a draft, and restarts at a reopen; closed periods produce nothing; a PR that never left draft produces nothing (and needs no convert_to_draft event to say so); a close inside the 5-minute grace window leaves no period, and a reopen restarts the clock at max(reopen, start); a stale-open PR is treated as open for 60 days.

Moments (candidates):

KindTrigger names
Openopen
Eventsfirst_maintainer_reply, changes_requested (up to 3), approval (first), author_fix_push (the author's first push after a changes request), outcome_label (a label such as duplicate, wontfix, invalid, needs-info, stale, blocked, waiting-on-author, not planned, or an unwanted_labels match), marked_duplicate, back_to_draft
Clocksclock at 1, 3, 7, 14, 30 days after the start; idle at 7 and 14 days without human activity; no_maintainer_reply 7 days after the author's last push or comment with no maintainer reply in between

Clean-up: event triggers within 30 minutes make one snapshot at the last of them (the higher-priority trigger's name kept); a snapshot whose state matches the previous one is dropped unless it is a clock; snapshots within 6 hours with the same state keep the higher-priority one; nothing at or after close; at most 6 per PR, kept by priority (open 0, first reply 1, changes requested 2, approval 3, author fix push 4, outcome label / marked duplicate 5, back to draft 6, clock 7, idle / no maintainer reply 8).

Dismissed reviews keep their pre-dismissal state (CHANGES_REQUESTED, APPROVED) until the dismissal time. For an old capture that lacks the dismissal record, changes_requested_count and approvals_count are null (unknowable), not a guess.

Each row = every at-open dataset column, then overwritten with: the code features of the head at that moment, the merge signals recomputed as of that moment (prior-only), and the state so far: time since start and since the last human activity, first maintainer reply, review rounds, changes requested, approvals, commits and force-pushes since start (merges from base not counted), whether the author pushed after the last review, maintainer/author/other comment counts, outcome label present, marked duplicate, back in draft, reopen count, and change since the previous snapshot (new commits, new comments, LA+LD change). The *_since_prev deltas compare with the previous candidate moment, before the 6-per-PR cap, so a snapshot's values do not depend on which later moments were kept (causal). first_maintainer_reply_h is clamped to 0. max_sim_open_prs is windowed by bisect. Labels are the PR's final outcome.

Training helpers.

  • snapshot_weight = 1 / the PR's kept snapshot count, so each PR adds up to 1 and slow PRs with many snapshots do not dominate. It depends on the PR's whole history, so it is a weight, never a model input.
  • moment_bucket (open, <1d, 1-3d, 3-7d, 7-30d, >30d) for reporting metrics per moment.
  • near_decided: the last hour before close, or after the PR entered a merge queue / got auto-merge (added_to_merge_queue, auto_merge_enabled, auto_squash_enabled, auto_rebase_enabled). For filtering, never an input.
  • Split by repo or PR (a PR's snapshots must not straddle train and test).
  • head_found: open (the at-open head), exact (from the head line with force-push history known), approx (force-push history unreadable), capped (the true head wasn't captured because of the cap, or its capture errored, so the nearest earlier good captured one is used).

Leakage test (tests/test_snapshots.py::test_no_leakage): every snapshot is recomputed from data cut at its moment (later events, head changes, closes and first responses removed) and nothing may change.

Commit-features table: one row per captured head commit (repo, pr_number, head_sha, base_sha, first_seen, at_open, error + code features), every feature column present and null where not computed.


12. Scaling, resuming and repairing

12.1 Sharding across PCs

  • --shard K/N (TUI: "Number of PCs", "This PC #", "Show split" = ctrl+g). --shard-plan N prints the split. Equal repo counts (±1), balanced total commits (the [~N commits] tags drive clone, history walk and SZZ cost; with a fixed --max-prs the API/LLM cost per repo is equal). Untagged repos count as the median and a warning says the split may drift. The plan also lists each repo's PR count and a PR total per PC from the [N PRs] tags (display only; the split does not use them).
  • Each PC writes dataset_pcKofN.parquet, dataset_pcKofN_snapshots.parquet, dataset_pcKofN_commit_features.parquet, dataset_pcKofN.shard.json and a logs/run-<time>-pcKofN.log.
  • Every PC must use the same flags that change the data, notably the same --merge-outcome setting, or the merge model sees only some repos.
  • merge_shards.py --out dataset.parquet [--dir .] reads each PC's manifest and merges only if: every PC 1..N is present, all used the same split ID, all used the same miner version (--allow-version-mix overrides), no repo is on two PCs, no row repeats ((repo, pr_number), snapshot_id, (repo, pr_number, head_sha)), either all or no PCs have snapshot tables, and the snapshots still match the dataset (consistency.py: compares miner_version, mined_at, head_sha, outcome_3class per PR; a re-mine after assembly makes them stale). It also refuses a manifest that lists failed repos, or whose dataset repos differ from the manifest's repos (--allow-incomplete overrides; rows for a repo the manifest does not list are always refused), and compares repos case-insensitively in the cross-PC check. It writes <stem>_merged<ext>, _merged_snapshots, _merged_commit_features and <stem>_merged.merge.json (sources, counts). The three tables stay three (different grains), joined at training load.

12.2 What survives a stop

LayerWhereBehaviour on rerun
Finished reposrows in --outskipped (--remine redoes all)
Finished PRs of a stopped repo<out>.partial.jsonlskipped when their _ckpt stamp matches this run (an unstamped row from an older version is accepted); rows reloaded for the repo-wide pass
PR selectionci_cache/<owner>_<name>.selection.jsonpinned (same settings)
Setup stagesci_cache/<owner>_<name>.setup.jsonauthor index, concurrency and the SZZ relevant set reused when the PR list, HEAD, shallow flag and merge style are unchanged
SZZ blamesci_cache/<owner>_<name>.szz.json + .blames.jsonlonly missing (fix, file) pairs blamed
CI historyci_cache/<owner>_<name>.jsonreused until 7 days old or older than the newest PR
Snapshot capturesnapshots_raw/<repo>/*.jsonlcontinued when resuming (resume=True), restarted on a fresh run
LLM answers (re-mine)ci_cache/<owner>_<name>.llm_reuse.jsonreused for PRs with the same commits, then deleted
Clonecloned_repos/<owner>_<name>kept after a failed/stopped repo (--skip-existing reuses)

Stop (ctrl+c / TUI Stop) never loses finished PRs. After a stop caused by the LLM, rerun the same command.

12.3 Auditing and re-mining

  • --audit (TUI: Audit, ctrl+k) reports each repo's blame coverage, measured only on PRs touching files that already existed (AGE set; a PR that only adds files has no history to blame): 95–100% is healthy; <- check (50–95%) means look for [blame] warning in the logs; <- re-mine (under 50%) means blame was failing (before v1.6.0, repos that ship .git-blame-ignore-revs).
  • --remine-repo OWNER/NAME drops that repo's rows, SZZ cache and checkpoint and mines it again; its earlier LLM answers are kept and reused for PRs with the same commits, so the re-mine makes no new LLM calls for them. Only for repos in this PC's share.
  • --auto-remine (off by default) runs the audit at Start and re-mines flagged repos in this PC's share.
  • --live-tracking (off by default; a lead setting) uploads the run trace every 10 s and writes a worker heartbeat every 10 s, instead of every 15 minutes (section 13.1). It changes no row.
  • --skip-big N (TUI "Skip big repos", default 100000 commits: 14 repos in the 100-repo corpus: llvm, rust, ClickHouse, swift, tensorflow, ...); TUI ctrl+n skips the repo selected in the table (a running one asks for a second press and is stopped with its finished PRs kept).

12.4 Cost and resource notes

  • Memory: about 0.8 GB per worker plus 1.5 GB base. Disk: one clone at a time, 5 to 20 GB for big repos (doctor warns under 20 GB free).
  • Semgrep is the biggest per-PR cost (about 4 s per start). Merge-outcome mining roughly doubled run time and API calls in one pilot repo (timelines, one check call and a ref fetch per unmerged PR). LLM spend grows with the row count because unmerged rows run every stage.
  • A pilot (night-1, one full repo) records: seconds per row, LLM calls and $ at API prices per row, whether the plan pause triggered, head commits per captured PR, snapshots_raw/ size, health flags; scale the full-run estimate from those.

13. Observability and operator tools

13.1 Logs

  • Run log logs/run-<time>[-pcKofN].log: readable summary lines with prefixes [stage], [pr i/N], [fetch], [window], [ci], [ci-history], [szz], [merge-style], [open-snapshot], [timeline], [lead-in], [merge-outcome], [llm], [llm-quota], [snapshots], [health], [skip], [skips], [resume], [blame], [api], [warn], [error]. A repo, path or option that was refused (unsafe path, invalid option) is reported there too.
  • Run trace logs/run-*.trace.jsonl (runlog.py): one JSON line per event: {"t", "lvl", "stage", "msg", "repo", "pr", ...}. Levels DEBUG (every git/API/LLM call), INFO (run and repo stages with durations, one line per PR with its time by stage, quota waits, heartbeats; default), WARN (retries, rate-limit waits, slow calls, slow PRs, stalls), ERROR. WARN/ERROR also echo to stderr. --log-level / RIFFLE_LOG_LEVEL; RIFFLE_TRACE names the file (unset: nothing written, per-call cost is a timer and a dict update). Stage totals per PR are inclusive (quota_wait inside llm counts in both).
  • Thresholds: a call of 30 s or more warns (SLOW_CALL_S), a PR of 300 s or more warns with its top stages (SLOW_PR_S), no PR finishing for 300 s is a stall, a heartbeat every 120 s names what every worker is doing (WARN with the longest-running PRs on a stall). This is what turns an unexplained gap into a named stage.
  • One log, read in three places (ADR 0017): the run trace is the only log the miner writes. The store keeps the file and forwards each new line to Loki (the store's log search and Grafana read it there), and Airflow's riffle_miner DAG reads the same file to draw each stage. Nothing else logs. Three environment settings tune a watched run, and each falls back to its default when empty, not a number or below 1: RIFFLE_HEARTBEAT_S (the in-trace heartbeat, default 120), RIFFLE_SYNC_BEAT_S (status upload, default 180) and RIFFLE_SYNC_LOG_BEAT_S (log and trace upload, default 900); --live-tracking sets all three to 10 unless the environment already names one. It is a lead setting (in SHAPING, a toggle in the TUI's "Live tracking" box and in the store's admin settings form), so the lead turns it on for every PC of a split; it changes no row and is off by default. The TUI's separate "Also run Loki + Airflow on this PC" box starts the Docker stack on one PC (localstack.py) and is personal. Against a store that says append in /v1/whoami, a trace upload sends only the new complete lines (X-Append-Offset); a 409, an older store or a replaced file falls back to the whole-file PUT, so the live upload can never leave the store with a wrong file. The trace also carries a batch repos event (the repos of the run) and batch done is written before the last upload. A PC that runs its own Loki and Airflow keeps reporting to the team store: RIFFLE_MIRROR_URL and RIFFLE_MIRROR_TOKEN name an optional second store that gets a copy of every upload (its own state, a short time budget, and a failure there never touches the first store's sync). With --live-tracking the quick 10 s beat sends the log and trace only, the tables go on their own 15-minute cadence, a failing store is asked less and less often (up to every 5 minutes), and a Stop gives the mirror 10 s.
  • trace_report.py logs/run-*.trace.jsonl [--top 10]: stage times, slowest PRs with their stage split, quota waits, every WARN/ERROR grouped by message, longest quiet gaps.

13.2 Health check (health.py, proposal Part 3)

After each repo: [health] repo: N rows · merged % · outside % (unmerged %) · bots % · agents % · open_snapshot {...}, plus saved health/<owner>_<name>.json with stats (rows, merged_share, outside_share, outside_unmerged_share, bot_share, agent_share, close_reasons, label_bad_rate, ci_unobserved_share, open_snapshot counts, llm_rows, llm_missing_share) and flags:

FlagThreshold
too few rowsunder 50
almost no unmerged outside PRsunder 5% of outside PRs (PRs may land outside the merge button)
mostly botsover 50% of rows
LLM columns missing on in-scope rowsover 2%
CI unobserved on most merged rowsover 80% (check retention is about 90 days)

llm_rows counts every in-scope row, including those whose LLM status is empty or skipped. Those rows have nothing to judge, so they are neither missing nor in the denominator: llm_missing_share = missing / (llm_rows minus llm_nothing_to_judge). llm_nothing_to_judge is a new additive stats key.

13.3 Progress events and the UIs

  • progress.py: with RIFFLE_EVENTS=<file> every emit() appends a JSON line {"t", "ev", ...}; no-op otherwise. Events include batch_start/done, repo_queue/start/done/failed, stage, pr, szz, breaker (an LLM stop), rate_limit, llm_quota, assemble. The TUI never parses log text. The events file is read in binary and split on b'\n', and the writer uses newline='\n'; the control file is written atomically.
  • tui.py (Textual): configure the batch on the left (repos file, output, PR cap, workers, LLM/SZZ/CI toggles, window, cap, lead-in, domain, shard, naming, re-mine, skip-big), ctrl+r start, ctrl+x stop (SIGINT), ctrl+l clear log, ctrl+y copy log, ctrl+b QR code of the web view, ctrl+k audit, ctrl+g show split, ctrl+n skip repo, ctrl+t test LLM, ctrl+e test run (--test), ctrl+o view the test data, ctrl+q quit (a second ctrl+q force-quits; the first returns at once and waits for the final upload in a worker). Right side: per-repo progress, PR/min, ETA (needs at least 3 points over at least 10 s, and grows during a stall), LLM ok/fail and latency, LLM stop state, plan usage and reset countdown, live RAM/CPU of the whole process tree, names what killed a run (e.g. out of memory), counts down rate-limit waits, flags repos with no progress, saves every run's log under logs/. The form always starts from the defaults (no saved settings). Notifications are shown with markup off and dynamic text is escaped, so a repo name or log line cannot inject formatting; the QR screen scrolls and falls back to the URL alone when the terminal is too small; the subprocess stream limit is 16 MiB; the data viewer is keyed by row position; the progress bar resets on Start; the CPU count is psutil.cpu_count() or 1. Shows the snapshot files and "ASSEMBLING" then counts.
  • dashboard.py: read-only web view (standard library only) for other devices on the LAN; page polls /api/status every 2 s; URL carries a random key new on every launch; binds all interfaces (do not port-forward); RIFFLE_DASHBOARD=0 disables, RIFFLE_DASHBOARD_PORT sets the first port (8765), RIFFLE_DASHBOARD_BIND restricts the address (default: all interfaces). Each connection has a 15 s timeout. The page is plain HTTP and the key is in the URL, so anyone on the network who sees the URL can read the view (also stated in the module docstring). Log search, level filter, wrap/follow/pause/expand, copy and download, ? lists shortcuts. Big picture (b, or #big): full-screen dark page readable across a room: run state word (MINING, LLM PAUSED, STOPPED: ERROR, FINISHED, OFFLINE), percent, time, CPU and memory, PRs mined, time left, elapsed, PRs/min, repo and repos done, any alert, last log lines; keeps the screen awake; d dims it. The state word also reads ASSEMBLING while the snapshot tables are built at batch end (the label "Now mining" becomes "Now building"). A small Riffle icon and miner v<version> sit bottom left on every screen size. On phones (narrower than 900 px) the same page stacks into one column: status word, large percent, bar, two facts per row (PRs mined / time left, elapsed / PRs per minute), current repo, repos and CPU/memory, alert, log; on touch screens Dim and Exit are buttons, because there is no keyboard and a stray tap must not leave the page.
  • doctor.py (runs on every TUI launch; python doctor.py [--check], exit 1 on failure; RIFFLE_NO_DOCTOR=1 skips; NO_COLOR=1 for plain output): detects the OS family from /etc/os-release (Debian, RHEL, Arch, SUSE, Alpine families, matched on ID then ID_LIKE), macOS, Windows and WSL, plus the package manager on PATH (apt-get, dnf/yum, pacman, zypper, apk, brew, winget/scoop/choco). Checks, in sections: OS, Python (3.10+), environment (system / venv / PEP 668), git (Windows: core.longpaths), Python packages, GITHUB_TOKEN (verified against GitHub, the .env.example placeholder doesn't count), LLM setup (claude CLI or API key + SDK), the three API-contract tools one row each (required while API contract diff is on: a missing one is a failure the TUI pauses on, but does not block launch) (oasdiff OpenAPI, buf protobuf, graphql-inspector GraphQL), disk (warn under 20 GB) and RAM (warn under 8 GB; psutil, else /proc/meminfo, sysctl or the Win32 API). Auto-update is opt-in. At launch doctor.auto_update fetches origin/main and, by default, only reports [update] main has a newer miner (X, you have Y). ... and returns available. With RIFFLE_AUTO_UPDATE=1 it fast-forwards a clean checkout on main and restarts once; RIFFLE_NO_AUTO_UPDATE=1 turns the check off. Fixes: pip-installs missing packages, asks for a token and saves it to .env (mode 600 where supported, gitignored), and offers the OS-specific install command for git, Node, Claude Code and the API tools (install_plan), never running one without a yes; with no known installer it prints a manual hint. Non-interactive runs only report, in plain text.
  • deps.py: on an externally managed Python (PEP 668, Ubuntu/Debian 23.04+) packages go into a private repo_miner/.venv created on first need; later launches re-run inside it automatically; no --break-system-packages. If python3-venv is missing the doctor offers the apt command. RIFFLE_NO_INSTALL=1, RIFFLE_NO_VENV=1. The pip install specs are bounded: textual>=8.0,<9, pandas>=2.0,<4, pyarrow>=14.0,<26, lizard>=1.17,<2, psutil>=5.9,<8, segno>=1.6,<2, semgrep>=1.0,<2.

13.3.1 Pre-run tools

  • scout.py ranks candidate repos without cloning: outside-contributor share, unmerged share among outside PRs, bots, agents, from each repo's 100 latest closed PRs (--agent-search adds agent PR counts over 12 months via search, 30 calls a minute). Caveat: GitHub reports org members with private membership as CONTRIBUTOR, so company-run repos look more "outside" than they are.
  • audit_export.py writes a stratified ~600-PR CSV (equal per outcome class: merged clean, merged rework, duplicate, unwanted, ci_technical, abandoned, other; agent PRs over-weighted to half of each class) with links and the miner's labels, and blank human_outcome, human_reason, notes columns for the human check of label precision (label_unwanted and label_duplicate especially) before training on them.
  • llm_compare.py: 7.8.

14. Schema reference: every column

Three tables, three grains, three contracts:

TableFileGrainContractColumns
dataset<out>.parquetone row per closed PRmined_dataset.schema.json v1.3210
snapshots<out>_snapshots.parquetone row per PR momentmined_snapshot.schema.json v1.4241 (the 210 plus 31)
commit features<out>_commit_features.parquetone row per head commit a PR had while openmined_commit_features.schema.json v1.153

14.1 Conventions that apply to every column

  • Every column is present on every row. A stage that is off, a tool that is missing, or a signal that could not be observed gives null, never a missing key. additionalProperties is false: a column the contract does not list fails CI.
  • null means "not observed or not enough history". 0 means "measured, found none". Never coerce one to the other (the repo-wide contract rule).
  • Types below are the contract's types. "or null" is written only where the column is not in the schema's required list's non-nullable set; the non-null columns are repo, pr_number, base_sha, head_sha, mined_at, miner_version, miner_commit.
  • Columns that are outcomes, ids or metadata are never model inputs. The contract marks them x-riffle-non-input (merge_via, open_snapshot, the five provenance columns, llm_diff_truncated, szz_window_closed, ...); the columns computed from the diff range carry x-riffle-diff-derived (125 of them), and frames.py reads both keywords instead of guessing from description prefixes: repo, pr_number, base_sha, head_sha, author_email, created_at, closed_at, merged_at, mined_at, observed_through, every label_*, days_to_*, close_reason*, outcome_3class, nowhere_reason, ci_* outcome details, windows, provenance, domain, and the post-open counts. frames.py derives the input list from the contract: x-riffle-non-input, descriptions that say "never a model input", ids, snapshot bookkeeping, merge_via, open_snapshot and the five provenance columns. (The old rule that dropped every column whose description began "Row metadata" also dropped about 25 real features.)
  • Descriptions in the contract name the module that computes each column; metadata originating in mine_repo.py is described as row metadata or Source 4c.
  • Schema versions are x-riffle-version; every schema change is breaking in the contracts sense: add fields as optional, backfill, then make required, bump every consumer in the same PR, regenerate contracts/REFERENCE.md, add a row to docs/contracts.md.

14.2 Dataset columns, by group

Notation: num = number, int = integer, bool, str, enum{...}. Where the contract leaves a type "any" (null in every sampled row when the contract was written) the column is written num*: the miner emits a number or null.

A. Row identity, provenance and domain

ColumnTypeMeaning
repostrowner/name
pr_numberintPR number. (repo, pr_number) is the row key.
base_shastrFork point the at-open diff is taken from (the final base when open_snapshot = final)
head_shastrThe PR's head at open (the final head when open_snapshot = final)
author_emailstr or nullMailmap-canonical, lower-case git author email of the PR's own head commit (not the maintainer who merged)
mined_atstrWhen the row was extracted (UTC). Maturity is computed from it.
miner_versionstrversion.py that wrote the row
miner_commitstrThe miner's own git commit (12 chars), -dirty if its files had uncommitted changes, unknown outside a checkout
llm_modelstr or nullThe resolved model id of the call (the CLI's modelUsage or the API response's model), else the alias; null if the LLM was off
llm_prompt_versionstr or null12-hex hash of the prompts, schemas, diff-slimming rules, redaction patterns, diff settings, backend and transcript format (7.2); null if the LLM was off
domainstr or nullDataset domain from the repos file's [domain: NAME] tag (ADR 0011); null if untagged
merge_viaenum{github, landed_commit} or nullHow the PR landed; null for unmerged rows
open_snapshotenum{exact, recovered, approx, final} or nullHow the at-open diff was found (4.4). final rows' diff features may leak: drop or mask them.

B. PR metadata and timing (Source 4c)

ColumnTypeMeaning
created_atstr or nullPR creation time. Null with --no-merge-outcome.
closed_atstr or nullClose time; null for stale-open rows, and with --no-merge-outcome
merged_atstr or nullMerge time; null for unmerged rows
observed_throughstr or nullHow far forward outcomes could be seen in the clone (6.1); null on unmerged rows
author_associationenum{OWNER, MEMBER, COLLABORATOR, CONTRIBUTOR, FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, MANNEQUIN, NONE} or nullGitHub's current association for the submitter (no git equivalent); privacy hides org members as CONTRIBUTOR
author_confirmed_maintainerbool or nullThe PR author had merged or closed someone else's PR before this one opened (5.12); null when no timelines were read
author_is_botbool or nullAutomation only (Dependabot, Renovate, github-actions, other [bot]); never true for an agent
ai_agent_authoredbool or nullAI coding agent (account, branch prefix, commit email/trailer, body boilerplate)
author_account_age_daysnum or nullOpen time minus the account's creation time
title_lennum or nullTitle length at open (rebuilt from renamed events)
body_lennum or nullBody length at mining time
has_milestonebool or nullA milestone was set at open (rebuilt from milestoned/demilestoned)
head_from_forkbool or nullHead repo differs from base repo; null when the head repo is gone (a deleted fork: unknown, not "same repo")
targets_default_branchbool or nullThe PR's base ref is the repo's default branch
declared_hotfixbool or nullA hotfix/rollback label was on the PR at open (+120 s); null when disabled or the timeline was unreadable
head_branch_agent_prefixbool or nullHead branch starts with an agent prefix (codex/, copilot/, ...); null if the head ref is unknown
FIXbool or nullThe PR (title at open + head commit subject) names a bug fix as a whole word
AGEnum or nullMean days since last change across changed files that have history
LTnum or nullTotal lines of the changed files at the base (Kamei "lines total")
la_norm, ld_normnum or nullLA/LT, LD/LT (0.0 when LT is 0)
base_behind_bynum or nullBase-branch commits the fork point was behind at open; null when open_snapshot = final
concurrent_open_prs_same_filesnum or nullOther PRs open at this PR's opening that touch a common file
concurrent_open_prs_same_hunksnum or null... that also touch overlapping base-side lines
author_prs_merged_reponum or nullPRs this git author had merged in this repo before this PR opened; 0 = none, null = no PR list
n_topk_hotspots_touchednum or nullChanged files among the repo's top-20 hotspots
touches_topk_hotspotbool or nulln_topk_hotspots_touched > 0
min_hotspot_rank_touchednum or nullBest (lowest) hotspot rank among changed files; null if none ranked

C. Diff size, spread and shape (Source 4a, Kamei metrics)

ColumnTypeMeaning
LAnum or nullLines added (non-generated files)
LDnum or nullLines deleted
NFnum or nullFiles changed
NDnum or nullDistinct directories
NSnum or nullDistinct subsystems (5.1)
Entropynum or nullNormalised change entropy 0 to 1
frac_generated_linesnum or nullShare of all changed lines in generated/vendored files
frac_test_linesnum or nullShare of kept lines in test files
lang_mixnum or nullDistinct recognised languages among kept files
n_files_added, n_files_removed, n_files_renamednum or nullFrom git diff --name-status -M
patch_missing_filesnum or nullFiles with no textual diff (binary / too large; git's -\t-)
docs_onlybool or nullEvery changed file is a doc
test_onlybool or nullEvery changed file is a test

D. High-risk path flags (Source 4b)

All bool or null, true if any non-generated changed file matches:

ColumnMatches
touches_migrationmigrations?/, *.sql, db/migrate/
touches_lockfilepackage-lock.json, yarn.lock, pnpm-lock.yaml, poetry.lock, Pipfile.lock, go.sum, Cargo.lock, Gemfile.lock, composer.lock, pubspec.lock, mix.lock
touches_dep_manifestpackage.json, requirements*.txt, pyproject.toml, go.mod, pom.xml, build.gradle, Cargo.toml, Gemfile, composer.json, pubspec.yaml, mix.exs, *.csproj/.vbproj/.fsproj
touches_ci_config.github/workflows/, .gitlab-ci.yml, Jenkinsfile, .circleci/
touches_infra*.tf, helm/, k8s/, Dockerfile*, docker-compose*.yml
touches_auth_patha token-boundaried auth, authn, authz, login, logout, signin, signup, session, oauth, jwt, permission, rbac, credential, password, secret (so author/authority don't match)
touches_codeowners_or_riffle_ymlCODEOWNERS, riffle.yml/.yaml

E. Repository history at the base (Source 2)

Per-file features collapsed over the changed files (5.2.1). Values are about the base commit.

ColumnTypeMeaning
max_file_revert_density, mean_file_revert_densitynum or nullShare of a file's prior commits that were reverts (max / mean over files)
max_revert_decay_densitynum or nullMax over files of the half-life-decayed revert sum (90 d)
min_days_since_last_revertnum*Fewest days since any changed file's last revert (null if none)
max_file_fix_density, mean_file_fix_densitynum or nullSame for fix commits
max_fix_decay_densitynum or nullHalf-life-decayed fix sum
min_days_since_last_fixnum or nullFewest days since any changed file's last fix
max_file_churn_90d, max_file_churn_365dnum or nullMax prior commits touching a changed file in 90 / 365 days
max_recent_merge_burst_7dnum or nullMax prior commits to a changed file in the last 7 days
max_hotfix_count_365d, sum_hotfix_count_365dnum or nullHotfix-worded commits in the last year (max / sum over files)
max_test_cochange_rationum or nullMax share of a file's prior commits that also changed a test
max_file_post_merge_ci_fail_ratenum*Max over files of the share of commits whose CI failed (needs the CI map)
mean_file_age_daysnum or nullMean file age (days since first commit)
max_NUCnum or nullMax prior commits to a changed file (Kamei NUC)
max_NDEVnum or nullMax distinct prior authors of a changed file (Kamei NDEV)
max_file_complexity_ccn, sum_file_complexity_ccnnum or nullMax / sum over files of the file's max function cyclomatic complexity at base
max_file_hotspot_scorenum or nullchurn_365d × CCN
min_file_hotspot_ranknum or nullBest repo-wide hotspot rank among changed files
max_file_top_author_sharenum or nullLargest single-author share of a file's lines (blame)
max_file_minor_contributor_countnum or nullAuthors owning under 5% of a file's lines
max_abandoned_lines_sharenum or nullShare of lines owned by authors inactive over 180 days
cochange_degreenum or nullMax strong co-change partners (≥3 shared prior commits) of any changed file
cochange_partners_missingnum or nullMax strong partners of a changed file that are not in this PR
expected_test_cochange_missingbool or nullChanges a source file that usually ships with a test change, but includes no test
EXPnum or nullAuthor's prior commits (Kamei EXP)
REXPnum or nullRecency-weighted prior commits (Kamei REXP)
SEXPnum or nullAuthor's prior commits in the touched subsystems (Kamei SEXP)
author_revert_ratenum or nullSmoothed share of the author's commits that are reverts
author_file_familiaritynum or nullShare of touched paths the author touched before
author_days_since_last_changenum or nullDays since the author's last commit before base
author_post_merge_ci_fail_ratenum*Smoothed post-merge CI failure rate of the author's commits; null with no CI history
repo_revert_base_ratenum or nullReverts / all commits before base (merge commits excluded)
pr_size_pctile_repo, nf_pctile_repo, entropy_pctile_reponum or nullWhere this PR's size / files / entropy sits in the repo's per-commit distribution (fraction strictly smaller)

F. Line-level blame (Source 3)

ColumnTypeMeaning
min_mod_line_age_days_minnum or nullYoungest modified line's age across files
median_mod_line_age_days_mediannum or nullMedian over files of the median modified-line age
max_mod_lines_distinct_authorsnum or nullMost distinct authors behind a file's modified lines
mean_mod_lines_self_authored_sharenum or nullMean share of modified lines the PR author wrote
max_mod_lines_prior_fix_sharenum or nullMax share of modified lines whose last commit was a fix
max_hunk_fix_proximity, mean_hunk_fix_proximitynum or nullDecayed count of fix-owned lines within ±5 lines of a modification (max / mean)

G. Scanners (Source 4d)

ColumnTypeMeaning
ccn_deltanum or nullSum of CCN changes of modified functions (lizard), deleted functions included; null on a lizard parse error
max_ccn_changed_funcnum or nullHighest CCN among modified functions
n_funcs_modifiednum or nullFunctions whose CCN changed (including new ones)
semgrep_findings_newnum or nullSemgrep findings new at head vs base (multiset, no line numbers); null when the scan did not run or failed (all three semgrep columns)
semgrep_high_conf_security_newnum or nullNew findings with category security and confidence HIGH
semgrep_cwe_top25_newbool or nullA new finding's CWE is in the Top-25 set

H. Dependencies and CVEs (Source 4e)

ColumnTypeMeaning
n_deps_added, n_deps_upgraded, n_deps_downgradednum or nullDependencies added / upgraded / downgraded (semver comparison); null when a touched manifest is unparsed or unreadable
n_major_bumpsnum or nullUpgrades whose major version changed
osv_vulns_introducednum or nullVulnerability IDs at head not at base (OSV), as of the PR's open time; null = no dependency change, unobserved, an unresolved version or an undatable record
osv_vulns_fixednum or nullIDs at base not at head
vuln_has_fix_availablebool or nullFalse when no introduced record has a fix; null if any has one (OSV does not date a fix's release)
ghsa_max_severity_introducedenum{LOW, MODERATE, HIGH, CRITICAL} or nullWorst severity of introduced records
ghsa_malware_depbool or nullAn introduced record is malware (MAL-)
osv_ghsa_disagreementbool*OSV and GHSA disagree on a newly present version; null with no token or no usable GHSA answer

I. Deterministic checks and API contract (Source 5)

ColumnTypeMeaning
hardcoded_secret_addednum or nullAdded source lines matching a secret pattern
new_env_or_config_keynum or nullAdded lines reading env/config keys
observability_deltanum or nullRemoved minus added logging/metrics lines
bound_or_comparison_editnum or nullAdded source lines with comparison / ±1 edits
unused_import_or_dead_code_addednum or nullRough count of added imports never used
dependency_major_bump_countnum or nullRegex-detected major version bumps in manifest/lockfile lines
migration_reversibilityenum{destructive, reversible} or nullMigration content check
api_contract_changeenum{additive, breaking} or nullSpec-diff verdict (5.7); null when any changed spec is unobserved, unless another is breaking

J. LLM flags (Source 6), diff only; null when out of scope, trivial or LLM off

llm_touches_authn_authz (bool), authz_check_change, changes_public_api_signature (bool), error_handling_change, exception_swallowed (bool), db_schema_or_query_change, transaction_boundary_change, input_validation_change, config_or_feature_flag_change, feature_flag_guarding, security_sensitive_ops (bool), resource_lifecycle_change, null_handling_change, numeric_money_precision (bool), time_handling_change (bool), retry_timeout_idempotency_change (bool), refactor_vs_behavior, change_intent, unrelated_change_share (int 0 to 3), tests_added_for_behavior_change, test_code_alignment, satd_added (int), semantic_risk_1to5 (int 1 to 5). Values for each are listed in 7.2. These 23 columns together with llm_model, llm_prompt_version and the optional llm_diff_truncated (bool or null; true when the diff cap cut code the model would have read; never an input) are the LLM surface.

K. Merge-model signals: per PR

ColumnTypeMeaning
has_linked_issuebool or nullBody links an issue (fixes #N, URL)
body_headingsnum or nullMarkdown headings in the body
body_checklist_itemsnum or null- [ ]/- [x] items
body_has_test_planbool or nullA test-plan style heading
agent_boilerplatebool or nullAn agent watermark in the body
agent_nameenum{claude_code, codex, copilot, cursor, devin, jules, sweep} or nullWhich agent opened it, null for none
has_contributingbool or nullA contributing guide at the base commit
has_agents_mdbool or nullAn agent instructions file at the base commit

L. Merge-model signals: repo-level, prior PRs only

ColumnTypeMeaning
repo_outside_merge_ratenum or nullShare of outsiders' prior closed PRs that merged (needs ≥5 and unmerged rows)
repo_outside_closed_countnum or nullHow many such prior PRs (always set; the count behind the rate)
agent_recent_merge_ratenum*Same for the same agent (agent rows with ≥5 prior)
agent_recent_closed_countnum or nullPrior closed PRs by the same agent (agent rows only)
repo_median_first_response_hoursnum*Median hours to the first human response on prior PRs (≥5 needed)
repo_agent_pr_load_30dnum or nullAgent PRs opened in the previous 30 days
max_sim_open_prsnum or nullHighest text similarity to a PR open at open time (0.0 if none open)
max_sim_recent_mergednum or nullHighest similarity to PRs merged in the previous 90 days (null if none)

M. Repo-local mined features (mined_features.schema.json)

similar_pr_bad_rate (num*), similar_pr_count (num; 0 when none), revealed_path_scrutiny (num*), path_failure_mode_revert, path_failure_mode_ci, path_failure_mode_hotfix (num*), path_detection_lag_days (num*), path_size_pctile (num or null), release_cycle_position (num or null). Definitions in 5.11.

N. Breakage labels and observability (ADR 0002): outcomes; null on unmerged rows

days_to_revert (num*), days_to_hotfix (num*), days_to_szz_fix (num*, null when SZZ did not run), label_revert (bool), label_ci_fail (bool*, tri-state), label_ci_fail_optional (bool*), label_hotfix (bool), label_szz_bug (bool, null when SZZ did not run), label_bad (bool), ci_observed (bool), revert_window_closed (bool), hotfix_window_closed (bool), szz_window_closed (bool, optional), label_bad_observed (bool), ci_fail_flaky_only (bool). Definitions in 6.1 and 6.4.

O. Merge-outcome labels and targets (ADR 0009, proposal Part 4): outcomes, never inputs

ColumnTypeMeaning
label_mergedbool or nullMerged. Null only when merge-outcome mining is off.
label_reworkbool or nullMerged rows: human change request, or commits/force-pushes after open
post_open_commitsnum or nullCommits after open + grace, up to merge (merged rows; base-branch merges excluded)
post_open_force_pushesnum or nullForce-pushes after open + grace, up to merge
changes_requested_reviewsnum or nullHuman non-author change requests, up to merge
label_duplicate, label_unwanted, label_abandoned, label_ci_technicalbool or nullUnmerged rows; rules in 6.3
close_reasonenum{duplicate, unwanted, ci_technical, abandoned, other} or nullWhy an unmerged PR closed
close_reason_sourceenum{llm, rules} or nullWho decided it
outcome_3classenum{lands_clean, lands_after_rework, goes_nowhere} or nullThe three-way target
nowhere_reasonenum{abandoned, duplicate, unwanted, ci_technical, unknown} or nullWhy a goes_nowhere PR went nowhere

With --no-merge-outcome the 13 merge-outcome columns, created_at and closed_at are null on every row (since v2.3.0 they are written as null, not left out).

14.3 Snapshot table: the 31 extra columns

The snapshot row is every dataset column (so the three at-open groups carry through), with three groups taken as of snapshot_at: the code features of the head then (groups C, D, G, H, I's non-LLM columns and api_contract_change), the merge signals (group L), and the state columns below. Git-history, blame, LLM and PR-metadata columns stay as at open. Labels are the PR's final outcome.

Identity and filtering (9):

ColumnTypeMeaning
snapshot_idstrHash of repo, PR number, trigger and time; stable across re-mining
snapshot_atstrThe moment (UTC)
snapshot_indexintOrder among the PR's kept snapshots
triggerenumapproval, author_fix_push, back_to_draft, changes_requested, clock, first_maintainer_reply, idle, marked_duplicate, no_maintainer_reply, open, outcome_label
trigger_roleenum{author, maintainer, other, clock}Who caused it
snapshot_head_shastr or nullHead at the moment (or nearest earlier captured, see head_found)
snapshot_base_shastr or nullFork point the head's code features came from
head_foundenum{open, exact, approx, capped}How the head was found (11.3)
near_decidedboolLast hour before close, or after merge queue / auto-merge. Filter; never an input.

State (20, "Snapshot state"):

ColumnTypeMeaning
time_since_start_hnum ≥ 0Hours since the clock started (open, ready-for-review, last reopen)
time_since_last_human_activity_hnum ≥ 0Hours since the last non-bot event
maintainer_repliedboolA maintainer has commented or reviewed
first_maintainer_reply_hnum or nullHours from start to the first maintainer reply
review_roundsintReviews by humans other than the author
changes_requested_countint or nullChanges-requested reviews so far, as the review stood at the moment (a dismissed review counts until its dismissal); null when unknowable (an old capture with no dismissal record)
approvals_countint or nullApproving reviews so far, same rule
commits_since_startintCommits pushed after the start (base merges excluded)
force_pushes_since_startintForce-pushes after the start
author_pushed_after_last_reviewboolThe author pushed after the latest review
maintainer_comment_count, author_comment_count, other_comment_countintComments by role
has_outcome_labelboolAn outcome label (duplicate, wontfix, stale, needs-info, ...) is on the PR now
marked_duplicate_so_farboolSomeone marked it duplicate
is_draft_nowboolBack in draft
reopen_countintReopens so far
new_commits_since_prevint or nullCommits since the previous candidate moment, before the 6-per-PR cap (null on the first)
new_comments_since_prevint or nullHuman comments since the previous candidate moment
diff_size_change_since_prevnum or nullChange in LA + LD since the previous candidate moment

Training (2): snapshot_weight (num in (0, 1]; weight, never input) and moment_bucket (enum{open, <1d, 1-3d, 3-7d, 7-30d, >30d}).

14.4 Commit-features table

repo, pr_number, head_sha, base_sha, first_seen (when this commit became the PR's head; commit time stands in for push time), at_open (the at-open head: its features are the dataset row's and are not repeated here, so they are null), error (why features could not be computed), plus these 46 feature columns: Entropy, LA, LD, ND, NF, NS, api_contract_change, bound_or_comparison_edit, ccn_delta, dependency_major_bump_count, docs_only, frac_generated_lines, frac_test_lines, ghsa_malware_dep, ghsa_max_severity_introduced, hardcoded_secret_added, lang_mix, max_ccn_changed_func, migration_reversibility, n_deps_added, n_deps_downgraded, n_deps_upgraded, n_files_added, n_files_removed, n_files_renamed, n_funcs_modified, n_major_bumps, new_env_or_config_key, observability_delta, osv_ghsa_disagreement, osv_vulns_fixed, osv_vulns_introduced, patch_missing_files, semgrep_cwe_top25_new, semgrep_findings_new, semgrep_high_conf_security_new, test_only, touches_auth_path, touches_ci_config, touches_codeowners_or_riffle_yml, touches_dep_manifest, touches_infra, touches_lockfile, touches_migration, unused_import_or_dead_code_added, vuln_has_fix_available.

These are exactly frames.DIFF_FEATURES: the diff-derived code features that frames.py nulls on rows whose at-open diff was not recoverable.

14.5 Private keys and intermediate record shapes

Carried on a row until written (never in the dataset): __llm__ ({"status", "raw", "secs", ...}), __capture__ (the capture record), _repo_local ({"opened_at", "files": {path: lines at final}, "open_files": {path: lines at open}, "reviews": {"comments": [...], "changes_requested": [...], "rounds": [...]} | null}), __merge_signals__ ({"number", "opened", "closed", "merged", "outsider", "agent", "first_response", "tokens"}), __ci__ ({"at", "failing": [...], "rerun_passes": [...]}).

PR object extras the miner sets: _unmerged, _stale_open, _events, _open ({"base", "head", "kind", "behind"}), _pushes, _git_base, _author_maintainer, merge_via.

14.6 Not this miner: mined_history.schema.json

The Go history miner (services/miner, ADR 0004) writes a manifest and six Parquet tables (commits, file_changes, pull_requests, reviews, check_runs, tags) as raw history for serving; it is a different component, not built yet, and shares only the idea. The Python repo miner here produces the training rows.


15. Files the miner reads and writes

15.1 Inputs

FilePurpose
repos.txtThe 100-repo corpus (v1 #1-50, v2 #51-100) from site/content/docs/dataset-plan.md, each with a # Train/Val/Test [~N commits] comment; the split labels are for reference, the miner ignores comments except the two tags.
repos_web_ts.txtThe first domain (ADR 0011): JavaScript/TypeScript web repos tagged [domain: web-ts], with [N PRs], [~N commits] (GitHub API, 2026-10-19) and [monorepo] on app monorepos; the header records what was left out and why (e.g. TypeScript is now Go, pnpm is Rust, nodejs/node uses a commit queue, angular has its own merge tooling, electron is mostly backport bots, astro is two-thirds bot PRs, sentry/posthog have few outside PRs). Hold out 6 to 8 repos for validation and 6 to 8 for test, at least one monorepo in each.
repos_vite.txtA one-repo list for the night-1 pilot (vitejs/vite).
.envAuto-loaded at import (GITHUB_TOKEN, LLM_*); real shell variables win over .env; looked up in the package dir, then the current directory. Gitignored. .env.example is the template.

15.2 Outputs and caches (all under the working directory unless noted)

PathWritten byContentsLifetime
dataset.parquet (.jsonl, .csv)mine_repodataset rows, appended and de-duplicatedkept
dataset.parquet.partial.jsonlmine_repocheckpoint: one row per finished PR (sidecars and a _ckpt stamp included)removed after a clean write
dataset.parquet.recover-<time>.jsonlmine_repothis run's rows when an append could not read the existing filekept; merge by hand
dataset_snapshots.parquet, dataset_commit_features.parquetsnapshots.pysnapshot and commit tableskept
dataset_pcKofN.shard.jsonbuild_datasetshard manifestkept
dataset_merged*.parquet, dataset_merged.merge.jsonmerge_shardsmerged tables and the merge reportkept
ci_cache/<owner>_<name>.jsonCI history{"v", "built", "map"} where map is `{sha: failuresuccess}`; written atomically
ci_cache/.rate_waitgithub_apithe shared rate-limit gatekept
.sync_state.json, .sync_state.json.locksyncwhat was last sent, per store URL, and its lockkept
ci_cache/<owner>_<name>.szz.jsonSZZ{"fingerprint", "window", "index"}kept
ci_cache/<owner>_<name>.szz.json.blames.jsonlSZZfirst line a fingerprint, then {"sha", "paths": {path: [bug shas]}} per fix commitkept (cleared by --remine-repo)
ci_cache/<owner>_<name>.selection.jsonmine_repopinned PR selection and maintainer evidenceremoved after a clean write
ci_cache/<owner>_<name>.setup.jsonsetup_cache{"v", "sections": {name: {"key", "data"}}} for author_index, concurrency, szz_relevant; written atomicallyremoved after a clean write and by --remine-repo
ci_cache/<owner>_<name>.llm_reuse.jsonre-mine[{"base_sha", "head_sha", "flags"}]removed after the re-mine
snapshots_raw/<owner>_<name>/capture.jsonl, pr_index.jsonl, maintainers.jsoncaptureraw PR, timeline, force-pushes, head line, heads with features; merge-signal sidecars; maintainer evidencekept (needed to assemble)
health/<owner>_<name>.jsonhealthstats and flagskept
logs/run-*.log, logs/run-*.trace.jsonlTUI / build_datasetrun log and structured tracekept
cloned_repos/<owner>_<name>build_datasetfull clonedeleted after a clean mine unless --keep-clones
test_output/--testtest dataset and parquetrecreated each test run
.venv/deps.pyprivate virtualenv on PEP 668 systemskept

.gitignore covers .env, out/, clones/, cloned_repos/, ci_cache/, data/, *.parquet, *.jsonl, test_output/, logs/, .venv/, snapshots_raw/, snapshots*.parquet, commit_features*.parquet, health/. Raw data files are DVC-tracked in the wider project, not committed.


16. Command-line reference

Defaults shown are the code's. Every stage flag has a --no- form (argparse.BooleanOptionalAction).

16.1 mine_repo.py (one repo)

FlagDefaultMeaning
--repo PATHrequiredlocal clone (full history)
--owner, --namefrom originGitHub owner/repo; auto-detected from the remote URL
--out FILE<owner>_<name>.parquet.parquet, .jsonl or .csv
--appendoffmerge into --out, de-duplicate by (repo, pr_number) keeping the newest
--max-prs Nallcap (newest merged PRs, or the newest contiguous block in window mode)
--min-age-days N90skip PRs whose outcome is younger (0 = off)
--window-from, --window-tononedate window (YYYY-MM-DD); must be given together
--lead-in-days N180history lead-in before the block (0 = off)
--cap-on {all,training}allwhat --max-prs counts; training needs a window
--domain NAMEnonewritten on every row
--llm / --no-llmonLLM flags and the close-reason classifier; the run stops if no backend is usable, so datasets never silently lack the columns
--llm-model MLLM_MODEL or sonnetmodel
--llm-testone call on a sample diff, report, exit
--llm-scope {training,all}trainingrows that get LLM calls
--llm-sample-pct N5% of other rows still sent
--llm-concurrency N4simultaneous LLM calls
--llm-max-diff-chars N25000cap on the diff sent
--close-reason-llm {other_only,always}other_onlywhen the classifier is asked
--llm-quota-wait / --no-llm-quota-waitonwait for the plan to reset
--full-repooffmine every closed, non-draft PR the repo has (4.3)
--szz / --no-szzonSZZ label
--ci-history / --no-ci-historyonhistorical CI-fail-rate features
--declared-labels / --no-declared-labelsondeclared_hotfix
--merge-outcome / --no-merge-outcomeonunmerged and stale-open rows plus merge labels
--max-unmerged Nallcap unmerged rows per repo, newest first
--pr-discovery {api,git}apihow merged PRs are enumerated
--no-review-historyoffskip review events; revealed_path_scrutiny null
--no-scannersoffskip lizard and semgrep
--no-api-contractoffskip spec diffing
--no-networkoffgit-only: no API, no OSV (pair with --pr-discovery git)
--workers NRIFFLE_WORKERS or min(6, CPUs) fitted to RAMPRs in parallel
--snapshots / --no-snapshotsoncapture for snapshots
--snapshot-dir Dsnapshots_raw/<owner>_<name>capture folder
--snapshot-max-heads N5head commits per PR with code features
--health-dir Dhealthper-repo health JSON

Exit codes: 0 ok, 3 LLM could not answer (output not written, rerun to resume), 130 stopped. Without --no-network and with --pr-discovery api and no token the miner warns and returns no rows.

16.2 build_dataset.py (many repos)

Pass-through of the mine flags above (--max-prs, --min-age-days, --ci-history, --szz, --declared-labels, --merge-outcome, --max-unmerged, --pr-discovery, --llm, --no-scanners, --no-api-contract, --no-review-history, --llm-model, --workers, --llm-concurrency, --llm-scope, --llm-sample-pct, --close-reason-llm, --llm-max-diff-chars, --llm-quota-wait, --window-from/--window-to, --lead-in-days, --cap-on, --full-repo, --snapshots) plus:

FlagDefaultMeaning
--repos-file Frepos.txtURL list
--out Fdataset.parquetsingle output, rows appended
--out-prefix, --out-postfix, --no-auto-nameoutput naming (4.1)
--clone-dir Dcloned_reposwhere to clone
--skip-existingoffreuse an existing clone
--keep-clonesoffdon't delete a clone after mining
--repo-retries N2extra clone/mine attempts per repo
--clone-stall-s S600 (RIFFLE_CLONE_STALL_S)kill a clone that prints nothing for S seconds and retry; 0 = never
--max-files-deep N300 (RIFFLE_MAX_FILES_DEEP)a PR touching more than N files gets per-file features for the N with the most churn only; 0 = never cap
--llm-backend {claude_cli,api}LLM_BACKEND, else claude_clireaches the miner as LLM_BACKEND
--llm-diff-context N1 (LLM_DIFF_CONTEXT)context lines around each change in the diff sent to the LLM
--sync / --no-synconsend output to the team store (only when RIFFLE_SYNC_URL and RIFFLE_SYNC_TOKEN are set)
--split-by {commits,prs}commitswhat --shard balances (12.1)
--issue-tokens, --token-days D, --tokens-out FILEwith --shard-plan: one write token and join code per PC, expiring after D days; the codes also saved to FILE
--only-domain NAMEmine only repos tagged [domain: NAME]
--shard K/N, --shard-plan Nsharding (12.1)
--remine, --remine-repo OWNER/NAME, --auto-remine, --auditre-mining and audit (12.3)
--skip-big N0 (off)skip repos with more than N commits by tag
--snapshot-root Dsnapshots_rawcapture root
--assemble-snapshots / --no-assemble-snapshotsonassemble at batch end
--log-level {DEBUG,INFO,WARN,ERROR}INFOtrace detail
--test, --test-prs N, --test-repo URL10, pallets/flasksmoke test; fixed window, so the same PRs on every PC

Scale example (the web-ts pilot and run):

bash
python build_dataset.py --repos-file repos_web_ts.txt --only-domain web-ts \
    --window-from 2023-07-01 --window-to 2026-07-31 --max-prs 8000 \
    --remine-repo vitejs/vite
python snapshots.py --dataset dataset.parquet --raw-root snapshots_raw

16.3 Other tools

ToolUsage
snapshots.py--dataset F --raw-root D [--out F] [--commit-features-out F]
merge_shards.py--out dataset.parquet [--dir .] [--allow-version-mix] [--allow-incomplete]
scout.py--repos-file F [--agent-search] [--out scout.json]
llm_compare.py--dataset F --clone-dir D --models sonnet haiku --n 300 [--dry-run]
audit_export.py--dataset F --out label_audit.csv [--n 600]
trace_report.pylogs/run-*.trace.jsonl [--top 10]
doctor.py[--check]
deps.pypython deps.py installs and reports what is missing
tui.pypython tui.py

17. Environment variables

VariableUsed byMeaning
GITHUB_TOKENAPI, GHSAtoken (read scope: public_repo, or repo for private); without one: 60 requests/hour, no labels
LLM_BACKENDllm_flagsclaude_cli (default) or api
LLM_API_KEYapi backendkey for the OpenAI-compatible endpoint
LLM_MODELbothmodel; default sonnet (cli) or claude-sonnet-5 (api)
LLM_BASE_URLapi backenddefault https://api.anthropic.com/v1/
LLM_TIMEOUTcliseconds per call (180)
LLM_DIFF_CONTEXTllm_flagscontext lines in the diff sent to the LLM (1)
LLM_CONCURRENCYllm_flagssimultaneous calls (4)
LLM_SCOPEconfigtraining or all
LLM_RETRY_BACKOFF_Smine_repoper-PR retry wait unit (60)
LLM_CALL_RETRY_Sllm_flagswait between quick retries inside one call (10)
RIFFLE_WORKERSconfigPRs in parallel
RIFFLE_MAX_FILES_DEEPconfigfiles per PR that get per-file features (300; 0 = no cap)
RIFFLE_GIT_TIMEOUT_S, RIFFLE_GIT_BLAME_TIMEOUT_S, RIFFLE_GIT_LONG_TIMEOUT_Sgitiotimeouts for a git call (300), a blame (120) and the whole-history log (3600); a timeout is unobserved
RIFFLE_MAX_BLOB_BYTESgitiolargest file read at a commit (4 MiB)
RIFFLE_CLONE_STALL_Sbuild_datasetkill a silent clone after this many seconds (600; 0 = never)
RIFFLE_SEMGREP_RULESscannersa rules file or directory; unset uses --config auto, which fetches rules from semgrep.dev
RIFFLE_SEMGREP_PARALLELscannersconcurrent semgrep processes (2)
RIFFLE_NO_API_CACHEapi_cacheturn the GitHub API cache off
RIFFLE_AUTO_UPDATEdoctor1 lets the launch fast-forward a clean main checkout; default only reports a newer miner
RIFFLE_NO_AUTO_UPDATEdoctor1 turns the update check off
RIFFLE_SYNC_URL, RIFFLE_SYNC_TOKEN, RIFFLE_SYNC_TIMEOUTsyncteam store (https except localhost), this PC's write token, how long a Stop waits for the last upload (120 s)
RIFFLE_TRACE, RIFFLE_LOG_LEVELrunlogtrace file and level
RIFFLE_EVENTSprogressprogress events file (TUI)
RIFFLE_CONTROLbuild_datasetfile the TUI writes repo names to, to request a skip
RIFFLE_NO_DOCTORtuiskip the doctor
RIFFLE_NO_INSTALLdeps, doctornever install, only report
RIFFLE_NO_WAITtui, doctorskip the 5 s pause before the TUI opens
RIFFLE_NO_VENV, RIFFLE_IN_VENVdepsdon't switch into .venv / set when inside it
RIFFLE_NO_PLAN_USAGEllm_flagsdon't poll claude -p /usage
RIFFLE_DASHBOARD, RIFFLE_DASHBOARD_PORT, RIFFLE_DASHBOARD_BINDdashboarddisable; first port (8765); interface to bind (default 0.0.0.0)
SZZ_EQUIV_REPOtestsa real clone for the SZZ equivalence test

Note: .env.example ships LLM_MODEL=claude-haiku-4-5-20251001. Because .env is auto-loaded and LLM_MODEL applies to both backends, copying the template unchanged makes headless Claude Code run that model instead of sonnet. Remove or change the line deliberately.


18. Tunables (Settings in config.py)

Most are not exposed as flags; they mirror rules.mining / labels in contracts/org_rules.schema.json where noted.

SettingDefaultMeaning
churn_windows(90, 365)churn windows (days)
decay_half_life_days90half-life of fix/revert decay
revert_window_days30window for label_revert and revert maturity
hotfix_window_days7window for label_hotfix
label_lookahead_days90how far after merge to search reverts/hotfixes
szz_maturity_days90cap on days_to_szz_fix
recent_burst_days7burst window
hotspot_top_k20hotspot top-k
blame_ignore_revsNonefallback ignore-revs file (the repo's own file wins)
hunk_proximity_k5± lines counted as "near" a past fix
maturity_days30a past PR counts once this old (labels.maturity_days)
similar_prs_k, _min_file_overlap, _lookback_days, _min_neighbours5, 0.2, 365, 3similar-PR neighbourhood
scrutiny_lookback_days, _exclude_bot_reviews, _min_prior_prs180, True, 5scrutiny
failure_mode_lookback_days, _min_bad_changes365, 2failure modes
path_size_lookback_days, _min_prior_changes365, 10path size percentile
release_tag_pattern, release_min_releases, release_interval_stat^v?\d+\.\d+(\.\d+)?$, 3, medianrelease cycle
author_smoothing_alpha5.0empirical-Bayes strength
enable_scanners, enable_semgrep, enable_deps_cve, enable_api_contract, enable_llm, enable_github_api, enable_ci_history, enable_szz, enable_review_history, enable_declared_labels, enable_merge_outcome, enable_snapshotsall Truestage toggles
declared_label_grace_s120include labels applied within N seconds of open
close_reason_classifierllmllm (rules fallback) or rules
max_unmergedNonecap unmerged rows
abandon_days60open this long with no human response = abandoned
merge_signal_lookback_days, merge_signal_min_prs365, 5prior PRs counted by repo-level merge signals; fewer than this gives a null rate
rework_grace_s300commits/force-pushes this soon after open are not rework; also the at-open head cutoff
stale_bot_loginsstale[bot], stale-bot, stalebot
unwanted_labelswontfix, won't fix, invalid, not planned, out of scope, declined, rejected, spam
ci_cache_dirci_cachecache folder
llm_backend, llm_model, llm_base_url, llm_timeout, llm_concurrency, llm_scope, llm_sample_pctsee 17 / 7(llm_breaker_threshold was removed in 7.0.0; llm_breaker_tripped stays as the stop flag)
llm_skip_trivialTrueskip docs/tests/lockfile/generated-only diffs
llm_diff_context, llm_max_diff_chars1, 25000diff sent
close_reason_llmother_onlyclassifier policy
llm_quota_wait, _pause_pct, _poll_s, _max_wait_hTrue, 95, 300, 12plan-limit handling
llm_row_retries, llm_retry_backoff_s, llm_call_retry_s3, 60, 10retry policy
flaky_detection, flaky_min_rerun_passes, flaky_lookback_daysTrue, 3, 90flaky checks
snapshot_max_heads, snapshot_dir5, Nonecapture
max_files_per_pr_deep300files per PR that get per-file features (0 = off); --max-files-deep, RIFFLE_MAX_FILES_DEEP
workers, request_timeout, max_prssee 4.6, 20 s, None
window_from, window_to, lead_in_days, cap_on, domain, min_age_daysNone, None, 180, all, None, 90selection
merge_signalauto`auto
pr_discoveryapi`api
WORKER_RAM_GB, BASE_RAM_GB0.8, 1.5RAM model for default workers

19. Tests

python -m pytest contracts/tests pipelines/ingestion/repo_miner/tests -q is what CI runs for the miner (the contracts job in .github/workflows/ci.yml). The training loader and Airflow script tests (pipelines/training/tests, infra/docker/airflow/tests) are run by the pipelines CI job, which installs pandas and pyarrow and also runs the miner tests again, so the parquet paths, merge_shards.py and the loader are covered; the contracts job runs the miner on core dependencies only, so tests that need pandas skip themselves there. Most tests use small throwaway git repos and synthetic timelines, with no network.

FileWhat it pins
test_dataset_contract.pythe output matches mined_dataset.schema.json both ways (no extra column, none missing), rows validate, rows round-trip through the flat form (ADR 0012), with stages on and off
test_history_leakage.pyevery repo-local and merge-signal history column is unchanged when recomputed from a world cut at the PR's open (later PRs, closes, outcomes, reviews, first responses, release tags removed)
test_snapshots.pyhead over time, capture, cap, triggers/state/clean-up, code features follow the head, merge signals as of the snapshot, no leakage, draft and reopen clocks, snapshot/commit rows match their contracts, snapshot and commit rows flatten losslessly, snapshot contract keeps every dataset column, weights sum to 1 per PR, one capture per PR, batch-end assembly
test_merge_labels.pymerged unchanged/after rework, commit within grace, own/bot change requests, duplicate/unwanted/abandoned/ci_technical, events after close ignored, unreadable timeline, targets, repo-local for unmerged rows, LLM reason, stale-open, at-open title and milestone, partial timeline, base-branch merges not rework
test_revert_labels.pyrevert subjects behind a conventional prefix; revert matching by PR number, not other numbers; urgent-fix phrasing vs ordinary fixes
test_szz_equivalence.pythe fast SZZ equals the serial full-file algorithm exactly (keys and order), resume from checkpoint, blame store survives a new window/selection, windowed and relevant-path pruning give identical labels (including renames), relative ignore-revs, absent path is not a failure, LLM reuse
test_flaky_maintainers.pyflaky-only failure is not a CI fail, only earlier rerun-passes count, check-runs mark rerun-passes, maintainer evidence and round trip, snapshot roles, response cut off mid-body is retried, only push-triggered workflows count as the merge commit's CI, automation-only commits have no verdict, unreadable runs fall back, preview jobs are optional
test_window_selection.pywindow takes every closed PR created inside it, cap keeps the newest contiguous block, too-recent outcomes left out, dependency bots are bots not agents, agent PRs stay in LLM scope, provenance columns, health flags, selection round trip, cap-on training
test_llm_savings.pydiff slimming, scope, trivial skips, rules-first close reasons, quota gate/limit waits, RAM-aware workers, clean stops, quota pause counted as wait
test_llm_failures.pya failed call is retried and the row keeps its LLM columns; an LLM that stays down stops the repo (exit 3) without empty rows (uses a fake claude on PATH)
test_merge_signals.pybody/agent parsing, repo columns are prior-only, rates null without unmerged rows, repo file patterns
test_merge_shards.pymerging each table, refusing pieces that don't fit or snapshots older than the dataset, batch writes a manifest merge_shards accepts
test_naming.pyauto tag for sharded runs only, idempotence, prefix/postfix/placeholders, run stems, snapshot names
test_scanners.pysemgrep base scan skipped when head is clean, findings compared with base, equals the original algorithm, relative paths, nothing to scan never starts semgrep
test_runlog.pytrace off writes nothing, PR scope collects stage totals, WARN echoes, stage marks, field names don't collide
test_tools.pyscout, llm_compare, audit_export
test_tui_options.pythe TUI passes the v3-v6 options to build_dataset and validates them; build_dataset accepts every flag the TUI sends
test_contract_sync.pya schema's x-riffle-miner-version is a semver and not ahead of version.py; every schema's x-riffle-version has a row in docs/contracts.md
test_dashboard_big.pythe big-picture elements are on the page and its script parses
7.0.0 audit tests: test_git_audit_fixes.py, test_history_shared.py, test_deps_cve_point_in_time.py, test_scanners_unobserved.py, test_api_limits.py, test_llm_audit_fixes.py, test_snapshot_audit.py, test_checkpoint_resume.py, test_batch_checkpoint_stop.py, test_batch_options.py, test_merge_shards_incomplete.py, test_sync_safety.py, test_tui_robust.py, test_mine_wiring.py, test_scout_health_small.py, test_audit_leftovers.pyone file per area of the audit: paths and renames, the shared history index, as-of CVEs, null scanners, rate limits and cache expiry, LLM validation and reuse, snapshot rules, checkpoint stamps and atomic writes, process-group stop, option validation, incomplete shard manifests, sync safety, TUI robustness, wiring, and the leftovers. pipelines/training/tests has test_frames_audit.py and test_contract_keywords.py; infra/docker/riffle-data/tests has test_audit_fixes_2.py

20. Known limitations and sharp edges

Documented, not bugs, unless noted.

Labels and outcomes

  • SZZ blames only deleted/modified lines (adds-only fixes trace to nothing) and has about 0.6 precision. It is kept out of label_bad (ADR 0002).
  • label_revert and label_hotfix are false (not null) when an outcome simply wasn't observed; label_szz_bug and days_to_szz_fix are null when SZZ did not run (7.0.0). Use the *_window_closed, ci_observed and label_bad_observed columns, and know the run mode.
  • GitHub keeps check results for about 90 days on public repos, so older ci_fail is unobserved (label_ci_fail null; label_bad_observed false). The health check flags repos where CI is unobserved on more than 80% of merged rows.
  • Revert and hotfix search uses git log --all of this clone: a revert that landed after the clone's newest commit is invisible, which is why observed_through exists.
  • Hotfix detection is keyword-based on commit subjects (hotfix, hot-fix, rollback, rolled back, quick fix, urgent/emergency/critical + fix, patch, bug, issue, regression or security); a repo that never words its fixes that way (vite) has no positives.
  • label_ci_fail is only as good as the merge commit's CI: on vite, 56% of merged PRs have no check run or status on the merge commit at all (checked on a sample of 14 across 2024-06 to 2026-07), so their verdict is null, and only 884 of 2,000 rows have a fully observed label_bad. Falling back to the PR's last pre-merge checks would answer a different question (did it pass before merging, not did it break main) and is not done. When branch protection is unreadable the required checks are guessed from a name denylist, which can still count an automation job that runs on push (6.2).
  • Merge-outcome rules (label_unwanted, label_duplicate) are phrase rules or an LLM classifier; precision is not yet validated (ADR 0009 open item: validate against Ehsani et al.'s 600 hand-coded PRs; use audit_export.py).
  • Still-open abandoned rows can later merge; whether to re-mine them is open.

Features

  • Commit times are commit dates, not push times: a commit made before open but pushed after counts as part of the PR at open.
  • The PR body is read at mining time, so body features can see post-open edits.
  • Concurrency features and repo-local history are computed over the PRs in the same run: a small --max-prs undercounts; the oldest rows have little history (null).
  • Percentile features use a per-commit population (a proxy for the PR-size distribution).
  • hunk_fix_proximity approximates "lines a past fix touched" by blaming the base and taking lines owned by fix commits, not the fix commits' own diff lines.
  • Blame uses -w -M; -C is off.
  • A repo allowing both squash and rebase with multi-commit PRs relies on the author-overlap confirmation; a repo flooded with rebase merges that cannot be sniffed (API off) uses merge^1 and can undercount multi-commit PRs.
  • A semgrep scan that did not run (tool missing, bad exit code, bad JSON, timeout) gives null for all three semgrep columns (7.0.0; a missing tool used to give 0). lizard missing or a parse error gives null. The default semgrep rules (--config auto) are fetched from semgrep.dev unless RIFFLE_SEMGREP_RULES pins local ones.
  • CVE fields are null both for "no dependency change" and for "unobserved", and also for an unresolved version (a range, workspace:, a git URL) or a record that cannot be dated. They are as of the PR's open time: later or withdrawn records are dropped.
  • Pnpm lockfiles, pyproject.toml and Gradle files are not parsed: a PR touching one has null dependency counts.
  • dependency_major_bump_count (regex) and n_major_bumps (parsed) can disagree; min_hotspot_rank_touched and min_file_hotspot_rank duplicate each other.
  • Rows with open_snapshot = final have diff features that may leak.
  • merge_signal's landed-commit mode gives no unmerged rows; git-primary discovery misses PRs with no recognisable reference and non-default-branch PRs.
  • Maintainer detection catches people who triage; an employee who never merges or closes others' PRs still counts as outside; author_association hides private org members as CONTRIBUTOR (which also biases scout.py).

Deferred in 7.0.0 (known, and not changed)

  • author_prs_merged_repo and similar_pr_count stay 0 for "none": the contract defines 0 that way.
  • Body-derived features (body_len, has_linked_issue, body_*, agent_boilerplate, author_association) cannot be as of open, because the timeline has no edit history; they stay as read at mining time.
  • An unmerged PR whose head cannot be fetched is dropped with a warning; it is not counted as gone and gets no null-feature row.
  • A wiped team store is not detected by .sync_state.json (it needs a store listing API).
  • Per-subprocess stop checks in gitio (a stop hook before each git call) are not in: a Stop takes effect at the next step.

Operations

  • GitHub quotas are per account; shard PCs need different accounts.
  • Every shard PC must use the same data-affecting flags and the same miner version (merge refuses version mixes unless overridden).
  • A fresh clone with new commits changes HEAD and invalidates the SZZ blame store.
  • If the batch process is SIGKILLed, its child miner is not signalled automatically.
  • Re-mining a repo after assembling snapshots makes the snapshot table stale; merge_shards and frames.py refuse stale snapshots.
  • .env.example's LLM_MODEL overrides the default model for both backends (17).
  • The README's .env sample shows LLM_MODEL=claude-sonnet-5, which is the api default, not the headless default sonnet.

21. Version history

The version rules are in version.py (MAJOR: output columns or meaning; MINOR: new features or flags; PATCH: output-identical fixes). From the git history of this directory (dates 2026):

VersionDateChange
1.6.0date not in this directory's historyblame fixed for repos that ship .git-blame-ignore-revs (affects audit thresholds)
2.0.010-16 (ADR date)merge-outcome target on by default (ADR 0009); features from the PR as it opened (ADR 0010); open_snapshot
2.2.010-18ADRs 0009 to 0011 reach main; 199-column mined_dataset 1.0
2.3.010-19with --no-merge-outcome the 13 merge-outcome columns are written null (ADR 0012); miner matches its contract; flat form
3.0.010-19LLM fits the Claude plan (quota gate, scope, trivial skips, diff slimming), no row without its LLM answer, quiet stops, RAM-aware workers
3.1.010-19PR snapshots: capture while mining, assembly offline
4.0.010-19date-window selection, history lead-in, bot/agent split, provenance columns, health check
(no bump)10-19web-ts repo list, scout.py, llm_compare.py, audit_export.py
5.0.010-19training targets (outcome_3class, nowhere_reason), flaky-check detection, maintainers from history
5.1.010-19resume exactly where a run stopped: pinned selection, per-(fix commit, file) SZZ blame store
6.0.010-19cap on training PRs (--cap-on training), capture cost log, snapshot weights
6.0.110-19snapshot assembly keeps one capture per PR
6.1.010-19TUI for the new options; web big-picture mode
6.1.110-19retry GitHub responses cut off mid-body (IncompleteRead)
6.2.010-20run trace, auto-naming of sharded runs, snapshot assembly at batch end, merge_shards.py, revert/hotfix label fixes, semgrep findings compared by relative path (they were overcounted), big picture on phones
6.3.010-20label_ci_fail counts only the merge commit's push-triggered CI; preview joins the optional-check denylist
6.4.010-20environment doctor: OS/distro detection, per-package-manager installs (Node and Go first when missing), user-prefix npm, PATH for user bin dirs, grouped report, 5 s pause before the TUI opens; fixed label: every GraphQL diff was breaking (the miner matched "No breaking changes detected"); a graphql-inspector failure is now null
6.5.010-20--full-repo: mine a repo's whole history instead of a date window or a cap; a missing LLM backend now stops the run instead of nulling the LLM columns; the shard plan (and TUI Show split) also lists PR counts from the [N PRs] tags
6.5.110-21--test mines a fixed window (2020-01-01 to 2025-06-30, no age cutoff), so test runs pick the same PRs on every PC and day; before, the cap, --min-age-days and the stale-open cutoff moved with today's date and the form
6.5.210-21TUI only, output identical: the form no longer saves or restores .tui_settings.json, so every launch starts from the defaults (everything on except Git PR discovery, Keep clones, Auto re-mine and Skip big repos); the default LLM model in the TUI is now Haiku (the CLI default stays sonnet)
6.5.310-21output identical: the CI-history pre-pass checkpoints its pages (ci_cache/<repo>.json.pages.json) so a stopped run resumes at the next page, and a walk that ended on a failed page is no longer cached as complete
6.6.010-21sync.py and --sync/--no-sync: logs, health, datasets, shard manifest go to the team store after each repo, on a Stop and at the end; on only when RIFFLE_SYNC_URL and RIFFLE_SYNC_TOKEN are set; output identical
6.7.010-21join.py, --issue-tokens, --token-days, --tokens-out: join codes with per-split write tokens; the run checks its token and split at start; sync settings are read from .env whoever imports sync first; doctor reports the team store; output identical
6.8.010-21TUI: Join code box and F2, form filled from the lead's flags, Start refused when shaping settings differ, store line and upload status, quit waits for the final upload; build_dataset.build_parser(), join.differences(); sync events for the TUI; output identical
6.9.010-21TUI Lead tools (F3): save the admin token, issue join codes for the form's settings (join.shared_argv keeps only what every PC must share), list and revoke PC tokens; RIFFLE_DOTENV and RIFFLE_NO_DOTENV for config; admin calls raise AdminError; output identical
6.9.110-21--issue-tokens also sends the plan's flags and the miner version to the store, so the admin page can rotate a split and rebuild its join codes; output identical
6.10.010-21doctor.py checks the join, read and admin tokens beside the GitHub token and can take a join code or a read token (RIFFLE_READ_TOKEN); TUI F4 opens the store and copies the read token; output identical
6.10.110-21the status heartbeat is sent at once at the start of a run and of each repo, then every 3 minutes (was 15, and nothing at the start); the store's Run page refreshes every 30 s; output identical
6.10.210-21TUI layout only: Start, Stop and the secondary actions sit in an always-visible bar under the scrolling form, the Join, Lead and Store buttons share one row, and the footer lists only the keys in daily use; the Lead tools screen is rebuilt as aligned one-line rows with the message pinned under the header, and the join box keeps its result and a pasted-characters hint on screen; output identical
6.10.310-21TUI: after joining, the lead's settings are locked (every field the join code decides is disabled, machine settings stay editable), remembered in .join.json across restarts, and unlocked only by Leave; output identical
6.10.410-21the lead's repo list and who issued the split travel through the store (--issue-tokens also sends the repo list, so a joining PC fetches it from /v1/split, writes it under joined_repos/<split>/ and points --repos-file at that copy); a joining PC uses the lead's copy and sees the lead's settings and its share (the TUI card, F5); see 6.10.3 for the lock
6.10.510-21audit of the token, sync and join features: a join code is validated (no line breaks into .env), health files are limited to the run's repos, a store answering early is reported as a rejected token, reads from the store are bounded, flags cannot be abbreviated, text from the store is shown as text in the TUI; store: tokens.json changes are locked and atomic, revoke by name means a PC token, an admin cannot revoke itself, bad lifetimes, bodies and run.json are refused, search regex bounded, socket timeout
6.10.610-21second audit round: a repo that was skipped, stopped or failed to clone is no longer shown as mining, an aborted batch (LLM breaker) is not reported as done, uploads share one total deadline and send small files first, upload state is kept per token, the doctor's join saves the lead's repo list, the TUI keeps the join lock when the store is down and re-checks the store every minute; store: failed token backups are retried and shown, rotate refuses to build codes for localhost
6.10.710-21store pages: stylesheet and scripts are linked by content hash, so a deploy shows at once; no miner output change
6.10.810-21store pages: refresh only while a PC is mining (15 s, idle otherwise), tables align with their headings, a big-picture mode for the overview and admin pages; no miner output change
6.10.910-21store pages: every table header stays in view, logs get a page-size choice, load-all, open-full, download and filtered download, colours by meaning, and mined-data reads are computed once per 10 minute window with a clock and a Refresh that waits for it; no miner output change
6.10.1010-21store pages: an activity graph (one bar per PC, a block per time slice, mining, done, stopped or offline, plus what each PC is doing now), a status strip under the header, aligned JSON trees, themed scrollbars and phone layout fixes; no miner output change
6.11.010-21fixes found by analysing the first real two-PC run: legacy-mode unmerged rows come from the merged PRs' own span (130 of 168 click rows had closed before the oldest merge); a commit with 100 or more check runs no longer raises NameError and drops its PR; commit-feature rows exist only for PRs that have a dataset row (a PC's table held rows of a repo it never mined); the health file flags PRs dropped by errors; run.json carries the run's settings (LLM model, SZZ, workers, window) for the admin page
6.11.110-21store pages: log search filters by level (errors, warnings and errors, info, debug) and opens a hit at its line, a team big picture in the pages' own style (a status tag, a strip of key figures, one table row per PC with a 24 hour activity bar), one timer on the data clock, "/" jumps to search, the tab title shows when something is mining; no miner output change
6.11.210-21store: an admin Storage panel (bucket size by kind and by PC, the largest files, bytes written, the VM disk, and an estimated storage cost; sizes only); big picture fits a phone; no miner output change
6.11.310-21TUI: Leave shows whenever the PC holds a store token (not only when the form is locked) and forgets the token when there is no saved join code; Lead tools get Join this PC as selected, so the lead joins its own PC without pasting a code; run status reports the default LLM model; no output change
6.11.410-21TUI: Leave forgets only the store token and keeps RIFFLE_SYNC_URL, which Lead tools need (it had cleared both and the lead saw "Store (not set)"); no output change
6.11.510-21admin page: a refresh no longer resets a pending Revoke confirm (revoke seemed to do nothing while a PC was mining), and a split can be removed (Remove split); no miner output change
6.12.010-21`--split-by commits
6.12.110-21join and Leave cannot leave a token without its saved join code (a failing join no longer crashes the TUI; the join code is saved before the token; the doctor warns on a token with no join code); config treats an empty environment variable as unset; the TUI follows a token the doctor saved while it was open; Clear log, Copy log, Skip repo, Test data and Web view are buttons again; no output change
6.12.210-22the PR total shows the plan's share (not ?) until every repo has its own; "not written" says why; PR commit counts for rebase-style repos are read from GraphQL in batches of 50 instead of one REST call per PR and so are each PR's reviews and review comments (10 PRs a call, checked identical to the REST read on 100 PRs; a PR too big for one page still goes through REST; the sidecar's event lists are now in time order) (far less of the 5,000/h REST quota); this run's log, trace and the repo-in-progress checkpoint go to the store every 15 minutes while it mines, not only after a repo or on Stop; a Stopped repo shows stopped at N/M (not mining) and the header says stopped; the status carries prs_total (the split plan's share) and the store shows mined-of-planned and PRs left; Stop and quit work on Windows (the TUI used os.killpg, which Windows lacks: it now sends CTRL_BREAK, which the miner turns into a normal interrupt, and kills with taskkill /T); no output change
6.12.310-22a GraphQL quota wait is logged (it slept silently for minutes); one Stop is one interrupt (a second within 5 s no longer prints a traceback at exit); the status carries rate_wait_until and the store shows waiting for GitHub until HH:MM; no output change
6.12.410-22a commit's check runs, reruns, workflow events and legacy statuses come from one GraphQL call instead of 3-4 REST calls (identical to the REST read on 190 live commits; a full page or a failed query goes through REST; the checks list is sorted by name); a PR's timeline, review activity and commit count are cached on disk per repo (ci_cache/<owner>_<name>.api.sqlite, 14 days) so a restart does not read them again; a Stop is reported as stopped, not mining, to the store; no output change
6.12.510-22every 15 minutes the whole sync runs (log, trace, checkpoint, finished tables and health files; only what changed is sent, and a failed send is retried), not only the log, trace and checkpoint; on Windows a skip or Stop signals the miner's child with CTRL_BREAK in its own process group (it was terminated outright, which skipped its cleanup); no output change
6.12.610-22GraphQL spends its own 5,000-point hourly quota by the pages a query could return, not the ones it returns: the review query asked for 100 threads of 100 comments (about 1 point a PR; 9,351 PRs were 9,000 points, spent in one go before mining started, after which everything fell back to REST) and now asks for 20 threads (about 0.2); the check query asks for 30 suites; the review prefetch reads only PRs not yet mined, and stops with 2,000 points left; an exhausted quota (HTTP 200 with a RATE_LIMIT error, which was not recognised) switches GraphQL off until its reset and reads through REST, except force-push events, which wait for it; an author's account age is cached on disk (it never changes: about 1,500 REST calls saved per restart on a repo like nuxt/nuxt); a failing or timing-out GraphQL call is tried 3 times, not 9 (every caller has a REST fallback), so it cannot stall a worker for minutes; no output change
6.13.010-22the historical CI map reaches past GitHub's 40,000-run list limit (page 401 of the repo's workflow runs is an HTTP 422, so on a big repo the list reached back a few months and author_post_merge_ci_fail_rate / max_file_post_merge_ci_fail_rate were null for about 98% of nuxt/nuxt's PRs): after the 40,000 newest runs it reads older runs by creation date, a week at a time (a week with more than 1,000 runs is halved, because a date-filtered list returns 1,000 at most), until 26 weeks in a row have none; checked identical to the plain list on 1,200 live runs; the walk is checkpointed by date, so a stopped run resumes; the same runs are counted as before (every non-pull_request event, gating workflows only); a repo whose runs fit one list is unchanged. Values that were null for older PRs are now filled where the repo has run history; a repo already mined with an older version keeps its nulls until it is mined again; about 1,300 REST calls once per big repo
6.13.110-22a PR whose changed paths clash in the exported semgrep tree (a file icons beside icons/a.svg, as in nuxt/nuxt #27021) is mined, with the clashing file left out of the scan, instead of being skipped with NotADirectoryError; PRs that were skipped this way are mined on the next run
6.13.210-23a workflow-runs page cut off mid-read (IncompleteRead on a ~1 MB body, seen 12 times in one run on nuxt/nuxt, remix-run/react-router, solidjs/solid and preactjs/preact) is retried with backoff like any other network error instead of failing the whole repo; 5xx on that path is bounded; a CI-history pre-pass that still fails leaves its features null and the repo is mined
6.13.310-23the team-store status counts each unfinished repo's own rows in the shared checkpoint: the repo being mined was credited with every checkpointed row (pc1of6 showed 4,499 rows of nuxt/nuxt as vitejs/vite), and a failed or stopped repo's rows kept for the rerun now show under it
7.0.010-23audit-fix release; re-mine before retraining. Contracts: mined_dataset 1.3, mined_snapshot 1.4, mined_commit_features 1.1 (optional llm_diff_truncated and szz_window_closed; x-riffle-non-input and x-riffle-diff-derived keywords). Labels: label_revert and label_hotfix no longer match the PR's own merge commit or commits and need PR-number order; a revert needs #N in its subject or Reverts owner/repo#N in its body, searched from the default-branch tip (--all only as a fallback); label_szz_bug and days_to_szz_fix are null when SZZ did not run; SZZ follows renames (SZZ_VERSION 3). Git and history: every git call uses -c core.quotePath=false, -z for numstat, name-status and renames, and a hunk-count patch parser, so non-ASCII, quoted and backslash paths get LOC, blame and SZZ; renamed files use the old path at the base and history follows renames; merge commits are out of author totals, EXP, REXP, SEXP, author_last, n_commits_total, repo_revert_base_rate and the populations; bots are out of NDEV and blame ownership; line ages clamp to 0; entropy counts only files with changed lines; percentile populations skip generated files; dependency_major_bump_count attributes a bare version line to the preceding package; similar_pr_bad_rate uses the repo's own bad-PR rate among PRs mature at T as its prior; a default branch with / is kept whole; landed-commit discovery skips reverts and prefers merge or (#N) hits and the API's merge_commit_sha; a fast-forwarded PR whose head is a merge commit gets its own range and a first-parent author; FIX uses the at-open head subject; author_confirmed_maintainer is null with no timelines; a failed git fetch is retried and raises FetchError (unobserved, not gone). History index built once per repo and answered by bisect (float last digit may differ in the decay and REXP sums). Limits: RIFFLE_GIT_TIMEOUT_S 300, RIFFLE_GIT_BLAME_TIMEOUT_S 120, RIFFLE_GIT_LONG_TIMEOUT_S 3600, RIFFLE_MAX_BLOB_BYTES 4 MiB, --clone-stall-s 600 (a timeout is null, never 0); --max-files-deep 300 (the rest of a big PR is null, with LT, la_norm, ld_norm and sum_*). GitHub API: a failed or truncated check-run, status or workflow-run page gives a null verdict (with the workflow-runs read failing, a would-be pass is null); review events are None and uncached on a failed page or at the cap; labels_at and force-pushes are None on failure; Retry-After as seconds or a date, secondary-limit bodies, capped waits and one process-wide gate; the CI map cache is versioned (v2), expires after 7 days and is rebuilt when older than the newest PR; open-PR timelines cached 1 hour, not 14 days. Dependencies and scanners: CVE columns as of the PR's open time (later or withdrawn records dropped; undatable means all null); vuln_has_fix_available is False or null, never True; dependency counts are null when a manifest is unparsed; parsers for package-lock v1 to v3, yarn.lock, poetry.lock, Pipfile.lock, composer.lock; counts per ecosystem and directory; unresolved versions null the CVE columns; GHSA errors null osv_ghsa_disagreement; semgrep failure is null for all three columns, findings compared as a multiset without line numbers, --metrics=off, RIFFLE_SEMGREP_RULES and RIFFLE_SEMGREP_PARALLEL; deleted functions count in ccn_delta; a lizard parse error is null; api_contract_change is null when any changed spec is unobserved; agent detection anchored (julesb, claude-docs, cursor-fix are not agents). LLM: a reply with zero valid flags fails; a diff with no code after slimming is status empty; llm_prompt_version also hashes the diff rules, redaction, diff settings, backend and transcript format; llm_model is the resolved id; reused answers are re-validated and skipped on a different prompt version or model; broader secret redaction; 529 is retried; the quota wait is capped; the close-reason transcript quotes bodies with `
7.0.110-24patch: --auto-remine is off by default (TUI checkbox too); a [redo] block before each batch (and in --audit) says which repos a resume will redo because the checkpoint or pinned selection came from another miner version; the audit and version checks read six columns once instead of the whole dataset three or four times; [timeline] n/N progress; open snapshots run in slices of 2,000; git runs with gc.auto=0 and maintenance.auto=false; _kill_children no longer raises on Windows (no os.getpgrp). GITHUB_TOKENS: several tokens of one owner, used in turn when one runs out of quota (REST and GraphQL counted apart; tokens named by position in logs); closed and merged PR timelines stay in the API cache 90 days (CLOSED_KEEP_S), other entries 14; open snapshots are kept in the pinned selection and restored on a restart (same grace and default branch, every head in the clone); the pages of a busy CI-history week are read four at a time. Output identical, and the checkpoint stamp (major.minor) is unchanged, so a resume from 7.0.0 reuses its work.
7.1.010-27minor: run tracking; output identical. A new miner version no longer throws earlier work away: the checkpoint stamp and the pinned PR list ignore the miner's version (an older row's miner key is ignored), the start-of-batch [redo] block is a [version] notice, and a repo is mined again only by --remine-repo (the TUI's "Re-mine repos"); a resume from any 7.0.x reuses its work. PCs on different versions need --allow-version-mix at merge time. New lead setting --live-tracking (off by default; SHAPING, the TUI's "Live tracking" box, the admin settings form): the trace and a worker heartbeat go up every 10 s. RIFFLE_SYNC_BEAT_S, RIFFLE_SYNC_LOG_BEAT_S and RIFFLE_HEARTBEAT_S fall back to their defaults when empty or not a positive number, never crash the miner. A trace uploads as new complete lines only (X-Append-Offset) to a store that says append; any mismatch falls back to the whole-file PUT. The trace has a batch repos event, batch done is written before the last upload, and long stages write progress events. Open snapshots: force-push lookups report progress (done, total, seconds a call, minutes left) and keep each answer in the API cache for 2 days (fp:<pr>), so a restart does not repeat hours of GraphQL calls; unknown answers are never kept; each slice of snapshots reports too. Web view: the headline names what the run is doing (RATE LIMITED, a stage name, MINING, LLM PAUSED, NO PROGRESS, ASSEMBLING), wide numbers fit instead of being cut off, and the repository count is shown once. TUI and doctor: "Also run Loki + Airflow on this PC" (localstack.py, never blocks or stops the miner) and a Run tracking group of checks. New stages.py (the stage and PR-step lists, tested against mine_repo.py) and publish.py (contract check and DVC for the merged tables). Sync: an optional mirror store (RIFFLE_MIRROR_*), log-only quick beats with back-off, a state file that survives damage, and a check that a trace is the same file before it is appended to. Open snapshots' force-push answers are kept for 6 hours. Store and infra, outside the miner's output: the store forwards each trace line to Loki (server/loki.py), takes appends, searches through Loki and has a Run tracking panel on the admin page; compose.logs.yaml; the riffle_miner Airflow DAG; infra/terraform/store-vm. ADR 0017 and 0018.
7.2.010-28minor: a timed-out or failed blame is null, never 0. The line features (mod_line_age_days_*, mod_lines_distinct_authors, mod_lines_self_authored_share, mod_lines_prior_fix_share, hunk_fix_proximity) and the ownership features (file_top_author_share, file_minor_contributor_count, abandoned_lines_share) were written as 0 or 0.0 for a file whose blame timed out, as the architecture already said they would not be (gitio.blame_failed() tells a failed blame from an empty one). SZZ no longer stores a failed blame as "no bug-introducing commit": the (fix commit, path) pair is left out of the blame store and tried again on the next run. A (repo, path) whose blame timed out is not blamed again in that process, so a PR of generated bundles no longer waits 120 s per file (one PR took 966 s). Stop: the SZZ pool is killed on an interrupt instead of draining its queue, and the PR consumer and the SZZ wait use timed waits, because on Windows an untimed result() or wait() is not interrupted by ctrl+c or CTRL_BREAK. Rows mined with a blame timeout under earlier versions have the zeros: --remine-repo them.
7.2.110-29patch: finished repos get their snapshot and commit-feature tables on the next run. At start, after the resume check, the miner lists finished repos that have captures in snapshots_raw/ but no rows in the snapshot or commit-feature table (a batch stopped or ended before assembly), compiles the tables from those captures without re-mining, and syncs them to the team store. The first full sync on 7.2.1 also resends every file once, ignoring .sync_state.json (sync.FORCE_RESEND; a heartbeat or a failed sync does not count, so it repeats until one finishes clean), so the store gets what earlier syncs missed; later syncs are incremental again. Change FORCE_RESEND to force another round. A repo whose captures yield no rows is looked at again each run. Output for existing columns is unchanged.
7.2.211-07patch: a restart no longer redoes the setup stages. The author index, the concurrency counts and the SZZ relevant-commit set are kept in ci_cache/<owner>_<name>.setup.json and reused when the PR list and the clone's history are unchanged (see the setup cache in 4.3). On n8n this was about 2.5 h per start before the first PR. Output is identical.

Contract versions: mined_dataset 1.0 (2026-10-18, miner 2.2.0) → 1.1 (provenance, domain; miner 4.0.0) → 1.2 (outcome_3class, nowhere_reason, ci_fail_flaky_only, author_confirmed_maintainer; miner 5.0.0). mined_snapshot 1.0 (miner 3.1.0) → 1.1 → 1.2 → 1.3 (snapshot_weight, moment_bucket; miner 6.0.0) → 1.4 (miner 7.0.0). mined_commit_features 1.0 (miner 3.1.0) → 1.1 (miner 7.0.0). mined_dataset 1.3 (optional llm_diff_truncated, szz_window_closed; miner 7.0.0). Full table: docs/contracts.md.

Related decisions: ADR 0002 (breakage target), ADR 0005 (feature parsers), ADR 0007 (corpus growth, Proposed), ADR 0008 (feature-map columns), ADR 0009 (merge-outcome target), ADR 0010 (features at open), ADR 0011 (domain base models, Proposed), ADR 0012 (flat form), ADR 0016 (point-in-time CVE data, null-when-unobserved scanners and the 7.0.0 re-mine, Proposed).


22. Glossary

TermMeaning
At openThe PR as it stood at created_at + 300 s (ADR 0010). All diff features are computed from this state.
Base commitThe fork point of the at-open head from the base branch. History features look strictly before it.
Final rangeWhat actually landed. Post-merge labels use it.
Point-in-timeUsing only information knowable at a given moment.
LeakageA feature that contains information from after the moment being predicted.
Row / datasetOne closed PR, features at open plus outcomes.
SnapshotA PR at a later moment while open, labelled with the final outcome.
CaptureThe raw data stored during mining so snapshots can be assembled after clones are gone.
Lead-inPRs just before the mined block, read as metadata only to give early PRs a history.
BlockThe newest contiguous run of --max-prs PRs in a window.
Training PRAn outside non-bot PR, plus a stable 5% sample of the rest.
Outside contributorauthor_association not OWNER/MEMBER/COLLABORATOR and not confirmed from history as a maintainer.
Agent PR / bot PRAI coding agent vs automation account; an agent PR is never a bot PR.
Landed-commit modeA repo whose PRs land outside GitHub's merge button, linked by commit messages.
ReworkA human change request, or commits/force-pushes more than 5 minutes after open.
Rerun-passA check that failed on a commit and passed on a rerun there.
Flaky (at a merge)3+ rerun-passes in the previous 90 days.
SZZFix-tracing: blame the lines a fix changes to find the commit that introduced the bug.
MaturityWaiting long enough (30 days; 90 for SZZ) that an outcome would have shown.
Observed throughHow far forward outcomes could be seen in the clone.
SidecarA private per-row record carried until the repo-wide pass needs it.
Shard / PCOne machine's equal share of the repos.
Split IDFingerprint of the repo list, sizes and PC count; equal IDs mean disjoint shares of the same list.
DomainA corpus category with its own base models (ADR 0011), tagged in the repos file.
CCNCyclomatic complexity number (lizard).
Kamei metricsLA, LD, LT, NF, ND, NS, Entropy, EXP, REXP, SEXP, NUC, NDEV, AGE, FIX: the just-in-time defect prediction feature set.
OSV / GHSAOpen Source Vulnerabilities database / GitHub Advisory Database.