Esc
↑↓ to move↵ to openEsc to close

DocsConcepts

Labels and maturity

What counts as a bad change, and when an outcome is old enough to learn from.

Proposed

A model can only learn what it is shown. The labels decide what "a bad change" and "a change that won't land" mean, and maturity decides when an outcome is old enough to count.

Riffle learns two things from the same mined history, and never mixes them into one label:

TargetQuestionTrained on
label_badWill this merged change break?Merged pull requests
label_mergedWill this pull request merge at all?Every closed pull request, plus ones abandoned while open

A combined risk, when wanted, is P(merge) × P(bad | merged). Keeping them apart matters for triage: "won't merge" means don't spend a senior reviewer on it; "will merge but break" means do.

Breakage: label_bad

text
label_bad = revert OR ci_fail OR hotfix
SignalFires whenWindow
revertA later commit on the default branch reverts the pull request (it names the merge commit, has #N in its subject or Reverts owner/repo#N in its body, or quotes the exact title); the pull request's own commits never count30 days
ci_failRequired checks fail in the merge commit's push-triggered workflows; auxiliary and flaky checks, and issue or schedule workflows, excludedat merge
hotfixA later commit worded as a hotfix, rollback, quick fix or urgent fix touches the same files7 days

SZZ bug-inducing links and follow-up fixes are mined as well, but are not part of the target; both are reported as ablations. They are null, not false, when SZZ did not run.

Merge outcome: label_merged

label_merged is whether merged_at is set. The state field is not used, because "closed" covers merged pull requests too. Around it, the miner records why a pull request did not land, and whether one that did needed rework:

LabelRowsFires when
label_reworkmergedA human reviewer requested changes, or commits or force-pushes came after opening (merges from the base branch don't count)
label_duplicatenot mergedMarked as a duplicate, or closed as superseded or already fixed
label_unwantednot mergedMaintainers declined the change itself: out of scope, against policy, rejected outright
label_abandonednot mergedNo human response before closing, closed by a stale bot, or still open after 60 days with no response
label_ci_technicalnot mergedRequired checks were failing at close, with no maintainer objection

close_reason names one of these per pull request. With an LLM available, a classifier reads the title, comments, reviews, labels and who closed it, which catches cases rules miss, such as a maintainer silently closing a pull request and retitling it "AI spam". Without one, phrase and label rules decide. The classifier only labels training data; it is never on the scoring path.

Maturity

An outcome is only usable once the window for it to happen has closed.

Label sourceMatures after
revert, ci_fail, hotfix30 days
SZZ90 days
label_merged and close reasonsAt close
Abandoned while open60 days without a response

Pull requests younger than their window are dropped from training, never counted as good.

What the signal looks like

MeasureRateSource
Revert rate in open source1–5% of commitsShimagaki et al., ICSME 2016
SZZ bug-inducing rateabout 26%ApacheJIT, MSR 2022

Point in time

Features and labels are point-in-time: a pull request only sees data from before it opened, and only labels that were already mature at that moment.

That includes the pull request itself. It is scored when it opens, so its features come from its first push, not from the version that eventually merged: commits pushed under review would otherwise teach the model about rework it cannot see yet. A force-push after opening is undone from GitHub's record of the replaced commit. The title and milestone are taken as they were at opening too, since maintainers retitle rejected pull requests and milestone accepted ones. null means not declared or not enough history; 0 means measured and found none. The two are never coerced into each other.