Extracting Value from Existing Structure: Leveraging Past Decisions to Predict Future Ones

Organizations accumulate archives of completed judgments: terms filed into a taxonomy, documents routed to departments, tickets resolved, claims approved or denied, records matched or rejected. Each entry pairs an input with a decision a qualified person already made. Viewed as history, these archives are inert. Viewed as data, they are something more useful: a labeled evaluation set whose labeling cost has already been paid.

That structure can be made to yield value a second time. Take a candidate automated procedure, run it against the historical inputs, and compare its outputs to the decisions on record. The procedure never touches live work, yet the exercise answers the question that matters most before adoption: what would it have done?

An archive of past decisions reframed as a labeled evaluation set: each input-decision pair becomes an example with ground truth, at zero labeling cost

The method

The replay is straightforward and repeatable:

Pin the inputs. Take a versioned snapshot of the historical records — the inputs exactly as they were when the original decisions were made, and the decisions themselves as ground truth.
Pin the procedure. Fix the version of the automated method under test: the model, the scoring function, the configuration. An unversioned test cannot be rerun or compared.
Replay. Run the procedure over every historical input, producing its full output — not only its top choice but its ranking of alternatives.
Score agreement. Measure how often the procedure’s output matches the recorded decision: exact agreement (top-1), agreement within a shortlist (top-k), and mean reciprocal rank of the correct answer.
Read the disagreements. Every mismatch is one of two findings, and both have value: either the procedure erred — a failure mode to catalog — or the historical record erred, and the replay has surfaced a defect in the archive itself.

The replay loop: pin inputs, pin procedure, replay, score with top-1/top-k/MRR, read mismatches — rerun on every procedure revision

What the replay buys

An accuracy figure before any operational risk. The procedure is scored against real inputs and real decisions, not synthetic benchmarks, and the score is obtained without exposing a single live case to automation.

A failure-mode inventory. Aggregate the disagreements and patterns emerge: categories the procedure confuses, input types it handles poorly, vocabulary it does not cover. This inventory defines the review protocol a production deployment needs.

A calibrated trust boundary. Agreement rates broken out by category or input type show precisely where the procedure can operate with light review and where a human must remain in the loop. The boundary is measured, not asserted.

Illustrative bar chart of replay agreement by category with a 90 percent threshold separating light review from human-in-the-loop

Audit of the archive itself. Replays regularly surface inconsistencies in the historical record — decisions that contradict each other or the written guidance. The validation exercise doubles as a data-quality audit.

Every mismatch resolves to one of two findings: a procedure failure mode to catalog, or a defect discovered in the archive itself

Zero labeling cost. The dominant expense of evaluating supervised systems is producing ground truth. Here the ground truth already exists; the marginal cost of the replay is compute, which for most procedures is trivial.

A concrete case

The DTIC term-categorization work illustrates both halves. In 2016, 1,400 new thesaurus terms were categorized against 272 Field & Group categories by scoring every term–category pair with WordNet and presenting a ranked shortlist for human election — 380,800 comparisons in about 4.3 hours of machine time.

The validation step now scoped for that procedure is exactly the replay described above: sample terms whose F&G categorizations are already accepted, run the same scoring pipeline against them, and measure top-1 accuracy, top-8 shortlist recall, and mean reciprocal rank, broken out per category. The archive of previously categorized terms — structure built by hand over years — becomes the instrument that measures the automation proposed to extend it. No new labeling, no operational exposure, and the output is a defensible number for how good the assistance is.

Scoring one replayed term: ground truth at rank 3 means top-1 misses, top-8 hits, and reciprocal rank is one third

Requirements

The replay is only as sound as its controls:

Versioned snapshots of both the input records and the decision record, so results pin to a known state.
A pinned procedure, under configuration management, so the test is reproducible.
A disagreement protocol — sampled human review of mismatches to attribute each one to procedure error or record error, rather than assuming either.
Honest reporting of coverage — inputs the procedure cannot process (out-of-vocabulary terms, malformed records) are reported as such, not silently dropped.

The strategic point

Replay-before-deploy belongs on any AI adoption checklist, for one structural reason: it converts an organization’s past labor into its evaluation infrastructure. Every archive of completed decisions is a benchmark the organization already owns. Before an automated procedure is trusted with future work, it can be made to sit the examination the organization has been writing, one decision at a time, for years — and the grade determines what to do next.

Written July 2026