· 4 min read
The Language of Surrender: Phrase Improvement Measurements
The trigger vocabulary that finds surrender statements was grown and tested: 373 mined candidates, 45 tested against ten labeled sources spanning four tiers of label quality, 9 promoted on human gold. Three measured regularities came out: precision rises with phrase length (0.603 at two words to 0.771 at five, over 134,600 decided spans); first-person forms separate self-acknowledged limitations from reviewer criticism (0.955 versus 0.000 for the same phrase family); and mining scores do not predict trigger quality. Every number is a measurement with its label tier stated.
Third in a series. The first article measured classifier models against a limitation gold set; the second measured larger models. This one measures the other half of the system: the trigger vocabulary itself — the phrases that propose candidate surrender statements before any model sees them.
Starting point
The vocabulary under test: 1,576 surrender phrases, 1,128 currently matchable, each carrying a per-phrase precision measured over the spans a classifier has ruled on. Its measured weaknesses: precision 0.23 on its home calibration corpus, recall 0.054 on clinical text, 0.016 on AI-conference text — the phrases are register-bound.
Length governs precision
The vocabulary’s own record, 134,600 decided spans across 462 measurable phrases with two to five words:
The same effect appears at the level of individual phrase families on human gold: extending a two-word trigger by one word raised gold precision in all four measured pairs.
Candidate generation and testing
Candidates were mined by log-odds contrast (Dirichlet prior, z-scored) between limitation-bearing and other text, per register: 265 candidates from a 1–3-gram pass, 108 from a 3–6-gram pass. 45 were tested — six agents in a first round of 30, then 15 re-tested comprehensively — against ten labeled sources with four tiers of label quality: human gold (Kilicoglu 2,257 sentences; SAL 952; Challenges & Directions 2,894), distant labels (BAGELS section headers; ART expert zones; OA-STM machine zones), machine labels (SAL large-scale predictions; ORION estate screen), and third-party silver (ASAP reviewer spans; LimitGen reviewer limitations).
The self versus third-party axis
The matrix’s strongest structure: first-person phrases score 0.75–1.00 on self-acknowledged gold and 0.000 on reviewer spans; reviewer-frame phrases invert.
“we did not” matched 92 reviewer spans and none was a positive; “the proposed method” matched zero gold positives in 2,257 sentences and runs 0.92–0.95 on reviewer spans. The two vocabularies do not overlap: a trigger set for self-acknowledged surrender and a trigger set for reviewer criticism are separate artifacts.
Mining score does not predict quality
Candidates with mining z-scores of 33–54 were rejected wholesale — they were checklist boilerplate, reviewer deixis, or fragments of existing triggers (five estate candidates overlapped the existing “beyond the scope” trigger at 59–92%). The nine promotions came from candidates with modest z-scores validated on human gold. The promotion gate is gold precision; the mining score is only a candidate generator.
Register transfer and the mining lift
The vocabulary’s recall by register, and the effect of adding 120 register-mined phrases:
Promotions
Nine phrases cleared human gold with multi-tier agreement:
| phrase | gold P (n) | strongest secondary evidence |
|---|---|---|
| we did not | 0.955 (22) | SAL gold lift 19.1×; BAGELS 4.14× |
| may not be | 1.000 (6) | OA-STM 8/8; BAGELS n=797 at 2.76× |
| limited by the | 1.000 (3) | SAL gold lift 25.8×; Challenges 1.000 |
| we were not | 1.000 (2) | SAL gold lift 23.9×; BAGELS 4.17× |
| could not be | 0.750 (8) | OA-STM lift 3.40×; ART 2.18× |
| it is possible | 1.000 (4) | SAL gold lift 16.3× |
| may have | 0.821 (28) | first-round promotion |
| were not | 0.677 (31) | first-round promotion |
| not be | 0.680 (25) | first-round promotion |
Eight further phrases are held for context rules (usable only with attribution or mitigation guards), and 28 were rejected or identified as fragments of existing triggers.
Measurement status notes
- Gold-tier match counts for several promoted phrases are small (n=2–8 on Kilicoglu gold); the multi-tier agreement is the basis for promotion, and per-phrase precision will accumulate from the classifier-ruled spans after the next vocabulary epoch.
- Distant, machine, and silver tiers are labeled as such throughout; none is treated as accuracy.
- Estate-register human gold does not yet exist; the OA-STM and ART figures are zone-based distant measurements.
References
- Kilicoglu, H., et al. (2018). JAMIA 25(7), 855–861; dataset Dryad doi:10.5061/dryad.06ds7.
- Lan, M., et al. (2024). J Biomed Inform 152, 104628; corpus github.com/MengfeiLan/SAL_Type_Classification.
- Lahav, D., et al. (2022). A Search Engine for Discovery of Scientific Challenges and Directions. AAAI.
- Al Azher, I., et al. (2025). BAGELS. arXiv:2505.18207.
- Yuan, W., Liu, P., Neubig, G. (2022). Can We Automate Scientific Reviewing? JAIR (ASAP-Review).
- Xu, Z., et al. (2025). LimitGen. arXiv:2507.02694.
- Liakata, M., et al. (2010). ART Corpus, LREC; Elsevier Labs OA-STM Corpus.
- Monroe, B., Colaresi, M., Quinn, K. (2008). Fightin’ Words. Political Analysis 16(4).
- Earlier articles in this series: model selection, larger models.
