The Language of Surrender: Model Selection Observations

Scientific papers record their own defeats in plain language. A study that could not enroll its calculated sample says so. An analysis that was computationally infeasible in 1996 says so. A cohort that could not be obtained at reasonable cost says so. These statements — surrender phrases — mark the places where a line of work stopped for reasons that may no longer hold.

They are the raw material of ORION, a program built on the premise that scientific literature can operate as humanity’s largest labeled dataset. Past research contains latent value that was bounded by the computational limits, data availability, and instrumentation of its time, and modern computation can recover it. ORION maintains an estate of 1,560,771 known documents — 1,028,640 indexed in full text — scans them for constraint statements, classifies each admitted span into one of seven inertia types, and scores every paper on six factors for whether the obstacle that stopped it could be relieved today. The estate currently holds 803,739 located candidate spans and 16,312 provisionally ranked revival candidates, published live with its methods and counts at orion.rrecktek.com. The work in this article concerns one joint in that pipeline: the step that decides whether a located span is a genuine surrender, because every downstream score inherits the quality of that decision.

Finding the statements is the easy half. Deciding whether a matched phrase is a genuine, self-acknowledged, still-relevant constraint is the hard half, and the difference between the two halves is measurable: a factor of four in precision. This article documents the measurement, the method that closes the gap, the model selection behind it, and the numbers at each stage.

Contents

1. The problem
2. Prior work
3. The gold data
4. One vocabulary, two dialects
5. Three lanes
6. Five models, one prompt
7. Two judges together
8. Context in the prompt
9. The cascade
10. Fit into ORION
11. Reproduction
12. References

The problem

ORION locates candidate constraint statements by surface matching: a controlled vocabulary of 1,576 surrender phrases (1,128 currently matchable triggers) run against 1,028,640 full-text documents, yielding 803,739 located spans. Measured on a 100-paper curated calibration run against a hand-labelled gold set, the vocabulary scores precision 0.23 at recall 0.82. The phrases catch nearly every real constraint and admit three false matches for each true one.

A trigger phrase alone cannot answer the questions that decide a match:

• Whose constraint is it? “These studies have certain limitations” attributes the defeat to someone else’s work.
• Is anything stated? “Our study has a number of limitations” announces limitations and states none.
• Is it being argued away? “However, an increase in sample size is unlikely to change our findings” is mitigation, not surrender.
• Is it negated or hedged? “It is now possible” contains the same trigger as “it was not possible.”

Each of these distinctions is a property of the sentence and its position in the document — context — and none is visible to a phrase list.

Prior work

Clinical research informatics solved a version of this problem and published everything. Kilicoglu, Rosemblat, Malički, and ter Riet (2018) built a recognizer for self-acknowledged limitation sentences in clinical publications: 8,431 sentences from 1,197 PubMed Central articles, 2,257 of them labelled by four annotators (Krippendorff’s α = 0.781), and three methods compared on a fixed train/test split. Their rule-based system — limitation zones derived from section structure, then sentence-type rules that reject announcements, mitigations, and citation sentences — reached precision 0.758, recall 0.848, F1 0.800 on the 1,505-sentence test split, beating supervised machine learning trained on the same data. Their best configuration used the rules to label 6,174 unlabelled sentences and trained an SVM on the expansion: 91.9% accuracy.

The motivating literature is a research-transparency lineage: Ioannidis (2007) found only 17% of articles in six top journals used even one word denoting limitations; ter Riet et al. (2013) found 27% of 300 biomedical articles reported none. The follow-on work is current: a 2024 typed-limitation corpus with BERT classifiers (Lan et al.), and 2025 benchmarks for extracting limitations from scholarly text (BAGELS) and for testing whether large language models can identify unstated ones (LimitGen).

One negative result from that literature carried directly into the model selection below: biomedical fine-tunes of small models score below their generalist base models on unseen clinical classification — OpenBioLLM-8B at 44.9% against Llama-3-8B-Instruct’s 74.1%, BioMistral-7B at 62.8% against base Mistral’s 69.9% (Dorfner et al., 2024). Domain-tuned models were excluded on that evidence.

The gold data

The Kilicoglu dataset is public (Dryad, doi:10.5061/dryad.06ds7) and its authors published their rule system and labels (limitation-recognizer). It provides:

• SEED: 752 human-labelled sentences (164 limitation, 588 not) — the training set.
• TEST: 1,505 human-labelled sentences (303 limitation, 1,202 not) — the evaluation set. Every number in this article that is not explicitly marked otherwise is measured on this split, in full.
• SEMI: 6,174 sentences with the authors’ automatic labels — the expansion pool.

Sentences carry their section name, and each sentence is labelled POS only if it states a specific limitation of the authors’ own study. The announcement sentence, the mitigation sentence, and the other-study attribution are all NEG by the annotation guideline — the guideline encodes exactly the context distinctions listed above.

One vocabulary, two dialects

The first measurement ran ORION’s vocabulary, unmodified, against the 2,257 human-labelled sentences: word-boundary matching, case-insensitive, longest phrase first.

Precision 0.833, recall 0.054. The vocabulary that recalls 82% of constraints in its home corpus recalls 5% of limitation statements in clinical text. The phrases of surrender in physics and engineering — “beyond the scope,” “computationally infeasible” — and the phrases of surrender in medicine — “slow accrual,” “loss to follow-up” — are different dialects of the same speech act. The consequence for system design: the context rules (announcement, mitigation, attribution, negation) are properties of scientific rhetoric and transfer across fields; the trigger vocabulary does not, and each field needs its own, measured separately.

Three lanes

Three independent adjudication lanes were built on the gold data, following the structure of the 2018 paper:

1. Lexical: tf-idf over word 1–2-grams plus ten binary context features (section zone, announcement, contrastive opening, bracketed citation, other-study attribution, self-reference, itemization marker, limitation lemma, vocabulary-match count), trained as a linear SVM on SEED plus the authors’ 6,174 expansion labels.
2. Embedding: the same classifier over 768-dimension sentence embeddings from an on-prem embedding service.
3. SLM judge: a small language model classifies each sentence POS or NEG from a few-shot prompt that encodes the annotation guideline — the announcement, mitigation, and attribution examples are in the prompt as worked NEG cases. First model: Mistral-7B-Instruct-v0.3, Q4_K_M quantization, 1,505 judgments in 221 seconds on one 16GB GPU.

The lanes fail on different sentences. Of the judge’s 152 missed limitations, the lexical lane catches 104; of the lexical lane’s 126 false positives, the judge shares 5. Union buys recall, intersection buys precision, and the choice between them is an operating point, not a compromise.

Five models, one prompt

The judge lane is the swappable part: it depends only on an HTTP endpoint. Five candidate models in the same size class (7–9B parameters, Q4_K_M quantization, all runnable on the same 16GB GPU) were measured with the identical prompt on the identical split. Selection of candidates followed published instruction-following and classification comparisons; the two domain-tuned biomedical models were excluded on the negative evidence cited above.

Qwen3-8B found 62 more true limitations than the Mistral baseline at a cost of 14 additional false positives. The two collapsed models illustrate a deployment fact rather than a model fact: the serving stack applies one fixed chat template to every model, and models far from that template lose the instruction. The stability of the winner was checked by repetition: three full passes of Qwen3-8B over the split produced identical confusion matrices — 213 true positives, 27 false positives, 90 misses each time. At the serving temperature of 0.1 the judge is deterministic, so the reported numbers carry no run-to-run variance.

Two judges together

The top two models come from different families, and their errors are measurably decorrelated.

The marginal contribution of the second model is measurable on both sides of the ledger. Added to Qwen3-8B, Gemma-2-9B contributes 15 true limitations the leader misses at a cost of 2 additional false positives — recall rises from 0.703 to 0.753 and F1 from 0.785 to 0.814, a gain no single measured model reaches. In the other direction, requiring the two models to agree cuts false positives from 27 to 10 — a 63% reduction — lifting precision from 0.888 to 0.945. One model sets the floor; the second model buys, from the same hardware class, both the highest recall and the highest usable precision in the study.

The union of two sentence-level judges — no document structure, no section information — measures F1 0.814 on the split where the published rule system, with full access to section sequence and paragraph boundaries, measures 0.800. The intersection is the useful gate: at precision 0.945 it is wrong once in 18 admissions, against the vocabulary’s once in 1.3.

The deployment is one judge per GPU host: Qwen3-8B on an RTX 5070 Ti, Gemma-2-9B-it on an RTX 4080 SUPER, each behind the same agent API, processing papers at whatever rate the hardware sustains. Throughput is a secondary concern; at 248 seconds per 1,505 sentences, a single host judges roughly half a million candidate sentences per day.

Context in the prompt

The gap between the two-judge union (0.814) and a hypothetical ceiling invited one obvious experiment: the published rule system earned most of its F1 from document structure — limitation zones derived from section titles and paragraph starts — so the judge prompt was extended to carry that structure. Each TEST sentence was re-judged with its section name and its true neighboring sentences on both sides, recovered from the authors’ parsed article XML (context present for 99.8% of the split), with the instruction to judge only the target sentence.

The result is negative, in a specific and instructive way.

With context in view the model misses only 18 of 303 limitations — the highest recall of any single system measured — and produces 422 false positives, collapsing precision to 0.403. The failure mode is context bleed: inside a limitation zone, the model credits the target sentence with limitations its neighbors state, the judge-only-the-target instruction notwithstanding. Used as a gate on the existing pair, the context judge improves F1 by 0.0008 — below noise — at the cost of a third model pass per span.

The measurement narrows the earlier hypothesis rather than merely rejecting it. The 2018 rule system used document structure to constrain which sentences are candidates; this experiment used it as evidence shown to the classifier, and the two are not the same operation. Structure as a filter before judgment remains untested. What the experiment leaves behind is a calibrated recall net: one model, one pass, 94.1% of true limitations — a known operating point if a high-recall proposal stage is ever worth a judge-priced pass.

The cascade

The measured operating points assemble into a triage cascade, and the cascade’s outputs feed the system that produced them.

• The vocabulary proposes: high recall, precision 0.23.
• The cheap lanes (rules, lexical, embedding) score every span at negligible cost.
• The two judges work the disputed band. Both-POS admits at 0.945 precision. Both-NEG rejects. Disagreement goes to an analyst.

Three feedback paths close the loop. Admitted and rejected spans become expansion labels that retrain the cheap lanes — the same mechanism that added 0.094 F1 to the lexical lane on gold data. Admitted spans, stratified by field, feed phrase mining: candidate n-grams become trial vocabulary entries, promoted or retired by measured per-phrase precision, versioned by epoch. Analyst decisions on the disagreement band — the highest-value labels in the system — extend the human gold store.

One rule governs the loop: judge output never scores judges. Machine labels are training signal, marked as such in provenance. The scoring gold stays human-labelled and frozen per epoch. Without that separation the loop grades its own homework, which is the failure mode the surrounding measurement program exists to prevent.

Fit into ORION

ORION’s pipeline runs document intake, span location, classification, scorecard computation, and ranking over an estate of 1.56 million known documents. The work described here slots between span location and classification: every one of the 803,739 located spans is a candidate for the cascade, and the estate’s 16,312 provisional revival candidates are downstream consumers of the admission decisions. The measured transfer gap sets the vocabulary roadmap — one shared context grammar, one trigger vocabulary per field, each with its own measured precision floor — and the 96-paper curated screen remains the human gold against which estate-scale performance is scored.

Reproduction

Data: the Dryad dataset (doi:10.5061/dryad.06ds7), the authors’ limitation-recognizer with labels and parsed XML, and CONSORT-TM. Models: Q4_K_M GGUF quantizations of the six models, served by llama.cpp behind a FastAPI agent exposing POST /job; hot-swapping is one POST /config carrying a HuggingFace repo and filename. Evaluation: word-boundary phrase matching, scikit-learn linear models, and a crash-safe judge harness that appends one JSON line per judgment and resumes on restart. A full model measurement is one config swap plus roughly four minutes of judging. The complete reference collection — 22 papers in 7 strands, with hashes — is published on the ORION working page.

References

1. Kilicoglu, H., Rosemblat, G., Malički, M., & ter Riet, G. (2018). Automatic recognition of self-acknowledged limitations in clinical research literature. JAMIA 25(7), 855–861. doi:10.1093/jamia/ocy038
2. Kilicoglu, H., et al. (2018). Data from: Automatic recognition of self-acknowledged limitations in clinical research literature. Dryad. doi:10.5061/dryad.06ds7
3. ter Riet, G., Chesley, P., Gross, A. G., et al. (2013). All that glitters isn’t gold: a survey on acknowledgment of limitations in biomedical studies. PLoS One 8(11), e73623. doi:10.1371/journal.pone.0073623
4. Lan, M., Cheng, M., Hoang, L., ter Riet, G., & Kilicoglu, H. (2024). Automatic categorization of self-acknowledged limitations in randomized controlled trial publications. Journal of Biomedical Informatics 152, 104628. doi:10.1016/j.jbi.2024.104628
5. Ioannidis, J. P. A. (2007). Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol 60(4), 324–329. doi:10.1016/j.jclinepi.2006.09.011
6. Vincze, V., Szarvas, G., Farkas, R., Móra, G., & Csirik, J. (2008). The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioinformatics 9(S11), S9. doi:10.1186/1471-2105-9-S11-S9
7. Farkas, R., Vincze, V., Móra, G., Csirik, J., & Szarvas, G. (2010). The CoNLL-2010 Shared Task: Learning to Detect Hedges and their Scope in Natural Language Text. ACL W10-3001
8. Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001). A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 34(5), 301–310. doi:10.1006/jbin.2001.1029
9. Teufel, S., & Moens, M. (2002). Summarizing Scientific Articles: Experiments with Relevance and Rhetorical Status. Computational Linguistics 28(4), 409–445. ACL J02-4002
10. Teufel, S., Siddharthan, A., & Tidhar, D. (2006). Automatic classification of citation function. EMNLP 2006. ACL W06-1613
11. Liakata, M., Saha, S., Dobnik, S., Batchelor, C., & Rebholz-Schuhmann, D. (2012). Automatic recognition of conceptualization zones in scientific articles. Bioinformatics 28(7), 991–1000. doi:10.1093/bioinformatics/bts071
12. Dorfner, F. J., et al. (2024). Biomedical Large Language Models Seem not to be Superior to Generalist Models on Unseen Medical Data. arXiv:2408.13833
13. Xu, Z., et al. (2025). Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers. ACL 2025. arXiv:2507.02694
14. Al Azher, I., et al. (2025). BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text. arXiv:2505.18207