Automated Categorization of Terms in the Defense Technology Information Center's Field and Group Taxonomy

In 2016 the Defense Technical Information Center (DTIC) was adding new terms to its thesaurus. Every new term had to be placed against the same 272 Field & Group (F&G) categories used by every existing term: DTIC’s search precision depends on those categories, and its users expect them. 1,400 terms required categorization.

Historically the work had been done by hand. Categorizing a single term is straightforward; categorizing 1,400 of them, each weighed against 272 candidate categories, is the same act of judgment repeated at a scale that consumes months. The objective of this effort was to determine whether a computer could carry the repetitive measurement and leave the final judgment to a human.

The shape of the problem

The cost of the task is combinatorial. Categorization is not 1,400 decisions; it is 1,400 terms each measured against 272 categories — 380,800 term–category judgments in all. Performed properly, every new term must be evaluated against each category and its scope notes. Most comparisons end in a negative, but none can be skipped.

A manual approach pays for every cell of that grid. A machine can score every cell, rank the results, and present only the highest-scoring candidates for review. The task is inherently supervised: the computer measures, and a human decides.

How the run worked

DTIC data is expressed in RDF, so every term and every F&G category has two forms: a human-readable label and a machine identifier (a URI — more precisely an IRI). This duality is what a semantic-similarity approach requires.

The scoring engine was WordNet, the Princeton lexical database — an open-source lexical database of English words and the sense relations among them. The precise list of terms requiring categorization was not provided, so the first step was to load a version of the thesaurus, write a query to isolate the terms, and proceed on a best-effort basis. The query returned 1,400 terms.

A program paired the human-readable form of each term with each F&G category and queried the University of Maryland, Baltimore County (UMBC) Semantic Similarity Service — a WordNet-backed scoring endpoint — for each pair: 380,800 term–category pairs in total. Each term’s roughly 270 category comparisons returned in about 11 seconds on average, just under 25 measurements per second. The machine measured the entire 1,400 × 272 surface in approximately 4.3 hours. The measurements were ranked, and a human reviewed the rankings to elect the category (or categories) that best fit each term.

Results

The work proceeded in two stages: a pilot on 30 June 2016, three days after tasking, followed by the production pass over the full 1,400 terms.

The pilot scored eight deliberately varied terms — weapon, sun, sandwich, horizon, silo, truck, luger, pluto — against the complete category list, writing one similarity score per category per term. The raw result files survive in the project archive, and they show exactly why the method works and where it needs the human:

The strong hits confirm the approach; the failures define the review protocol. WordNet places all vehicles near one another, so truck ranks aviation categories highly; proper nouns absent from WordNet score zero everywhere and must be filed entirely by hand. These are precisely the cases the ranked-shortlist review exists to catch.

The production script — preserved in full as Appendix C of the August 2016 whitepaper — extended the pilot in three ways. It consumed the reconstructed term list with each term’s IRI and the category records with their scope notes, it measured every pair under two approaches (category label alone, and label plus full scope note — roughly 750,000 queries over the 380,800-pair surface), and it emitted its output directly as RDF: triples asserting sc:categoryLexeme between category and term IRI, the same predicate already carrying the 13,352 existing term-to-category assignments in DTIC’s Subject Categories model. Several WordNet::Similarity measures were evaluated for the scoring function — path, resnik, vector, and Wu–Palmer — with Wu–Palmer judged most consistent with existing F&G practice.

Delivery followed in three sets: an initial set of categorized terms on 21 July 2016, a second on 29 July, and the final set on 4 August 2016 — five and a half weeks from tasking to final delivery. The delivered assignments themselves reside with DTIC; what the project archive preserves is the pilot data, both scripts, the category list, and the customer’s mapping ontologies.

A man-year in an afternoon

Consider the same 380,800 comparisons performed by a person. A considered term–category judgment — read the term, weigh one category, check its scope note, decide — takes an estimated 20 to 40 seconds. The estimate is conservative: experienced catalogers are faster on clear cases and slower on difficult ones. At that rate the manual pass requires 2,116 to 4,231 hours.

A man-year of labor is approximately 2,080 hours. This single run therefore stood in for roughly one to two man-years of exact, repetitive human judgment and completed in an afternoon — a speedup of roughly 500× to 1,000×. Accounting for real productive hours only increases the calendar cost of the manual approach.

Every figure in the calculation except one comes directly from the project: the 1,400 terms, the 272 categories, the 11 seconds per term. The single assumption is the 20–40 seconds per human comparison, and it is conservative — shortening it only widens the gap.

What the human still does

The assistance changes what the reviewer reads, not who decides. Without assistance, filing one term means evaluating it against all 272 categories. With assistance, the same term arrives with a short list already ranked by likelihood, and the reviewer confirms or adjusts.

The trade is direct: 4.3 hours of machine time scores the entire surface so that human review is spent on a ranked shortlist of about eight candidates per term instead of 272, with candidates that arrive sorted rather than shuffled. The judgment remains human. The repetition goes to the machine.

Limitations

Three factors limited this run, and all three are remediable:

• No rigorous term list. The exact set of terms needing categorization was never specified, so it had to be reconstructed from a query — best effort, not ground truth.
• No versioning. There were no fixed versions for the thesaurus or the category list, so results could not be pinned to a known snapshot.
• No feedback loop. The schedule left no room to review decisions with stakeholders mid-task. Probability of success would have improved given an example of the required deliverable format and the IRIs for the F&G categories.

Where it goes next

The 2016 run was a single best-effort pass. Each of the following steps makes the next run cheaper, more accurate, and repeatable.

Respond to customer feedback, then repeat the task with precise, versioned input: an explicit list of terms with their IRIs, and the F&G categories with their IRIs and scope notes, all under configuration management. Bring WordNet in-house — it is open source and free, and depending on an outside service for a core capability is an unnecessary dependency. From there, extend WordNet with DTIC’s own terms and relationships through its programming APIs, and eventually stand up a full DTIC version of WordNet, following the approach used to build WordNet libraries for other languages — inheriting the ecosystem of tools the community has produced.

Two further steps establish reliability. Measure accuracy by running the categorizer against terms that were already categorized and scoring how often it agrees, producing a measured accuracy figure. And because RDF is a staple of DTIC’s data, build a repeatable RDF ⇄ WordNet workflow, so converting the thesaurus into a standalone, queryable lexical resource becomes a known process instead of a one-off.

Conclusion

The general result is a division of labor: computers are the appropriate tool for tedious, repetitive measurement; humans are the appropriate tool for judgment. A single afternoon of WordNet queries did not decide where 1,400 terms belonged — it scored the comparison space so that a person could review a short, ranked list. The measurement scales; the judgment stays with the reviewer.

Computer-Assisted Term Categorization — DTIC, 2016 · written up July 2026 — 8 min read