Readability Score Distributions Across Project Gutenberg: A 2026 Extension

In 2007, Dr. Ruth A. Reck and I presented Generating and Rendering Readability Scores for Project Gutenberg Texts at Corpus Linguistics 2007 in Birmingham, UK. The paper computed seven readability measures for each of the 15,511 Project Gutenberg texts then available, rendered every text as an RDF metacard, and plotted the frequency distribution of each measure across the whole collection. Its central observation was not any single score but the shape of those distributions: across the ten curves, three recurring families appeared — that is, the corpus did not spread evenly across the range of a measure but piled up into a few characteristic humps, and the same handful of shapes kept recurring from one measure to the next. This article extends that study to the present, much larger corpus with a pipeline built for reproducibility, adds lexical, structural, sentiment, and emotional-arc measures over the same texts, joins author biography from Wikidata, and ships an interactive tool for exploring the result.

Readability measures

A readability measure reduces a passage of prose to a single number meant to approximate how difficult it is to read. Most of the classical measures were derived in the mid-twentieth century by regressing reading-test performance against surface features of text — chiefly the average length of sentences and the average length of words, whether length is counted in characters or in syllables. A measure like Flesch–Kincaid returns a United States school grade level; Flesch Reading Ease returns a 0–100 score on which higher means easier; Lix and the others sit on their own scales. None of them models meaning, syntax, or a particular reader; they are deliberately shallow, which is precisely what makes them cheap enough to run across an entire corpus. The seven measures used here, with their formulas and sources, are given in Table 1.

The 2007 study

The original study computed the Automated Readability Index, Coleman–Liau, Flesch Reading Ease, Gunning Fog, Flesch–Kincaid, Lix, and SMOG, together with three supporting quantities — characters per word, words per sentence, and syllables per word. Plotted across all 15,511 texts, the distributions fell into three groups. One group was single-peaked and narrow (Coleman–Liau, SMOG, characters per word). A second showed a double extremum — a main peak with a secondary shoulder (the Automated Readability Index, Gunning Fog, Flesch–Kincaid). A third had more complex structure, the Flesch Reading Ease being the broadest and least regular. The paper also recorded directions for later work: distinguishing English from the many other languages in the collection before applying formulas designed for English; reconciling the differing ways tools count words; and recording exactly which files produced a given result so the study could be reproduced. This extension takes up each of those directions.

Corpus and reproducibility

Texts are drawn from a plain-text mirror of Project Gutenberg. Every work is recorded with its ebook identifier and a SHA-256 hash of its cleaned body; that set of hashes is a manifest that fixes the exact input of a run, so any distribution shown here can be traced back to the precise files that produced it. Multiple encodings of the same work are collapsed to one record per ebook identifier. Language is assigned by a stopword-ratio heuristic, which separates the English texts — the only ones to which these formulas properly apply — from the rest. A prose filter then sets aside material that is not running prose: a work must contain at least twenty sentences and yield a plausible grade level, which removes gazetteers, indexes, and tabular reference matter (for example, an almanac with a handful of enormous “sentences”). After these steps 60,787 English-prose works remain, and every figure below is computed over them.

Word counts and the 2007 reconciliation

Every readability formula divides by a count of words, so the definition of a word is not a detail — it determines the result. In 2007 this surfaced directly: for the reference text, Mark Twain’s A Horse’s Tale, three tools returned three different counts. The pipeline states its convention explicitly and reconciles it to the others rather than hiding the difference. A canonical, Unicode-aware tokenizer treats a word as a maximal run of letters with internal apostrophes or hyphens; on the current edition of the text it counts 17,594 words. A whitespace count (equivalent to wc -w) gives 17,511, and an alphanumeric count that also keeps numerals gives 17,650. These conventions differ by small, well-understood amounts. The larger gap — all three of the 2026 counts fall below the figures reported in 2007 — is attributable not to method but to edition: the transcription of A Horse’s Tale served by Project Gutenberg today is not byte-identical to the one processed in 2007. Because every count now carries a content hash, that kind of difference is recorded rather than silent.

The reference text

Run through the pipeline, A Horse’s Tale scores a Flesch–Kincaid grade of 7.2 — roughly a seventh-grade reading level, which is consistent with Twain writing in the voices of a horse and a young girl. The full set of measures, with the formula and original source for each, is below. In the formulas, W is words, S is sentences, Y is syllables, C is letters, C_x is complex words (three or more syllables), and L_w is long words (more than six letters).

Distribution of reading grade

The single most useful summary of the corpus is the distribution of Flesch–Kincaid grade level. Across the 60,787 works the median is 9.1 and the mean 9.4, with the middle eighty per cent falling between grades 5.5 and 13.7. The distribution is right-skewed: most works cluster in the middle grades, with a long tail of more demanding texts — dense philosophy and theology, legal and scientific prose — reaching well above it. The reference text and the mean are marked on the figure.

The three distributional families

Plotting all nine measures — the seven readability scores together with characters per word and words per sentence — reproduces the grouping the original study reported, now over a corpus several times larger. Each panel is scaled independently, because the measures live on different ranges: a Flesch–Kincaid grade runs from roughly 3 to 15, a Lix index from 20 to 60, and Flesch Reading Ease from 0 to 100. Read across the families rather than within a single panel: the single-peaked measures are tight and symmetric, the double-extremum measures show a main peak with a secondary shoulder, and the complex group is broad and irregular.

Correlation among the measures

Because the classical formulas are built from the same two ingredients — sentence length and word length — they largely rise and fall together. The correlation matrix makes this explicit: the six grade-scaled measures are almost perfectly correlated with one another, with a coefficient near +1. The Flesch–Kincaid and SMOG scores, for instance, correlate at about +0.96, which is the quantitative form of the original study’s remark that several of the measures may encode much the same information. The one measure that runs the other way is Flesch Reading Ease, which increases with ease rather than difficulty; it correlates strongly and negatively with the rest (-0.93 against Flesch–Kincaid). That inverse relation is visible directly in the scatter plot, where every work is a point.

Vocabulary and structure

The same tokenized text supports a second family of measures that describe vocabulary rather than difficulty. The type–token ratio is the number of distinct words divided by the total number of words; it falls as a text grows longer, because common words recur, so a length-robust variant — the moving-average type–token ratio, computed over a sliding window — is retained alongside it. Yule’s characteristic constant K measures how concentrated the vocabulary is in a few frequent words; Guiraud’s R, Herdan’s C, and Honoré’s H are further indices of lexical richness, the last weighting words that appear exactly once. Structural measures describe the shape of the prose rather than its vocabulary: mean sentence length and its variability, the rate of commas, questions, and exclamations, and the proportion of one-syllable and three-or-more-syllable words. Together these give each work a fuller fingerprint than a grade level alone, and they are the features on which authorship and topic analysis will later draw. As a check that the underlying token inventory is well formed, the rank–frequency curve of the reference text follows Zipf’s law — a straight line on logarithmic axes — as it should for any natural-language text.

Sentiment and emotion

Sentiment is estimated by lexicon: each word is looked up in a scored word list and the scores are averaged over the text. The primary lexicon is AFINN, which assigns integers from −5 to +5; a value above zero means the average word carries positive affect. Across the corpus the median work is mildly positive (median valence 0.29), with a minority of net-negative works. A notable result is that sentiment is almost unrelated to difficulty: the correlation between mean valence and Flesch–Kincaid grade is only +0.18, so how hard a work is to read tells you very little about its emotional tone. In the correlation matrix above, the sentiment row is near-zero against every readability measure.

The interactive tool carries the sentiment analysis further. The NRC Emotion Lexicon classifies words into eight emotions — anger, anticipation, disgust, fear, joy, sadness, surprise, and trust — so each work also has an emotional profile beyond a single positive-or-negative number. And each book is divided into twenty equal segments, with sentiment computed for each, tracing the emotional arc of the narrative from beginning to end; those arcs can be matched against the small set of archetypal shapes (a steady rise, a fall, a fall-then-rise, and so on) described by Reagan and colleagues. In the tool, an author’s arc is drawn against the corpus average.

Author metadata from Wikidata

Book titles and identifiers come from the Project Gutenberg catalogue; facts about the people who wrote the books come from Wikidata. For each author the pipeline resolves a Wikidata entity and reads birth and death years, gender, and country of citizenship. Where an author’s own dated works are recorded in Wikidata, matching them to the Gutenberg titles yields the author’s age at the time of writing — publication year minus birth year — for the works that can be matched. This makes the corpus queryable along dimensions the text alone does not contain: one can colour any distribution by author gender, nationality, language, genre (by Library of Congress class), or age at authorship, and ask whether difficulty or tone varies with them. Coverage is partial and honestly bounded: not every author is in Wikidata, and not every work resolves to a dated edition, so these dimensions describe the resolvable subset, not the whole corpus.

The interactive tool

The interactive tool holds all of this in one place. Search for an author and their works appear as coloured points on the full distribution of every measure, with a line at that author’s mean and the corpus median for reference; add a second author and the two are compared head-to-head as a profile of standardised scores. The emotional arcs of the selected authors are drawn against the corpus average, and the distributions can be split by any of the metadata dimensions. Each panel carries a short explanation of how to read it.

Method and future work

The pipeline is written in the Python standard library: it cleans Project Gutenberg boilerplate, tokenizes, computes the seven readability measures and the supporting quantities, the lexical and structural features, and the sentiment and emotion measures, and writes each work as an RDF metacard and as a row in a normalised database. Every value is bound to a content hash of its source text and to the run’s manifest. From here the work continues along the same lines: computation over the complete corpus rather than the current snapshot, a fuller catalogue join for analysis by period and subject, and authorship and topic analyzers built on the vocabulary and structural features already computed.

Acknowledgment. This work extends and is indebted to the 2007 study conducted with Dr. Ruth A. Reck, whose method and framing it carries forward.

References

Björnsson, C. H. (1968). Läsbarhet. Stockholm: Liber.

Coleman, M., & Liau, T. L. (1975). A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2), 283–284.

Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221–233.

Gunning, R. (1952). The Technique of Clear Writing. McGraw-Hill.

Kincaid, J. P., Fishburne, R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of New Readability Formulas for Navy Enlisted Personnel. Naval Air Station Memphis.

McLaughlin, G. H. (1969). SMOG grading: A new readability formula. Journal of Reading, 12(8), 639–646.

Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3), 436–465. (NRC Emotion Lexicon.)

Nielsen, F. Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. ESWC Workshop on Making Sense of Microposts, 93–98. (AFINN.)

Reagan, A. J., Mitchell, L., Kiley, D., Danforth, C. M., & Dodds, P. S. (2016). The emotional arcs of stories are dominated by six basic shapes. EPJ Data Science, 5(1), 31.

Reck, R. P., & Reck, R. A. (2007). Generating and Rendering Readability Scores for Project Gutenberg Texts. Corpus Linguistics 2007, Birmingham.

Smith, E. A., & Senter, R. J. (1967). Automated Readability Index. Wright-Patterson AFB.

Zipf, G. K. (1949). Human Behavior and the Principle of Least Effort. Addison-Wesley.