Scoring the Fine Print: A Consumer-Issue Audit of Sixty-Four Agreements

Abstract

Measuring a terms-of-service document by its length answers the wrong question. Length rises and falls with boilerplate, localization, and formatting; it says nothing about which obligations the document imposes or which rights it grants. This audit takes a fixed catalog of 34 consumer issues — arbitration, content licensing, termination, data sharing, and so on — and checks each one against three real documents: YouTube’s Terms of Service (July 2025), Reddit’s User Agreement (December 2018), and Twitter’s Terms of Service (July 2016). Each issue in each document is scored three ways: similarity (how closely the document’s wording matches the issue), length (how many words the document spends on it), and sentiment (how harshly the clause reads). YouTube addresses 29 of the 34 issues, Reddit and Twitter 25 each. The scoring, the ranking, and the reading are all derived from the text itself, using the same 768-dimension embedding service that powers the rest of this corpus. Every part of the pipeline runs on-premises at fixed cost — no per-call API fees, no text leaving the building, and no metered charge that scales with the size of the corpus.

From size to coverage

An earlier visualization on this site plotted how three terms-of-service documents changed in size over two decades. It was honest about one thing — that these documents mostly grow — and silent about everything that matters to a reader. A document can double in length without adding a single new obligation, and it can be rewritten end to end while its word count barely moves. Reddit’s agreement ended 2018 almost exactly the size it began, having replaced most of its wording along the way.

The fix is to stop measuring the container and start measuring the contents. That requires a fixed list of things a consumer agreement can say, so that every document is checked against the same yardstick.

The issue catalog

The catalog holds 34 issues grouped into eleven sections: dispute resolution, changes, your content, termination, liability, eligibility, data, user rights, IP enforcement, restrictions, and general legal terms. Each issue is tagged with a polarity — whether a clause of that kind typically works against the user (a content license, a liability cap, a unilateral-change right), for the user (advance notice before changes, a right to delete an account), or is neutral (governing law, severability, an entire-agreement clause).

The list is deliberately comprehensive rather than convenient. It includes clauses none of the three documents carry at these dates — binding arbitration and jury-trial waivers among them — precisely so that their absence is visible. An empty row across all three columns is a finding: it marks a clause that had not yet arrived in these particular agreements.

How a clause is detected

Every sentence of each document is converted into a 768-dimension vector by the all-mpnet-base-v2 sentence-transformer model, the same service described in the previous post. Each of the 34 issues is expressed as a short probe sentence — “disputes resolved by binding arbitration not courts,” “you grant us a broad license to use content you upload” — which is embedded the same way.

An issue is detected in a document when the closest sentence-to-probe cosine similarity clears a threshold, confirmed by a check for the legal phrasing that names the clause. The two signals cover each other’s weaknesses: the embedding catches paraphrase that keyword matching misses, and the phrasing check rejects sentences that merely sound similar. The result is graded — explicit when the document spells the term out, present when it clearly carries it, hinted when the language only grazes it, absent otherwise.

The model runs locally. all-mpnet-base-v2 is loaded once into a resident process and answers every embedding request from on-premises hardware, so scoring a document — or scoring the thousands of revisions this corpus is growing toward — carries no marginal cost per call and sends no clause to a third-party service. The cost of the analysis is the cost of the machine it already runs on, whether it processes three documents or three thousand. For a corpus assembled from other people’s contracts, keeping the text and the computation in-house is a property of the design, not a configuration option.

Three scores, not one

Detection tells you whether a clause is there. It does not tell you how much it matters. So each cell carries three numbers:

• Similarity is the closest sentence-to-issue cosine, from roughly 0.44 at the detection floor to 0.90 for a near-verbatim match. It measures how squarely the document’s language sits on the issue.
• Length counts the words across every passage that addresses the issue — how much of the contract the topic occupies. YouTube spends 837 words on its content license; Twitter spends 541 on the same issue; Reddit, 305. The gap is not stylistic. It is the difference between a clause that grants a few rights and one that enumerates many.
• Sentiment is the density of adverse legal markers — “sole discretion,” “without notice,” “as is,” “indemnify,” “irrevocable” — in the passages that address the issue. It runs from 0 to 1, and it is where the tone of a clause shows. Reddit’s warranty disclaimer scores 0.44, the harshest single clause in the set; Twitter’s termination-without-notice language scores 0.42.

None of the three scores is a verdict. Together they let a reader ask a specific question — which document says the most about content licensing, and in the firmest language? — and get an answer grounded in the text.

What the ranking shows

Ranked by breadth of coverage, then by how many clauses cut against the user:

YouTube is the most complete of the three — it is also the most demanding, carrying the largest number of user-adverse clauses and the longest treatment of the ones it has. Twitter and Reddit tie on breadth but diverge in tone: Twitter’s clauses read harder, Reddit’s read softer, and the sentiment score is what separates them. The three documents agree on the everyday machinery — unilateral changes, liability caps, warranty disclaimers, a right to delete an account — and diverge exactly where the stakes are highest, in content licensing and termination.

A note on bad captures

One data hazard is worth naming, because it nearly corrupted the result. The most recent full Twitter capture in the archive was a JavaScript shell served in Korean — a page that passed a naive length gate but contained none of the actual agreement. Scored blind, it registered near-zero on every issue and would have made Twitter look as though it had no terms at all. The audit uses the last capture that is demonstrably the real, English, full-text document. Detecting and rejecting these shells is a standing requirement for any corpus built from web archives, not an afterthought.

The whole field: sixty-four agreements, ranked

Three documents read closely establish the method. The method then scales. The same 34-issue scoring was run across sixty-four documents — the terms of service and privacy policies of thirty-two companies, spanning social platforms, cloud services, streaming, e-commerce, fintech, gig work, and AI — and the result is the interactive leaderboard below. Each row is one document; the bar beside it is a 34-stripe barcode, one stripe per issue, colored by whether the clause works against you, for you, or is neutral, and taller the more firmly it is committed. Sort by coverage, by how many clauses cut against the user, or by length; filter to compare terms with terms and privacy policies with privacy policies.

Three things the leaderboard makes plain. First, length is not coverage. TikTok’s and Block’s terms of service each cover all 34 issues in well under 20,000 words; Coinbase’s user agreement runs to 54,000 words and covers 32; PayPal’s 26,000 words cover just 24 — fewer than Facebook’s 3,500-word terms. A long contract can say a great deal about a few things, and a short one can be comprehensive. Second, the arbitration divide is generational. Binding arbitration and class-action waivers — the clauses that keep disputes out of open court — are absent from the early social-media terms of YouTube, Reddit, and Twitter, but near-universal among the fintech, gig-work, and AI companies: Coinbase, Block, Stripe, Venmo, Lyft, DoorDash, OpenAI, and Perplexity all carry them. The dispute-resolution rows, empty for the pioneers, fill in solidly for the newer arrivals. Third, privacy policies rank lower than terms of service — by design of the yardstick, not by virtue. The 34-issue catalog is written for terms of service, so it asks about content licensing and dispute resolution that a privacy policy correctly omits; the filter exists so that like is compared with like.

Where it goes next

Scoring one capture per document is a snapshot. Running the same three scores across every revision in a document’s history turns the static matrix into a moving one — it shows when each issue first appeared, when its language hardened, and, the case worth watching, when a clause that was removed came back. That temporal layer is built, and it is where this work continues: across the growing corpus, both terms and privacy policies, a systematic record of what the industry adds, drops, and restores. The interactive matrix for the original three documents, with the score toggle, and the twenty-one-document leaderboard above both accompany this post.