Research notes
How we measure, and what we have not published yet.
Anything published here is built from consented, aggregated results, never from a text anyone pasted in, and never about one learner, however anonymised. Cohorts are published at 200 consented results or more. Nothing has reached that threshold yet, so every entry below says so.
The method and its 34 sources Quarterly data releases [email protected]
Index
Correction: the findings on this page have been withdrawn
Research notes: vigilance
What changed: until today this page reported that across 10,412 consented benchmarks readers marked the planted error in 38% of texts, broke that rate down by error type, and offered three quarterly aggregate releases for download under an open licence. Those figures were written as design mock-ups while the site was being laid out, before the benchmark could record anything. No study produced them, and nothing in the product ever computed them. They have been removed, and the three CSV files now return a withdrawal notice in place of the data, so that anyone who cited or scripted against those URLs finds the retraction rather than the numbers.
If you downloaded or cited any of the three releases, they do not describe measured results and should not be used for anything. Withdrawn 23 September 2026.
What is true today
The instrument below is real, and it is what you take when you start. It records effective reading speed, comprehension, and whether the planted error was marked. Consented results accumulate from there. What does not exist is any body of results large enough to report: the thresholds listed in the index above are the ones we publish at, and every one of them is still at zero.
We also no longer show a cohort comparison on the result screen. It was drawn from the same withdrawn files, and it told readers who had just finished the benchmark what people “like them” scored. Until there is a real cohort, that screen reports your own two numbers and says plainly that there is nothing yet to compare them against.
Whether error-catching can be trained is the open question, and it sits in the “we’re testing” column on the method page. We do not have a baseline, and we do not have a before-and-after. Until we do, this page will keep saying so.
- Instrument
- Benchmark v1.4: one 280–340-word AI-generated passage in the reader's own field, six four-option questions, and one planted error detectable from the passage alone; then a second, shorter text of a different kind, reported separately. Passages are matched on length, question count and structure rather than calibrated against each other.
- Readers
- No published cohort. Cohorts are reported at 200 consented results or more, split by first language, field and starting band.
- Consent basis
- Opt-in at the result screen, unticked by default, revocable. Screen-reader sessions are excluded from norms.
- Limits
- Self-reported level; one error type per text; benchmark conditions, not work conditions; no control group.
- Version
- Findings withdrawn 23 September 2026. Nothing published since.
Quarterly data releases
When there is an aggregate to release, it goes out under an open licence with attribution, and each release lists the cohorts it covers, the instrument version and the consent basis. There is none yet: the three releases previously offered here were mock-ups and have been withdrawn.
| Release | Cohorts | Instrument | Results | File |
|---|
Citation: dub.mom research notes, quarterly aggregate release, <quarter>. Retrieved <date>.
Requesting other aggregates
We will prepare an aggregate that is not in a release if the cohort is 200 consented results or more and you name the purpose. Write to [email protected].
Never released
- Anything derived from a text a learner pasted in. Those are not stored and not analysed
- Anything about a single learner, however anonymised
- Any cohort smaller than 200 consented results
- Percentiles for an individual, or a name next to a number
- Team-level data identifying a company without that company’s written agreement