Dictionary data: provenance and quality

What the dictionary is made of, where it comes from, what is known to be wrong with it and how to report errors. The terms of use are in DATA_LICENSE.md: the data is CC BY 4.0, the code is MIT. The published copy of the data is the HuggingFace dataset Fristail27/vocab-bloom-hub-en; its dataset card is the full, revision-specific version of this page (counts per model, field statistics, content notes).

This page is about the project's own dataset, the one an instance is born with. An instance can hold datasets of other sources next to it — the English Wiktionary, WordNet, OpenGloss — each under the terms of its source: Datasets from other sources.

Comparing installed datasets in the admin

Statistics → All datasets compares every installed dictionary, including own datasets, forks, closed dictionaries and empty datasets. Uninstalled catalog sources are omitted. Each card shows its title/name, version, public/closed and active status, and the same counters as its detailed statistics: entries, base words, forms, phrases, grammar patterns, meanings and both kinds of translation. Shared spellings and copied entries count in each dataset; these are not unique-word totals across the instance.

The links open that dataset's common, translation and issue statistics. The dataset URL parameter takes precedence over the editing cookie and does not activate the dictionary. From issue statistics, a record can be opened for correction as described below.

The admin-only GET /api/en/statistics/datasets reads installed datasets sequentially through their readers and reuses the detailed statistics' count definitions. It ignores dataset selection, works with the single default dataset on SQLite, and does not cache results. read_at records when collection finished; use Refresh statistics to read again. Counts are live reads, not a cross-dataset snapshot. An import overlapping a dataset's read marks it updating; the UI warns that figures may be intermediate. Refresh after the import ends. An unreadable dataset stays visible with totals: null and statistics_unavailable, while other datasets remain readable. An empty dictionary has actual zero counts instead.

Inspecting missing and empty data in the admin

Statistics → Data issues opens matching records from each category. The list counts words, meanings, meaning translations or short translations as labelled: several meanings of one word are separate records. Word categories use base forms, excluding phrases and grammar patterns; meaning and translation categories include those entries too. The counter and list use the same filters.

Each row opens the word editor at the affected meaning or translation where applicable. Back to matching records keeps the dataset, category and page and reloads the counts and list. Corrected records leave the list; removing the last match on a page moves to the last remaining page. An empty category remains available and says there are no matching records. The URL pins the dataset for reads and edits, including inactive and closed datasets, without changing the publicly active dictionary. The header's dataset switch returns a review to its list when changing datasets, so a word ID is never reused in another dataset.

Missing fields are not a universal quality score. Catalog capabilities explain expected omissions (for example, WordNet translations and CEFR levels). Synonyms, antonyms and the word-level description are informational; a meaning need not have an antonym. Own datasets have no declared completeness requirements, so their missing fields are informational too. Stored translations with blank text are listed separately for inspection. Every category can be opened, including informational ones.

The authenticated API is GET /api/en/statistics/issues/records with a required category (one of the keys returned by GET /api/en/statistics/issues), optional dataset, page (default 1) and limit (default 25, maximum 100). Results contain the exact record ID, parent word ID, meaning and language where applicable, record kind, interpretation and pagination. Lists use ascending record IDs. Counts describe the current data; concurrent edits can change them between requests.

Where the data comes from

Every entry is produced by an LLM-assisted pipeline — a model is asked for the entry (transcription, CEFR level, senses with definitions and examples, translations, inflected forms), the answer is stored in the Hub database, and the database is what gets exported and published. Nothing is scraped or copied from other dictionaries.

Two columns on every base-form word record how it was generated (sources and licenses are recorded separately in origins and in the history's contribution snapshots):

Field	Meaning
generated	true when the entry was produced by a model; false for entries authored by hand in the admin
generated_by_model	The model behind the entry, as an OpenRouter-style id (deepseek/deepseek-v4-flash); may be null

The published data is generated with DeepSeek models — the bulk with DeepSeek v4 Flash, smaller batches with deepseek/deepseek-v4-pro; early batches were labelled by hand, so the same model can appear under several spellings. The exact numbers for a revision are on the dataset card. Only models whose terms are compatible with the data license are used — why that rules some providers out is recorded in DATA_LICENSE.md.

Meanings link to other headwords as synonyms and antonyms (~509k and ~183k links in the v0.1.0 revision, counted in manifest.json as synonym_links / antonym_links); the API serves them under every entry and as /words/{word}/synonyms / /antonyms, and the dataset carries them in the meanings file (earlier datasets nested the meanings in the words, phrases and grammar-patterns lines).

Known limitations

• Not a lexicographic authority. Definitions, examples, CEFR levels, register and domain labels are model judgements. Hallucinated senses and invented examples are possible; do not use the data as ground truth for evaluating other dictionaries or as a citable source of English usage.
• Translations are generated too and have not been reviewed by a translator. The schema carries seven translation languages, Russian, Spanish, French, German, Portuguese, Chinese and Arabic (GET /api/v1/meta lists them under available_languages). The published dataset ships all seven since the tagged v0.2.0 revision (v0.1.0 holds the first five), one file per language, and manifest.json counts the rows per language (translations) — the dataset card gives the coverage of each. Translations ship in files of their own, one per language (meaning-translations.<lang>, short-translations.<lang>, offline-import.md), so a language can be loaded on its own into an instance that already has the entries.
• Entry-level language_register is unreliable on words — almost every word says formal because the field defaulted that way during generation. The per-sense register inside meanings is the meaningful one.
• No frequency data. CEFR levels are model judgements, not corpus-derived; there is no frequency ranking or attestation.
• "" means "not set" for every enum field (level, transitivity, phrasal object pattern).
• Offensive vocabulary is included — the word list aims at broad coverage, so it contains slurs and vulgar and outdated terms, described rather than endorsed. Applications that surface random entries should filter on language_register, is_obsolete and their sense-level equivalents; a dedicated sensitivity flag does not exist yet.
• Bias. Model-generated text inherits the biases of the generating models, most visibly in which senses are listed first and in the connotations attached to social, political and religious vocabulary.

Personal data

The published dataset of the project contains no personal information. The data of an instance may come to hold one kind: the name a reader asked to be credited by for a correction. It is taken only with an explicit consent on the form — the form says that the name is shown next to the correction and travels with the copies of the dictionary data, and that a published name cannot be taken back from the copies others have made. A report without a name is as welcome. The name is stored with the report and, once the owner applies the correction, in the history of the entry: shown on the word page, served by GET /api/v1/words/{word}/history — and by GET /api/v1/words/{word}/datasets/{dataset}/history for every dataset the instance holds, active or not — written into the history file of an export.

On request the owner removes a name with Take a name out of the history (Admin → History, POST /api/en/changes/forget-author): it leaves the history and the reports of every dataset of the instance, the corrections stay. Exports made before that still carry it — whoever received one is asked separately.

Datasets from other sources

An instance on PostgreSQL keeps several datasets, one of them active (datasets.md), and datasets of its owner's own, under a license the owner chooses (datasets.md). The datasets page of the admin UI downloads a public source's files directly from catalog links, converts them and imports them. Manual upload is available when the server cannot download them:

Source	License of the data	What an entry has	What it lacks
English Wiktionary	CC BY-SA 4.0	definitions, examples, IPA, forms, synonyms and antonyms, translations as single words	CEFR levels, definitions of the translated senses
Open English WordNet	CC BY 4.0	definitions, examples, synonyms and antonyms, irregular plurals and degrees	translations, levels, registers, verb forms
Princeton WordNet 3.x	WordNet license	the same	the same
OpenGloss 2.4	CC BY 4.0, plus WordNet terms on marked words	canonical definitions, neutral/plain examples, supported inflections, synonyms and antonyms	translations, IPA and CEFR levels
CMU Pronouncing Dictionary	BSD 2-Clause	pronunciations of American English for the WordNet entries, turned from ARPAbet into IPA	—

Wiktionary and the WordNet datasets are human-authored: their forms, API and import refuse entries marked as generated. OpenGloss is a synthetic dictionary and permits generated content; the converter preserves its generation flags and model names. Each dataset carries its own notice. Three things follow from keeping the datasets separate:

• Reads do not combine dictionaries. A word can still have several origins and licenses: OpenGloss words marked as derived from WordNet retain both sources. Deliberate copying and forking preserve sources and record later contributions; they do not silently borrow fields.
• The license of the active dataset is the license of what the instance serves: the API answers, the exports, the corrections the readers send. Wiktionary is share-alike — a product built on an instance that serves it keeps derived data under CC BY-SA 4.0.
• Fields the source does not have stay empty ("" for the enums, as everywhere): no level is guessed, no register is assumed, an irregular verb of WordNet is flagged without forms because the source does not say which form is which.

OpenGloss installation uses all six pinned Parquet files (about 1.32 GB). It converts the supported dictionary fields, not the complete tables: encyclopedia text, graded rewrites, etymologies and training metadata have no matching fields in the Hub. Malformed text is counted and omitted while valid senses remain. See the OpenGloss conversion details for the exact boundaries and offline import for how the converted data, licenses and history travel together.

Edits made on an instance

The owner of an instance may edit any entry, and apply the corrections readers send. The licenses of the datasets ask that such changes are indicated, so an instance keeps a history of edits with every dataset: what was changed, with the values before and after. An entry that was changed or added says so — modified on a word of the public API, a line on the word page — and its history is public (GET /api/v1/words/{word}/history; for a dataset that is not the served one, GET /api/v1/words/{word}/datasets/{dataset}/history). The history travels with an export and is read by an import, so a copy of the data keeps the indication. How it works, how a change is taken back: datasets.md.

If you use the data of an instance, the indication is yours to keep: credit the source, say of a changed entry that it was changed, keep it under the license of its dataset. What the instance gives for that and what to do with it: DATA_LICENSE.md.

Dataset versions

Each published revision of the HuggingFace dataset carries its version in manifest.json (manifest.version) and is git-tagged with that version on the dataset repository (HF datasets are git repos; the publishing step ends with git tag <version> && git push origin <version> there — v0.1.0 is the first tagged revision). The tags make revisions addressable:

• HF serves any revision via resolve/<revision>/…, and the server imports one with POST /api/en/dictionary/import { "source": { "kind": "huggingface", "revision": "<tag>" } } — the import of the project's dataset in the admin UI offers the tags in a Dataset version selector;
• DICTIONARY_DATASET_VERSION=<tag> pins the automatic first-start import (environment.md); unset means the moving main;
• the list of tags comes from the HF refs API (https://huggingface.co/api/datasets/Fristail27/vocab-bloom-hub-en/refs).

The dataset version is independent of the application version: it is bumped at the next export after a release.

Reporting errors

Fixes land in the Hub database, never in the published JSONL files (they are overwritten by the next export):

• A reader of a word page has two flows right on the page, both landing in that instance's own moderation queue (POST /api/v1/suggestions — no account, strictly rate-limited): Report a mistake opens one form with two modes — a free-text report, or the whole entry opened in editable fields to suggest corrected values — the admin sees the before/after diff on the Suggestions page and applies it in one click (the change goes through the normal edit flow: it is recorded in the history of the dataset as a correction of a reader, and the entry is marked as the owner's). A reader who wants to be credited gives a name and agrees to have it published (Personal data). The loop stays inside the instance deliberately: its dictionary may hold the owner's edits the published dataset does not have. An applied correction is dictionary data like the rest and takes the license of the dataset it corrects — CC BY 4.0 for the project's own, the license of the source for another; the form names it.
• Against the published dataset itself (a wrong definition, translation, level or missing word in what HuggingFace serves) → open an issue with the bug template, quoting the headword, the field and what it should be.
• On your own instance, fix it in the admin UI (which marks the entry as yours and keeps it through dataset updates, see operations.md) and re-export.

Where the terms are exposed

Place	What it carries
manifest.json of every export	source, license, license_url, attribution, attribution_url, notice; modified_entries when the data was edited
LICENSE of every export	The license and the notices of the source in full
GET /api/v1/meta	dataset, source, license, license_url, attribution, attribution_url, notice, license_text, modified_entries (api.md)
Every word of /api/v1	source, and modified when the entry was changed or added on the instance — on the parts of an entry, in the searches and on the edits of the history too
Sources and contributions of a word	origins, contributions, licenses; source names, versions, links, attributions and notices, with license associations
Word pages of the website	The license and the attribution of the dataset under every entry and next to Report a mistake; what was changed on the site
/dataset-terms of the website	The terms of the active dataset, with the notices of its source in full
Admin → Datasets	The terms of every dataset of the catalog, as the code states them
Admin → Datasets → Export	License, link and attribution line next to the download
HuggingFace dataset card	license: cc-by-4.0 front matter, LICENSE, NOTICE, this notice (the project's dataset)

/meta and /dataset-terms describe the active dataset. Each group of the all-datasets read and each word-page tab describes its own dataset; an admin export describes the dataset selected on its card. Catalog terms come from apps/server/core/constants/dataset_catalog.ts; own datasets and forks have owner-supplied contribution terms. Word origins and contribution snapshots retain the terms captured from their sources, including through export/import.

Local content review (#647)

In the admin word editor, each definition, meaning translation and short translation has an independent review marker: Not checked, Checked, or Changed since checking. A definition check covers its title and definition text. A meaning-translation check covers the language, translated title, definition and variants, in the context of its source meaning. A short-translation check covers its language, description and variants. Examples, quotations, grammar, forms, pronunciation and source/license metadata are outside these scopes. A checked portion does not certify the entire word or guarantee correctness. No reviewer identity is collected; existing content starts unchecked.

Use Mark as checked, Confirm again, or Clear check beside the portion in the editor. Confirmation verifies that the displayed text is still current; on a concurrent edit, reload the word before checking it. Under the word search on the Managing page, expand Content review and select a portion kind and status. The word prefix above also filters this list. Counts refer to individual portions, while the paginated list contains distinct words with matching portions. A word may appear under several statuses. All reads and writes use the dataset selected for editing, including inactive/closed ones.

A marker stores a content snapshot under the portion's portable identity, separately from content-edit history, generation flags, versions and origins. Comparing the current content with that snapshot handles admin edits, applied suggestions and source/merge publication. Unchanged saves and unrelated changes preserve checks; a changed source definition also makes checks of its translations stale. An exact revert restores the applicable checked state. A removed portion is not listed; a new portion with only a reused database ID starts unchecked. Notes for removed identities are retained for restoration of the original content.

Reviewing does not modify content history, user_modified, public modified, release versions, source attribution, or contribution licenses, and does not protect content from source updates. Review notes are local to the dataset and instance: native exports, imports, copied words and forks do not transfer them. Source updates can preserve an existing local note when the portable identity survives; their current content still has to match its snapshot. A new identity starts unchecked. A full database backup, unlike a dataset export, retains local notes. There are no public review badges, public API fields, review accounts, or review markers for examples, pronunciation or other scopes in this first version.

The admin-only endpoints are GET /api/en/reviews/word/{id}, GET /api/en/reviews/words?scope=definition&status=unchecked, and POST /api/en/reviews/{scope}/{id} with {checked, expected}. Scopes are definition, meaning_translation, and short_translation. Each endpoint supports the usual dataset query parameter. The listing accepts search (word prefix), page and limit (1–100). PostgreSQL stores en_content_reviews in each dataset schema through a dataset migration; SQLite supports the same markers for its single dataset.