Moving a dictionary between instances offline

The project's dataset can be downloaded from HuggingFace; other catalog datasets can be downloaded on the server from their source links. Installations without internet access — corporate networks, air-gapped labs, CI — and admins who edited an exported dataset can load the same files from a local source instead:

• files uploaded through the admin UI — the zip that Export on the card of a dataset produces (the Archive tab), or the dataset files in their own slots with the manifest either as a file or typed by hand (the Separate files tab);
• a directory or zip on the server — inside the folder named by DICTIONARY_IMPORT_DIR (a mounted volume, for instance).

Both go through the same import pipeline as the HuggingFace source; only the download stage is skipped.

Source files such as OpenGloss's six Parquet shards use How to install → Upload files manually on the source's card, not the archive import described below. The server converts them before importing. To convert on another machine, use the converter CLI, then import its output into the matching opengloss dataset. Source installation and manual multipart fields are documented in datasets.

Dataset format

A dataset is the set of files the export writes, flat or inside a single wrapper folder:

manifest.json
dataset-format.json                                    # required by current exports with sources and licenses
LICENSE                                                # the license and the notices of the source, in full
vocab-bloom-hub-en-words.jsonl
vocab-bloom-hub-en-phrasal-verbs.jsonl
vocab-bloom-hub-en-grammar-patterns.jsonl
vocab-bloom-hub-en-phrases.jsonl
vocab-bloom-hub-en-meanings.jsonl
vocab-bloom-hub-en-meaning-translations.<lang>.jsonl   # one per language: .ru, .es, .fr, .de, .pt, .zh, .ar
vocab-bloom-hub-en-short-translations.<lang>.jsonl
vocab-bloom-hub-en-changes.jsonl                       # the history of edits, when the data was edited

The entry files (words, phrases, grammar-patterns) carry the entries themselves and their forms; phrasal-verbs is the linking map of base verbs to their phrasal variants. The collections are files of their own, one line per row next to the key of its parent:

File	One line per	Key of the parent
meanings	meaning (with its links)	word, part_of_speech
meaning-translations.<lang>	translation of a meaning	word, part_of_speech, meaning_sort_order, meaning_title
short-translations.<lang>	short translation of an entry	word, part_of_speech

The translations are one file per language (<lang> is a member of available_languages), so a file stays a manageable size and a language loads on its own; a language without rows has no file. Exports made before this split wrote one combined meaning-translations.jsonl / short-translations.jsonl for every language — the import still reads those.

Phrases and grammar patterns are keyed the same way, with phrase / grammar_pattern as the part of speech. The lines follow the order of the entry files, then the natural keys of the rows, so two exports of the same data are byte-identical. Datasets published before this layout nest meanings and short_translations inside the entry lines; the import still reads them.

For legacy files without structured origins, only the .jsonl files you actually have are needed — at least one of them — and manifest.json is optional. Current exports require their manifest and format marker, and the complete history when declared; see sources and licenses. The collection files are imported after the entry files and go to the entries and meanings that exist by then, whether they came from this dataset or were there before: a file of translations alone — one language, say — loads into a dictionary that already holds the entries, and rows the dictionary already has (a meaning with the same sort order and title, a translation in the same language with the same title, a short translation in the same language with the same description) are skipped like duplicate entries. When the manifest is present its version is stored as Your version after the import (and its synonym / antonym link counts refine the progress bar); without it the version stays unknown. The terms an export writes into the manifest — source, license, license_url, attribution, attribution_url, notice (DATA_LICENSE.md) — say where the export was taken from. The terms of the dataset the import fills must match the target: a manifest that names another source than that dataset is refused (dataset_source_mismatch), and so is one that names another license than a dataset of the owner's has: datasets are never mixed (datasets.md). Line counts for the progress bar are always taken from the files themselves, so a hand-assembled dataset needs no bookkeeping. Catalog primary terms remain fixed. A current-format import restores dataset origins and, for an own dataset or fork, its exported title, description, notice, attribution and license details. This metadata restoration does not permit changing the target's primary license by import.

The history of edits travels with the data (datasets.md): changes holds one line per edit made on the instance the export was taken from — the headword and the part of speech, what was edited, the values before and after, where the edit came from, the name of a reader who asked to be credited — and modified_entries of the manifest counts the entries that differ from their source by them. The licenses ask that an indication of earlier modifications is kept, so the import reads the file last (stage 14) and merges it into the history of the dataset it fills: a line the history already has is skipped, and an edit shows on the importing instance only where the entry itself was taken from the copy — for an entry the instance kept as it had it, the edit is recorded as past. The file is absent from an export of a dataset nobody edited, and data that comes without it is taken to be clean: an import marks no entry as changed on its own. LICENSE is written for whoever receives the files; the importer does not parse it. Machine-readable terms come from the manifest, word origins and contribution snapshots. Keep the notice file with the copy even though it is not the source of imported metadata.

Export on A → copy → import on B

1. On instance A open Managing → Datasets → Export on the card of the dataset (or call GET /api/en/dictionary/export and download the archive from GET /api/en/dictionary/export/download/:exportId). You get vocab-bloom-hub-en-export.zip.

2. Copy the zip to a machine that can reach instance B.

3. On instance B open Managing → Datasets → Import → Archive on the card of the dataset, drop the zip into the upload area and press Start importing. The progress stream is the same as for the HuggingFace import; the manifest version is stored as Your version when the import completes.

   To load the files separately open the Separate files tab instead: one slot per file (the slot decides what the file is, its own name does not matter), and the manifest either as manifest.json or filled in by hand (version, optional synonym / antonym link counts).

Equivalent API calls (the admin cookie or a Bearer token is required). The multipart fields are archive for the whole zip, or words, phrasal_verbs, grammar_patterns, phrases, meanings, meaning_translations_<lang>, short_translations_<lang> (one slot per language, e.g. short_translations_es), changes, manifest, and provenance for dataset-format.json; the text fields version, synonym_links and antonym_links stand in for (and override) a manifest file:

Handwritten version/count fields are for legacy imports. For an export with sources and licenses, upload its original manifest and marker rather than replacing them with text fields.

The examples use localhost:3010, the port of a start without Docker; under docker compose the API is on localhost:3240 by default.

# the whole archive
curl -N -b cookies.txt -F archive=@vocab-bloom-hub-en-export.zip \
  http://localhost:3010/api/en/dictionary/import/upload

# only the words, the version typed by hand
curl -N -b cookies.txt -F words=@my-words.jsonl -F version=1.4.0 \
  http://localhost:3010/api/en/dictionary/import/upload

# translations only, for entries the dictionary already has
curl -N -b cookies.txt -F short_translations_es=@es-short-translations.jsonl \
  -F meaning_translations_es=@es-meaning-translations.jsonl \
  http://localhost:3010/api/en/dictionary/import/upload

# an export of a WordNet dataset into the WordNet dataset of this instance (installed when it
# is not yet); the active one keeps serving
curl -N -b cookies.txt -F archive=@wordnet-export.zip -F dataset=wordnet \
  http://localhost:3010/api/en/dictionary/import/upload

# a complete OpenGloss export, retaining word sources, licenses and any edit history
curl -N -b cookies.txt -F archive=@opengloss-export.zip -F dataset=opengloss \
  http://localhost:3010/api/en/dictionary/import/upload

Datasets on the server (DICTIONARY_IMPORT_DIR)

When the zip is already on the server — mounted into a container, copied by a deploy script — point DICTIONARY_IMPORT_DIR at the folder holding it:

DICTIONARY_IMPORT_DIR=/data/dictionary-imports

The Archive tab then also lists what the folder offers (zip archives and sub-folders containing a manifest.json, one level deep) next to the upload area, and the request names the pick relative to that folder:

curl -N -b cookies.txt -H 'Content-Type: application/json' \
  -d '{"source":{"kind":"file","path":"vocab-bloom-hub-en-export.zip"}}' \
  http://localhost:3010/api/en/dictionary/import

GET /api/en/dictionary/import/sources returns the same listing the UI shows:

{
  "import_dir_configured": true,
  "files": [
    {
      "path": "vocab-bloom-hub-en-export.zip",
      "kind": "zip",
      "size": 12345678,
      "modified_at": "2026-09-15T18:18:25.486Z"
    },
    { "path": "hand-assembled", "kind": "directory", "size": 0, "modified_at": "2026-09-15T18:18:25.486Z" }
  ],
  "revisions": ["v0.1.0"]
}

revisions are the version tags of the published dataset repository, newest first (the Dataset version selector of the HuggingFace tab; [] when the HuggingFace refs API is unreachable).

Notes

Provenance export format

Current exports preserve dataset and word origins, associated licenses, full notices, and inherited edit history. They use the provenance-v1 metadata contract and keep the original filenames listed above, for example vocab-bloom-hub-en-words.jsonl and vocab-bloom-hub-en-changes.jsonl. The archive also contains dataset-format.json with {"format":1} and its original manifest.json. Earlier archives with provenance-v1. filename prefixes and the provenance.v1.json marker remain readable.

The on-disk manifest contains:

{
  "version": { "format": "provenance-v1", "dataset": "2.0.0" },
  "provenance_format": 1,
  "provenance": {
    "title": "My dictionary",
    "notice": null,
    "description": null,
    "license_text": null,
    "origins": []
  },
  "files": {
    "vocab-bloom-hub-en-words.jsonl": { "lines": 1 }
  }
}

The ordinary manifest source and primary license/attribution fields are still present. The provenance snapshot additionally retains the title, description, notice, custom license text, origins, and attribution/link settings. Entry lines carry origins; licenses[] is derived from them. Source snapshots include their acquisitions[] (copy/fork events, dates, and revisions); repeated acquisitions do not repeat source terms. An acquisition's optional via names an intermediate dataset without assigning its license to unchanged material. History lines retain inherited_from, correction reason, and the contribution snapshot of the editing dataset's terms when known. Active history supplies the word API's contributions[] and their licenses; keep the history file with the words to preserve that attribution. For provenance-v1 uploads, the declared history file must be present with the full line count from the manifest, even when importing only a subset of the other files. A missing or truncated history is rejected before importing words. Superseded edits retain their original terms in history without imposing those terms on the current word or blocking a later license change. The API's dataset version remains a string or null; the envelope is only the portable file contract.

Keep the complete archive when transferring a dataset. With separate files, retain the exported filenames and upload both the manifest and the Sources and licenses marker (dataset-format.json). A handwritten legacy manifest is insufficient for files with structured origins. Imports reject missing markers, mixed legacy/new names, and unsupported provenance formats. Legacy imports without origins remain supported and acquire their target dataset's known defaults.

Upgrade the receiving server before importing these exports. Released local importers reject the format marker and the manifest envelope before writing words. With the original filenames, older remote importers can still download words after ignoring an unfamiliar manifest and may lose source metadata. Keep those servers pinned to a compatible older dataset revision until they are upgraded; matching filenames do not make the metadata backward-compatible. Publish the complete new-format files together, and do not remove the format marker or strip source metadata to force an old importer to accept an export.

OpenGloss conversion writes this format from the start, even before anyone edits the data. Words with a WordNet source retain both OpenGloss and Princeton WordNet 3.0 snapshots; other words retain the OpenGloss snapshot. Export/import preserves the converted dictionary, its source versions, links and notices. It does not restore upstream Parquet fields that the converter deliberately leaves out.

Alternative spelling links

Word, phrase and grammar-pattern lines may carry alternatives: string[]; form objects may carry the same field for their own headword. Export repeats the headword's list for every part of speech. Import unions supplied lists for a headword before resolving them, so file/chunk/POS order does not affect the result. Links are reciprocal and use exact spellings, never local IDs or part-of-speech qualifiers. Missing targets are skipped.

An omitted field in an older file preserves existing links. An explicitly empty list can propose clearing an imported headword's links during review, unless the reciprocal endpoint explicitly requests the pair. Links involving a user-modified headword remain protected. Export with a version that understands this field to preserve it on round-trip; older importers may discard it. Alternative-only Wiktionary words retain the source gloss.

Update review and synchronization metadata

A complete native export declares snapshot: { complete: true, adapter: "native", revision: 1, skipped: 0 }. Only verified complete input can propose deletions for absent records. A separate-file upload, a legacy archive without this marker, or a conversion with skipped records is partial: absent words and nested records are retained, including in the baseline used by the next update. A missing common baseline requires explicit manual comparison; version labels are not ancestry.

vocab-bloom-hub-en-synchronization.jsonl format 2 preserves dataset identity, causal links between immutable snapshots, integration baselines and applied operations with their decisions. Format 1 is still readable, but lacks causal links and cannot establish an unambiguous common ancestor for sibling merges. Pending reviews belong to their instance. Keep the full archive; upgrade older importers before transferring it. Imported summary counters are recalculated.

Decisions are saved independently of publication. Resume a review from its dataset card, or cancel it without changing the dictionary. A failed apply rolls back content, history, baseline and operation status together; retry the same plan. If local content changed during review, prepare a fresh plan. Applied updates and merges appear in edit history and the audit journal; selecting source text keeps its original contribution terms, while newly authored fields receive the destination's current contribution terms.

Etymologies and flat meanings

Word, phrase and grammar-pattern lines may carry etymologies: [{number, text}]. Numbers are positive integers unique within that word and part of speech; array order is not significant. Meaning lines (including nested meanings) carry etymology_number, which is nullable and must refer to a group on their own word. Database IDs never travel in these files. Imports rebuild the references after creating each word's groups, so an export can be installed in a database with different IDs. Unknown references and repeated group numbers are rejected. Old files without these fields remain valid.

Updates compare groups and meanings with their previous source state and local edits. Conflicting edits, deletions and ambiguous identities require review; independent changes can be combined. Export and history retain the local numbers and associations. Use an importer that understands these optional fields to preserve them on round-trip.

The admin word response includes group IDs. When replacing etymologies through the common-word edit endpoint, send the existing ID to keep a group and its meaning links; omit the ID to create a new group. Changing a kept group's number preserves its links. Omitted groups are deleted and their meanings become unassigned, even if a new group reuses the old number. Omit the entire field to leave the groups unchanged. The public word response returns only {number, text}, and its flat meanings expose the nullable etymology_number. History and reverting retain the associations without database IDs.

Meaning quotations

Meaning lines (and legacy nested meanings) may carry an ordered quotes array:

{
  "quotes": [
    {
      "text": "An invented quotation.",
      "reference": "Test Author, Book (2026), p. 7",
      "source_url": "https://example.org/book"
    },
    { "text": "Another quotation.", "reference": null }
  ]
}

A quote has nonblank text, a bibliographic reference (a string, or null when unknown) and an optional HTTP(S) source_url. An omitted reference becomes null; an absent URL remains absent. Text, reference, line breaks and array order survive conversion, import/export, copy/fork and history/revert. No quote-specific length or count cap is applied; normal request and import-file size limits still apply. Ordinary examples: string[] remain separate and retain their existing behavior.

Old files without quotes import an empty list. Updating an unprotected meaning replaces its quotes with the imported list; protected entries retain theirs. On the admin meaning edit endpoint, omit quotes to keep it, or send [] to clear it (null also clears it). Public meaning responses expose an optional quotes field so clients remain compatible with older servers. Storage uses a nullable JSON-encoded column on each meaning, with array order determining quotation order; no separate quote IDs need remapping.

Use an importer that understands quotes to preserve them; older versions may discard this optional field. Bibliographic attribution is retained verbatim, independently of the word's existing data-origin and contribution license snapshots.

Pronunciations

Word, phrase, grammar-pattern and nested form lines may carry pronunciations: [{type, text, area_variant, sort_order}]. type is ipa or enpr; area_variant is common, british, american or australian; sort_order is a nonnegative integer. Multiple pronunciations for one region are preserved. Export and history contain no pronunciation IDs. Import and copy allocate new IDs; forks preserve their snapshot's IDs within the new dataset.

Lists are presented by sort_order, then type, region, text and portable audio content for ties. The primary IPA is the first American variant, otherwise the first British variant, otherwise the first IPA in this order. EnPR is never put in transcription. Conversion writes the primary IPA into this legacy field. Import and creation preserve an explicitly supplied nonempty transcription (in particular a form's existing transcription); when empty, they use the primary IPA of that record's own list.

Public word and form responses expose optional pronunciations. A record with no usable list falls back to its own legacy transcription as one IPA item. A form with neither falls back to its base word's pronunciation. This is a read projection: it writes no duplicate rows or human edits. Admin responses, export, copy payloads and history contain only the record's own list. Old datasets without lists need no backfill and keep their legacy transcription.

The admin create/edit endpoints accept the same list. Admin reads also return internal IDs: retain an ID when editing its record; omit it to create a new one. IDs from other owners are rejected on edit, and ignored on import/copy. Omitting the list on edit preserves it; [] clears it. Replacing a list also updates transcription to its primary IPA (or '') unless the request explicitly supplies transcription. Clearing transcription alone leaves the list intact. All content edits and deletions can be reverted through the portable history.

The storage column text is nullable for audio-only pronunciations, described below. Text-only writes require nonblank text; public responses omit unusable records. Use an importer supporting pronunciation lists to preserve them: older versions can retain only the legacy transcription.

Pronunciation audio

A pronunciation may carry an optional audio list. Each recording has url, sort_order, licenses (the existing OriginLicenseT format: name, url, optional spdx and text), optional nullable source_url and attribution. For custom licenses, retain their full text, as with word origins. Unknown recording terms are licenses: []; no text or dataset license is copied into that list. No audio file is downloaded, proxied or hosted by the server.

{
  "type": "ipa",
  "text": null,
  "area_variant": "common",
  "sort_order": 0,
  "audio": [
    {
      "url": "https://example.org/word.ogg",
      "source_url": "https://example.org/recording",
      "attribution": null,
      "licenses": [],
      "sort_order": 0
    }
  ]
}

text: null is accepted when at least one recording is present. Non-null text must be nonblank; a record with neither text nor audio is rejected. Audio-only records do not invent an IPA or change the primary-IPA selection rule. A form's own audio-only pronunciation takes precedence over the base word's data.

Recordings sort by sort_order, with their portable content breaking ties. Their IDs are internal: admin responses include them for editing, while public responses, export and history omit them. Copy/import allocate new IDs; forks preserve the snapshot in a separate dataset. Recording metadata and order survive edits, export/import and history/revert. Removing a pronunciation or word deletes its recordings through foreign-key cascades.

When editing a pronunciation by its existing ID, omit audio to preserve the recordings (including when using an older client), or send [] to clear them. Clearing the last recording of an audio-only pronunciation requires also supplying text or removing that pronunciation. Existing recording IDs must belong to that pronunciation; otherwise the entire edit is rejected. Without a pronunciation ID, the list item is a replacement/new record. Old files without audio continue to import, and no empty audio property is required in public responses. Use an importer supporting this addition to preserve recordings.