Tools for digitising archives
A desktop application to create, correct and validate the XML metadata of digitised documents, with installable modules for working on the images.
What it is for / What it can do
For anyone digitising archives and books who must describe each document with MAG metadata. It is part of the cataloguing and indexing work ETT offers under its Scripta line.
- Creates, corrects and validates MAG files, the Italian XML standard for digitised documents.
- Checks files against the XSD schema and extra rules.
- Crops and straightens scans automatically.
- Renames files and recognises the page type.
- Extends with installable modules, with licensing and updates.
Context
Whoever digitises archives and books produces thousands of images per document. Each one must be described in XML (page type, sequence, checksum, technical data), cropped and renamed, almost always by hand. The format is MAG, the Italian standard for metadata of digitised documents.
I develop it at the software company where I work as a developer. The project is a rewrite that merges two old programs (one in Python 2.4, one in compiled Python 3.8) into one that can be maintained.
The editor and the validator
The editor shows the project as a tree and the metadata tabs (general, bibliography with repeatable Dublin Core fields, naming). The built-in validator checks the file against the XSD schema and extra rules, and lists the problems in a panel: in the image below, an empty required field. The document is an example.
The modules
Around the editor there are modules installed from an internal catalogue, each with a licence that is verified even offline and a signed update:
- Crop and straighten: the rectangle can be moved, resized from the corners and rotated with a dial. Each image is first brought down to a 1024-pixel version on which the detector and the operator work; with this choice the time per image went from 111 to 0.37 seconds.
- Renaming of files according to configurable conventions.
- Page type: a machine learning model suggests whether a page is a cover, text, plate and so on. It uses a classifier on 147 hand-built features and a search for the most likely sequence of labels; it learns from the operator's corrections.
- Colour check.
The management panel
For administrators there is a separate web panel: users, per-module licences with expiry and revocation, release of new versions. The screenshots below come from a test instance with invented data, not from the one in use.
Management panel (test instance, fake data)
Management panel: user list with fake data
Management panel: licences per module grouped by user, with expiry and revocation
Management panel: modules and versions
Some technical choices
- The interface is moving from Qt to Tauri with Vue; each module has its own small local Python service.
- Modules are distributed with SHA-256 checksum checks and signed updates, first on a test channel and then to everyone.
- The Qt test suite crashes if run all at once, because of shared library state: I wrote a runner that executes one file at a time.
What I did
I wrote almost all the code myself, in about seven and a half months: more than 2,800 commits, about 200,000 lines of Python and 46,000 of Vue and TypeScript, covering the data model, validation, interface, modules, licences and panel. I also used AI coding agents; I have no measure of how much of the code came about that way.
Status
It is in use, with real users and licences, and still under development. The program belongs to the company: it is not public and there is no demo. The Qt interface is still being phased out and some parts are unfinished.
Stack
Python · lxml · Pillow · OpenCV · FastAPI · Tauri · Vue · TypeScript · gradient boosting · U²-Net · YOLO · Docker · pytest