Skip to content



Released on 8 Nov 2023 • View on Gitlab

Breaking changes

  • The arkindex_worker.reporting module has been removed as the JSON report file was no longer needed.
  • The --model-dir CLI argument was renamed to --extras-dir as it was more suited to its use. This folder now stores dataset archives, hence the more generic name.

Arkindex API

  • Following Arkindex 1.5.2 release,
    • new helpers for Task-related endpoints were introduced,
    • A new worker class is available, to support Dataset processes
    • new helpers for Dataset-related endpoints were introduced,
  • Added a unicity check on the input of the create_transcription_entities helper.
  • The partial_update_element helper was updated to better match the endpoint.


  • Some modules were poorly displayed in the documentation. Class methods are now only listed under their class’s section.

Release Management

  • A Makefile was added to the worker template to deploy new releases more easily. The default branch expects master, make sure to change it to main depending on your settings.
  • The base image used in the worker’s docker image was changed from python:3.11 to python:3.11-slim, in an effort to reduce their size.


  • During the configuration stage, a summary of the worker is now logged instead of the revision’s hash. This was changed to support workers not linked to any revision on Arkindex.
  • A retry mechanism on HTTP 50x errors was added. Additionally, when the requested size exceeds the maximum size allowed by the IIIF server, a new try is done with max instead of full as size parameter. More information about these parameters in the IIIF documentation.
  • When running the worker locally without the ARKINDEX_CORPUS_ID variable set in the environment, an explicit exception will be raised when trying to access the corpus_id attribute.
  • This release adds support for Python 3.12.


Released on 14 Sept 2023 • View on Gitlab

  • The worker template was updated to correctly install Git submodules if it depends on any.
  • Base-worker now uses ruff for linting. This tool replaces isort and flake8.
  • New Arkindex API helper to update an element, calling PartialUpdateElement.
  • New Arkindex API helper to list an element’s parents, calling ListElementParents.
  • Worker Activity API is now disabled when the worker runs in read-only mode instead of relying on the --dev CLI argument. The update_activity API helper was updated following Arkindex 1.5.1 changes.
  • Worker can now resize the image of an element when opening them. This uses the IIIF resizing API.


Released on 26 May 2023 • View on Gitlab

  • The Timer class previously defined in arkindex_worker.utils was removed as it was already defined Teklia’s python toolbox.
    # Old usage
    from arkindex_worker.utils import Timer
    # New usage
    from teklia_toolbox.time import Timer
  • The create_element_transcriptions API helper now accepts an element_confidence float field in the dictionaries provided through the transcriptions field. This confidence will be set on the created element.
  • More query filters are available on the list_element_children API helper. More details about their usage is available in the documentation:
    • transcription_worker_version
    • transcription_worker_run
    • with_metadata
    • worker_run
  • Arkindex Base-Worker now fully uses pathlib to handle filesystem paths as suggested by PEP 428.
  • Many helpers were added to handle ZSTD and TAR archives as well as delete files cleanly. More details about that in the documentation of the arkindex_worker.utils module.
  • A bug affecting the parsing of the configuration of workers that use a Machine learning model stored on an Arkindex instance was fixed.


Released on 8 March 2023 • View on Gitlab

  • A helper to use the new API endpoint to create transcription entities more efficiently was implemented.
  • Training workers may now publish a model configuration when creating a new model version on Arkindex. This will make the execution of a generic worker much smoother.
  • The model version API endpoints were updated in the latest Arkindex release and a new helper was introduced subsequently. However, there are no breaking changes and the main helper, publish_model_version, still has the same signature and behaviour.
  • The latest Arkindex release changed the way NER entities are stored and published.
    • The EntityType enum was removed as type slug are no longer restrcited to a small options,
    • create_entity now expects a type slug as a String,
    • a new helper list_corpus_entity_types was added to load the Entity types in the corpus,
    • a new helper check_required_entity_types to make sure that needed entity types are available in the corpus was added. Missing ones are created by default (this can be disabled).
  • The create_classifications helper now expects the UUID of each MLClass instead of their name.
  • In developer mode, the only way to set the corpus_id attribute is to use the ARKINDEX_CORPUS_ID environment variable. When it’s not set, all API requests using the corpus_id as path parameter will fail with 500 status code. A warning log was added to help developers troubleshoot this error by advising them to set this variable.
  • The create_transcriptions helper no longer makes the API call in developer mode. This behaviour aligns with all other publication helpers.
  • Fixes hash computation when publishing a model using publish_model_version.
  • If a process is linked to a model version, its id will be available to the worker through its model_version_id attribute.
  • The URLs of the API endpoint related to Ponos were changed in the latest Arkindex release. Some changes were needed in the test suite.
  • The classes attribute no directly contains the classes of the corpus of the processed element.
    # Old usage
    self.classes = {
        "corpus_id": {
            "ml_class_1": "class_uuid",
    # New usage
    self.classes = {
        "ml_class_1": "class_uuid",


Released on 8 November 2022 • View on Gitlab

  • A breaking change, affecting mostly the API, was introduced in Arkindex’s 1.3.4 release:
    • Workers were mostly unaffected but the REST schema was updated.
  • Workers will progressively not be able to publish results with a worker_version_id anymore on Arkindex. They will have to use a related but more general field, worker_run_id:
    • Most publication API endpoint helpers have been updated accordingly,
    • A new version of the cache was released with the updated Django models.
  • Improvements to our Machine Learning training API to allow workers to use models published on Arkindex.
  • Support workers that have no configuration.
  • Allow publishing metadata with falsy but non-null values.
  • Add .polygon attribute shortcut on Element.
  • Add a major test speedup on our worker template.
  • Support cache usage on our metadata API endpoint helpers.
  • Drop support for Python 3.6 and add support for Python 3.11.
  • Update arkindex-client to version 1.0.11.
  • Update shapely to version 1.8.5-post1


Released on 12 September 2022 • View on Gitlab

  • A large refactoring effort was made on the worker initialization, to streamline most of the workflow:
    • developer setup is now set in a dedicated method configure_for_developers
    • cache setup is now set in a dedicated method configure_cache
    • deprecated useless attribute features
    • add a simpler debug mode for developers
    • depend only on Arkindex RetrieveWorkerRun API to get all the information needed, instead of relying on multiple API calls.
    • remove ARKINDEX_CORPUS_ID environment variable usage, replaced by corpus information from API, except for developers
    • do not erase defaults when reading configuration
  • Support new Machine Learning training APIs on Arkindex to allow workers to create model versions and publish them as zstandard archives on a remote S3-compatible bucket.
  • Add API helpers
    • list_corpus_entities
    • create_metadatas
    • list_metadata
    • list_transcription_entities
    • create_required_types
    • publish_model_version
    • create_model_version
    • upload_to_s3
  • Create missing element types when checking if they are available on the Arkindex instance (disabled by default).
  • Update arkindex-client to version 1.0.9.
  • Update automated rotation code (revert_orientation) to support reverse application


Released on 6 July 2022 • View on Gitlab

  • Document source code using Sphinx and docstrings with parameters. Documentation is available here.
  • Update workers inner config with default values from user_configuration
  • Support confidence in API helpers create_sub_element and create_elements as they are not available in Arkindex
  • Port rotation code from tesseract worker
  • Add helper to trim polygons so that they fit inside their image


Released on 28 March 2022 • View on Gitlab

  • Update arkindex-client to version 1.0.8.
  • Replace all transcription scores with confidences (also renamed on Arkindex)
  • Support cache versioning and detect compatibility in workers
  • Support confidence in create_transcription_entity API helper
  • Support Text orientation for transcriptions
  • Return the response payload in all creation helpers so that workers can use them
  • Support new metadata type URL


Released on 17 September 2021 • View on Gitlab

  • Update arkindex-client to version 1.0.7.
  • Detect already processed elements using worker activity, and skip them
  • Support rotation, mirroring and fix image crop in open_image method used by a lot of workers
  • Change default value for user_configuration from None to {} which simplifies usage code in workers
  • Support new metadata type Numeric
  • Add API helper create_classifications
  • Set worker version in transcription entities API helpers


Released on 30 June 2021 • View on Gitlab

  • Add API helper check_required_types
  • Add a developer mode via --dev argument to simplify boot process for local development
  • Send process_id when updating worker activities
  • Remove nb_best from ML classes list as it’s not supported anymore by Arkindex


Released on 6 May 2021 • View on Gitlab

This is a larger release which brings a new caching system to share data across workers (avoiding a lot of API calls in some workflows), and split the codebase in multiple files for helpers & unit tests (one file per topic).

  • Add cache system using a local SQLite database, shared from workers to workers. Currently supports Arkindex models:
    • elements and their hierarchy,
    • transcriptions,
    • images,
    • classifications,
    • entities,
  • Add API helpers:
    • create_elements
    • create_transcriptions
    • create_transcription_entity
  • Split ElementsWorker API helpers and unit tests in sub files
  • Drop TranscriptionType & DataSource as they are not used anymore in Arkindex
  • Retry all managed API calls that result in a 50x


Released on 8 April 2021 • View on Gitlab

  • Support weak SSL DH key when downloading images (needed for some outdated IIIF servers with old SSL certs).


Released on 2 March 2021 • View on Gitlab

  • Support new Arkindex feature Worker Activity, to track process progress.
  • Add new API helpers:
  • list_element_children
  • list_transcriptions
  • create_metadata
  • Extend git support with merge & rebase operations
  • Allow any worker type in cookiecutter template


Released on 8 December 2020 • View on Gitlab

  • Bugfix to avoid loading remote images from local file system
  • Deprecate TranscriptionType.


Released on 26 November 2020 • View on Gitlab


Released on 23 November 2020 • View on Gitlab

  • Support git base operations to allow workers to clone and checkout repositories
  • Setup automated CI task to update Python dependencies
  • Update arkindex-client to version 1.0.5.


Released on 19 October 2020 • View on Gitlab

  • Update arkindex-client to version 1.0.4.
  • Add API helpers:
    • get_worker_version
    • get_worker_version_slug
    • get_ml_result_slug


Released on 30 September 2020 • View on Gitlab


Released on 30 September 2020 • View on Gitlab

  • Support Arkindex secrets for workers, using API but also local storage for developers. More information on Arkindex documentation.
  • Do not crash when a worker tries to create a classification that already exists.


Released on 23 September 2020 • View on Gitlab

  • Automatically create missing Arkindex ML classes when using get_ml_class_id and creating classifications through API helpers.
  • Update arkindex-client to version 1.0.2.


Released on 22 September 2020 • View on Gitlab

  • Update arkindex-client to version 1.0.1.
  • Bugfix on score & confidence type checks in api helpers


Released on 2 September 2020 • View on Gitlab

  • Load worker configuration from Arkindex API, or local file (for developers)
  • Add API helpers:
    • load_corpus_classes
    • get_ml_class_id


Released on 25 August 2020 • View on Gitlab

  • Add API helper create_element_transcriptions
  • Return created instance ID in API helpers
  • Add cookiecutter variables to be able to easily rebuild


Released on 19 August 2020 • View on Gitlab

  • Use WORKER_VERSION_ID environment var in helper methods to identify the worker automatically
  • Add API helpers:
    • create_transcription
    • create_classification
    • create_entity
  • Extend cookiecutter template to generate clean Python packages
  • Add the Timer helper class in tools submodule


Released on 7 August 2020 • View on Gitlab

  • Add API helper create_sub_element
  • Add unit tests in cookiecutter template & base project.
  • Change cookiecutter base to use ElementsWorker


Released on 21 July 2020 • View on Gitlab

Initial version of the base worker, with cookiecutter support to easily create workers using this project.