YAML configuration¶

This page is a reference for version 2 of the YAML configuration file for Git repositories handled by Arkindex. Version 1 is not supported.

The configuration file is always named .arkindex.yml and should be found at the root of the repository.

Required attributes¶

The following attributes are required in every .arkindex.yml file:

version: Version of the configuration file in use. An error will occur if the version number is not set to 2.

Example configuration¶

---
version: 2
workers:
  - workers/config.yml

This would match workers/config.yml starting at the root of the repository.

Worker repository attributes¶

The workers attribute is a list of the following:

Paths to a YAML file holding the configuration for a single worker
Unix-style patterns matching paths to YAML files holding the configuration for a single worker
The configuration of a single worker embedded directly into the file

Single worker configuration¶

The following describes the attributes of a YAML file configuring one worker, or of the configuration embedded directly in the .arkindex.yml file.

All attributes are optional unless explicitly specified.

name

Mandatory. Name of the worker, for display purposes.

slug

Mandatory. Slug of this worker. The slug must be unique across the repository and must only hold alphanumerical characters, underscores or hyphens.

type

Mandatory. Type of the worker, for display purposes only. Some common values include:

classifier
recognizer
ner
dla
word-segmenter
paragraph-creator

gpu_usage

Whether or not this worker requires or supports GPUs. Defaults to disabled. May take one of the following values:

required: This worker requires a GPU, and will only be run on Ponos agents whose hosts have a GPU.
supported: This worker supports using a GPU, but may run on any available host, including those without GPUs.
disabled: This worker does not support GPUs. It may run on a host that has a GPU, but it will ignore it.

model_usage

Whether or not this worker requires a model version to run. Defaults to disabled. May take one of the following values:

required: This worker requires a model version, and will only be run on processes with a model.
supported: This worker supports a model version, but may run on any processes, including those without model.
disabled: This worker does not support model version. It may run on a process that has a model, but it will ignore it.

docker

Regroups Docker-related configuration attributes:

build

Path towards a Dockerfile used to build this worker, relative to the root of the repository. Defaults to Dockerfile.
command

Custom command line to be used when launching the Docker container for this Worker. By default, the command specified in the Dockerfile will be used.
shm_size: Size of the available shared memory in /dev/shm. The default value is 64M, but when training machine learning models an increase might be necessary. The given value must be either an integer, or an integer followed by a unit (b for bytes, k for kilobytes, m for megabytes and g for gigabytes). If no unit is specified, the default unit is bytes. See the Docker documentation.
environment

Mapping of string keys and string values to define environment variables to be set when the Docker image runs.

configuration

Mapping holding any string keys and values that can be later accessed in the worker’s Python code. Can be used to define settings on your own worker, such as a file’s location.

user_configuration

Mapping defining settings on your worker that can be modified by users. See below for details.

secrets

List of required secret names for that specific worker. For more information, learn how to use secrets in workers on the official Arkindex documentation.

Setting up user-configurable parameters¶

The YAML file can define parameters that users will be able to change when they use this worker in a process on Arkindex. These parameters are listed in a user_configuration attribute.

A parameter is defined using the following settings:

title

Mandatory. The parameter’s title.

type

Mandatory. A value type. The supported types are:

int
bool
float
string
enum
list
dict
model

default

Optional. A default value for the parameter. Must be of the defined parameter type.

required

Optional. A boolean, defaults to false.

choices

Optional. A list of options for enum type parameters.

subtype

Optional. The type of the elements of list type parameters.

This definition allows for both validation of the input and the display of a form to make configuring workers easy for Arkindex users.

User configuration

String parameters¶

String-type parameters must be defined using a title and the string type. You can also set a default value for this parameter, which must be a string, as well as make it a required parameter, which prevents users from leaving it blank.

For example, a string-type parameter can be defined like this:

subfolder_name:
  title: Created Subfolder Name
  type: string
  default: My Neat Subfolder

Which will result in the following display for the user:

String-type parameter

Integer parameters¶

Integer-type parameters must be defined using a title and the int type. You can also set a default value for this parameter, which must be an integer, as well as make it a required parameter, which prevents users from leaving it blank.

For example, an integer-type parameter can be defined like this:

input_size:
  title: Input Size
  type: int
  default: 768
  required: True

Which will result in the following display for the user:

integer-type parameter

Float parameters¶

Float-type parameters must be defined using a title and the float type. You can also set a default value for this parameter, which must be a float, as well as make it a required parameter, which prevents users from leaving it blank.

For example, a float-type parameter can be defined like this:

wip:
  title: Word Insertion Penalty
  type: float
  required: True

Which will result in the following display for the user:

Float-type parameter

Boolean parameters¶

Boolean-type parameters must be defined using a title and the bool type. You can also set a default value for this parameter, which must be a boolean, as well as make it a required parameter, which prevents users from leaving it blank.

In the configuration form, boolean parameters are displayed as toggles.

For example, a boolean-type parameter can be defined like this:

score:
  title: Run Worker in Evaluation Mode
  type: bool
  default: False

Which will result in the following display for the user: Boolean-type parameter

Enum (choices) parameters¶

Enum-type parameters must be defined using a title, the enum type and at least two choices. You cannot define an enum-type parameter without choices. You can also set a default value for this parameter, which must be one of the available choices, as well as make it a required parameter, which prevents users from leaving it blank. Enum-type parameters should be used when you want to limit the users to a given set of options.

In the configuration form, enum parameters are displayed as selects.

For example, an enum-type parameter can be defined like this:

parent_type:
  title: Target Parent Element Type
  type: enum
  default: paragraph
  choices:
    - paragraph
    - text_zone
    - page

Which will result in the following display for the user:

Enum-type parameter

List parameters¶

List-type parameters must be defined using a title, the list type and a subtype for the elements inside the list. You can also set a default value for this parameter, which must be a list containing elements of the given subtype, as well as make it a required parameter, which prevents users from leaving it blank.

The allowed subtypes are int, float and string.

In the configuration form, list parameters are displayed as rows of input fields.

For example, a list-type parameter can be defined like this:

a_list:
  title: A List of Values
  type: list
  subtype: int
  default: [4, 3, 12]

Which will result in the following display for the user:

List-type parameter

Dictionary parameters¶

Dictionary-type parameters must be defined using a title and the dict type. You can also set a default value for this parameter, which must be a dictionary, as well as make it a required parameter, which prevents users from leaving it blank. You can use dictionary parameters for example to specify a correspondence between the classes that are predicted by a worker and the elements that are created on Arkindex from these predictions.

Dictionary-type parameters only accept strings as values.

In the configuration form, dictionary parameters are displayed as a table with one column for keys and one column for values.

For example, a dictionary-type parameter can be defined like this:

classes:
  title: Output Classes to Elements Correspondence
  type: dict
  default:
    a: page
    b: text_line

Which will result in the following display for the user:

Dictionary-type parameter

Model parameters¶

Model-type parameters must be defined using a title and the model type. You can also set a default value for this parameter, which must be the UUID of an existing Model, and make it a required parameter, which prevents users from leaving it blank. You can use a model parameter to specify to which Model the Model Version that is created by a Training process will be attached.

Model-type parameters only accept Model UUIDs as values.

In the configuration form, model parameters are displayed as an input field. Users can select a model from a list of available Models: what they type into the input field filters that list, allowing them to search for a model using its name or UUID.

For example, a model-type parameter can be defined like this:

model_param:
  title: Training Model
  type: model

Which will result in the following display for the user:

Model-type parameter

Example user_configuration¶

user_configuration:
  vertical_padding:
    type: int
    default: 0
    title: Vertical Padding
  element_base_name:
    type: string
    required: true
    title: Element Base Name
  create_confidence_metadata:
    type: bool
    default: false
    title: Create confidence metadata on elements
  some_other_parameter:
    type: enum
    required: true
    default: 23
    choices:
      - 12
      - 23
      - 56
    title: Another Parameter
  a_model_parameter:
    type: model
    title: Model to train

Fallback to free JSON input¶

If you have defined user-configurable parameters using these specifications, Arkindex users can choose between using the form or the free JSON input field by toggling the JSON toggle. If there are unsupported parameter types in the defined user_configuration, the frontend will automatically fall back to the free JSON input field. The same is true if you have not defined user-configurable parameters using these specifications.

Example configuration¶

---
version: 2

workers:
  # Path to a single YAML file
  - path/to/worker.yml
  # Pattern matching any YAML file in the configuration folder
  # or in its sub-directories
  - configuration/**/*.yml
  # Configuration embedded directly into this file
  - name: Book of hours
    slug: book_of_hours
    type: classifier
    docker:
      build: project/Dockerfile
      image: hub.docker.com/project/image:tag
      command: python mysuperscript.py --blabla
      shm_size: 128m
      environment:
        TOKEN: deadBeefToken
    configuration:
      model: path/to/model
      anyKey: anyValue
      classes: [X, Y, Z]
    user_configuration:
      vertical_padding:
        type: int
        default: 0
        title: Vertical Padding
    secrets:
      - path/to/secret.json