Skip to content

Corpus

arkindex_worker.worker.corpus

BaseWorker methods for corpora.

Classes

CorpusExportState

Bases: Enum

State of a corpus export.

Attributes
Created class-attribute instance-attribute 
Created = 'created'

The corpus export is created, awaiting its processing.

Running class-attribute instance-attribute 
Running = 'running'

The corpus export is being built.

Failed class-attribute instance-attribute 
Failed = 'failed'

The corpus export failed.

Done class-attribute instance-attribute 
Done = 'done'

The corpus export ended in success.

CorpusMixin

Functions
download_latest_export 
download_latest_export() -> _TemporaryFileWrapper

Download the latest export in done state of the current corpus.

Returns:

Type Description
_TemporaryFileWrapper

The downloaded export stored in a temporary file.
Source code in arkindex_worker/worker/corpus.py 
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
def download_latest_export(self) -> _TemporaryFileWrapper:
    """
    Download the latest export in `done` state of the current corpus.

    :returns: The downloaded export stored in a temporary file.
    """
    # List all exports on the corpus
    exports = self.api_client.paginate("ListExports", id=self.corpus_id)

    # Find the latest that is in "done" state
    exports: list[dict] = sorted(
        list(
            filter(
                lambda export: export["state"] == CorpusExportState.Done.value,
                exports,
            )
        ),
        key=itemgetter("updated"),
        reverse=True,
    )
    assert (
        len(exports) > 0
    ), f'No available exports found for the corpus ({self.corpus_id}) with state "{CorpusExportState.Done.value.capitalize()}".'

    # Download latest export
    export_id: str = exports[0]["id"]
    logger.info(f"Downloading export ({export_id})...")
    export: _TemporaryFileWrapper = self.api_client.request(
        "DownloadExport", id=export_id
    )
    logger.info(f"Downloaded export ({export_id}) @ `{export.name}`")

    return export