biothings.api
biothings.api copied to clipboard
DockerContainerDumper class
This can be a new type of Dumper class, which triggers a docker container (typically runs on a different server) to run and generate the output file, and then stop the container. The dumper class will then get the processed file(s) and send it to the Uploader as normally.
Typically, this processed file can be a NDJSON file (one JSON per line), so the uploader class can be quite simple and generic.
The typical use case is some complex workflow with heavy dependencies, so we can isolate them in a docker container.