biothings.api icon indicating copy to clipboard operation
biothings.api copied to clipboard

DockerContainerDumper class

Open newgene opened this issue 1 year ago • 0 comments

This can be a new type of Dumper class, which triggers a docker container (typically runs on a different server) to run and generate the output file, and then stop the container. The dumper class will then get the processed file(s) and send it to the Uploader as normally.

Typically, this processed file can be a NDJSON file (one JSON per line), so the uploader class can be quite simple and generic.

The typical use case is some complex workflow with heavy dependencies, so we can isolate them in a docker container.

newgene avatar Oct 25 '22 23:10 newgene