sos
sos copied to clipboard
vatlab
SoS workflow system for daily data analysis
As we have found out in #1181, file locking is slow and can cause trouble in high-intensity operations. #1181 involves a lock that locks all config files which is shared...
For docker based steps we do pull from the web the docker image every time we run them. The problem is that with the prompt below: ``` INFO: Running 5:...
There is an [`active` option of task](https://vatlab.github.io/sos-docs/doc/documentation/Remote_Execution.html#Option-active). The usage is generally as demontrated in the example ``` !echo "hello" > a.txt task: active=not path('a.txt').exists() sh: echo "Task executed" ``` or...
A typical work patternn for `ipyparallel` is to start multiple local or remote workers and `ipyparallel` would send multiple tasks to them using `map`, `map_async` etc. `ipyparallel` would handle task...
I've got a multi-step workflow running on PBS system. I only had the first step completed, with an error message: ``` ERROR: 47 jobs completed, 2 jobs aborted ``` I...
Right now we are requiring that there is one and one step that generates the required target, something like ``` [convert: provides='{filename}.csv'] input: f'{filename}.xlsx' sh: expand=True xlsx2csv {_input} {_output} [plot]...
We have magic `%sessioninfo` to report sessioninfo for all involved kernels. SoS has a `report` feature, which can be used to report such information but there is no standard way...
This can be useful but would be applicable to only conda environment. Of course `conda(['p1', 'p2'])` needs to be allowed.
This also seems an attractive feature and is perhaps even more appealing than #1047 because one can build docker images and use that from singularity (unless even that involves `sudo`)
An obvious advantage of singularity images is that it can be transferred to remote hosts. For example, if we build an `app.simg` locally, and run it with ``` run: container='app.simg'...