sciluigi
sciluigi copied to clipboard
pharmbio
A light-weight wrapper library around Spotify's Luigi workflow library to make writing scientific workflows more fluent, flexible and modular
Addresses #66. The newest version of luigi has dropped `six` which leads to errors if we want to use sciluigi. I went ahead and removed `six` usages from sciluigi and...
see: https://github.com/spotify/luigi/commit/ee3618ce0b9f7967d33cc817ca52f94fb996ab56 This now no longer allows sciluigi to work with the latest luigi. either: change setup to require version 2.8.13 of luigi, or likewise remove python 2 support.
In `setup.py`, there are dependencies on `boto3` and `psycopg2` (i.e. the AWS SDK for Python and the PostgreSQL database adaptor). I don't see any references to these anywhere in the...
README.md states that Luigi versions 1.3.x-2.0.1 are supported, but setup.py only has install_requires luigi, so the latest (v3) package is installed instead. This change captures the requirement, so that if...
I have modified sciluigi to allow ContainerTasks (ie tasks that run in a container, using docker, singularity on HPC clusters, or AWS), including some file handling work to make this...
relative paths break logging if workflow changes working directory: 2017-12-19 14:50:43 | ERROR | Error in event callback for 'event.core.processing_time' Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/luigi/task.py", line 155, in...
I found SciLuigi creates 'log' directory and writes out internal log files in there. But this produces an issue that the application cannot run if it is running from a...
Hi all, I am getting a weird results with my pipeline. I have a pipeline with two subworkflow. If I run my last_task in the SubSubworkflow the pipeline run in...
In Python3, strings by default are Unicode, not ASCII. Therefore, the output of subprocess commands is of the "bytes", rather than "str", type. This makes re.search/match crash inside the function.
Good afternoon, I have to report that adding the "-A {pr}" clause inside the salloc command does cause salloc to crash silently on my system, probably for configuration issues inside...