Bo
Not sure why option `trunk_size` was chosen but it should be `chunk_size` or `batch_size`, I think.
Right now our tests uses unittest, and have things such as ``` class TestActions(unittest.TestCase): def setUp(self): env.reset() self.temp_files = [] def tearDown(self): for f in self.temp_files: if file_target(f).exists(): file_target(f).unlink() ```...
Saw an error message ``` Failed to connect to franklin: Failed to check remote connection franklin.dldcc.bcm.edu:22: Failed to interpolate ssh franklin.dldcc.bcm.edu -i '{'aws': '/home/bpeng/.ssh/pcluster-bopeng.pem'}' -p 22 true: Invalid format specifier...
Currently sos status actively reads multiple files to determine the status of tasks. It can be extremely slow on our cluster when even loading python takes seconds. A more straightforwad...
Nextflow has been used on AWS batch and other environments, and we are still catching up. I think we will need to checkout how nextflow handles S3 etc and figure...
``` $ sos status WARNING: INFO: t4t485980d519c783 started INFO: Note: NumExpr detected 24 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. INFO: NumExpr defaulting to 8 threads....
When submitting workflows and tasks to a cluster (eg ``` %run check_validation -q yale_hpc_task_spooler -r yale_hpc_task_spooler ``` is there a way to synchronize the output back to my local computer?...
https://gist.github.com/kfei/5c9222dad94c3e588fe7 could help a bit. This would reduce the number of processes on the server, but might slow down the server since file copying will take a bit longer.
Using `sos server` + zmq would be a more robust way to pass needed information to remote host for execution.
SoS has been in `beta` for a few years and indeed we have changed features of SoS from time to time, including the latest work on remote executions. I am...