medaka
medaka copied to clipboard
medaka smolecule concurrent.futures.process.BrokenProcessPool
Describe the bug When I run medaka smolecule I often get this error, a few data sets work but usually its this error. 2 of the 20 samples I have go through fine but the rest all fail like this. I think it's a bug with medaka rather than my command or data. I have python 3.8 in the conda environment.
Logging
[07:35:05 - DataIndx] Loaded 1/1 (100.00%) sample files.
terminate called after throwing an instance of 'std::system_error'
what(): Invalid argument
Exception in thread QueueManagerThread:
Traceback (most recent call last):
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/process.py", line 394, in _queue_management_worker
work_item.future.set_exception(bpe)
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/_base.py", line 547, in set_exception
raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x2ac3f09ac910 state=cancelled>
Traceback (most recent call last):
File "/home/rking/miniconda3/envs/UMI/bin/medaka", line 11, in <module>
sys.exit(main())
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/site-packages/medaka/medaka.py", line 720, in main
args.func(args)
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/site-packages/medaka/smolecule.py", line 428, in main
medaka.stitch.stitch(args)
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/site-packages/medaka/stitch.py", line 231, in stitch
for (rname, start, stop), seq_parts, qualities in contigs:
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/site-packages/medaka/stitch.py", line 157, in collapse_neighbours
for contig in contigs:
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
for element in iterable:
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
terminate called after throwing an instance of 'std::system_error'
what(): Invalid argument
Environment (if you do not have a GPU, write No GPU):
- not using GPU, medaka 1.6.1, conda install
Additional context using UMI amplicons with medaka smolecule.
I don't get the error if I split the files up to 100 pieces then run medaka on each. So got a workaround at least.
actually no this is still failing for some files with that future error even chopping up the input. For one set of data it went through fine. now I'm trying another and it's failing at the medaka stage even though dividing up to 100 files and 1 cpu per job. I've set TMPDIR to local location now and increased mem per job to 2G, so see if that has an effect. Normally tool issues are tmp or mem but it might be more annoying.
Were you able to find a reliable workaround for this? I'm struggling to get around this issue
@rob234king
Please try v1.7.0, which contains some amendments to the smolecule
command to fix this issue in at least some circumstnaces.