CLoDSA
CLoDSA copied to clipboard
problem in instance-segmentation
I have 53 RGB images and their corresponding annotation in coco JSON format. I used the colab notebook (instance-segmentation) from the example and just replaced datasets with mine. It starts the processes, generated few images andgives the following error. I can't figure it out. Please help. I am running it on google colab pro.
`/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/process_executor.py:700: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
warnings.warn(
ERROR:concurrent.futures:exception calling callback for <Future at 0x7f40abc1a640 state=finished raised TerminatedWorkerError>
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/_base.py", line 26, in _invoke_callbacks
callback(self)
File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 385, in __call__
self.parallel.dispatch_next()
File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 834, in dispatch_next
if not self.dispatch_one_batch(self._original_iterator):
File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 901, in dispatch_one_batch
self._dispatch(tasks)
File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 819, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py", line 556, in apply_async
future = self._workers.submit(SafeFunction(func))
File "/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/reusable_executor.py", line 176, in submit
return super().submit(fn, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/process_executor.py", line 1129, in submit
raise self._flags.broken
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
The exit codes of the workers are {SIGSEGV(-11)}
---------------------------------------------------------------------------
TerminatedWorkerError Traceback (most recent call last)
[<ipython-input-39-1d9c8eaf67b6>](https://localhost:8080/#) in <module>
----> 1 augmentor.applyAugmentation()
13 frames
[/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/process_executor.py](https://localhost:8080/#) in submit(self, fn, *args, **kwargs)
1127 with self._flags.shutdown_lock:
1128 if self._flags.broken is not None:
-> 1129 raise self._flags.broken
1130 if self._flags.shutdown:
1131 raise ShutdownExecutorError(
TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
The exit codes of the workers are {SIGSEGV(-11)}`
I'm running into the same problem. Even with 32gb of ram, I can only run augmentations on 2 images at a time. Seems to use lots of memory depending on how many annotations there are.