multi=False causes Errno 29 illegal seek during pytest teardown

[donaldcampbelljr]

          Reproduce by setting the multi flag to false or just leaving it out:

def test_temp_test():
    pp = pypiper.PipelineManager(
        "sample_pipeline", outfolder="/home/drc/Downloads/BED_CLASSIFIER_OUTPUT/", multi=True
    )

If you set the flag multi=True, it works fine and we no longer get the illegal seek error.

Originally posted by @donaldcampbelljr in https://github.com/databio/bedboss/issues/30#issuecomment-1944492602

[nsheff]

is it just that pytests require the multi flag to be set?

not sure this is a bug...

[khoroshevskyi]

hmm, I think it's not.. But what we should do in this case? Should we skip tests or set multi=True

[nsheff]

I think all the pypiper tests set multi=True. it's just what you have to do to use pypiper with pytests.

look at all the existing pypiper tests. I think they're already doing that. This is just a known thing with pypiper, because pytest is capturing output.

[donaldcampbelljr]

I am able to use multi=True to get the tests to pass when using pypiper in our Bedboss tests:

    with TemporaryDirectory() as d:
        pm = pypiper.PipelineManager(
            name="bedclassifier",
            outfolder=d,
            recover=True,
            pipestat_sample_name="Generic_Digest",
            multi=True
        )
        bedclass = BedClassifier(input_file=values[0], output_dir=d, pm=pm)
        pm.complete()

However, terminal gets filled with logging errors claiming that the pipeline failed:

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.10/logging/__init__.py", line 1103, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/home/drc/GITHUB/bedboss/bedboss/.venv2/lib/python3.10/site-packages/pypiper/manager.py", line 2201, in _exit_handler
    self.fail_pipeline(Exception("Pipeline failure. See details above."))
  File "/home/drc/GITHUB/bedboss/bedboss/.venv2/lib/python3.10/site-packages/pypiper/manager.py", line 2036, in fail_pipeline
    self.info("Failure reason: " + str(exc))
  File "/home/drc/GITHUB/bedboss/bedboss/.venv2/lib/python3.10/site-packages/pypiper/manager.py", line 1468, in info
    self._logger.info(msg, *args, **kwargs)
Message: 'Failure reason: Pipeline failure. See details above.'
Arguments: ()
Exception ignored in atexit callback: <bound method PipelineManager._exit_handler of <pypiper.manager.PipelineManager object at 0x7f0cf268aad0>>
Traceback (most recent call last):
  File "/home/drc/GITHUB/bedboss/bedboss/.venv2/lib/python3.10/site-packages/pypiper/manager.py", line 2201, in _exit_handler
    self.fail_pipeline(Exception("Pipeline failure. See details above."))
  File "/home/drc/GITHUB/bedboss/bedboss/.venv2/lib/python3.10/site-packages/pypiper/manager.py", line 2045, in fail_pipeline
    raise exc
Exception: Pipeline failure. See details above.

This becomes exhausting if there is a test failure and you need to read the log.