Previously working code has stopped to work with import error

[jhennies]

Hi all,

Due to my stuff not working anymore complaining about some pytorch internal issue and core-dumps, I re-installed my conda environment but still my previously working code won't work due to some import error (code and error below).

Is this a me-problem or might something be broken on the bioimage-io side?

This is what I have installed:

bioimageio.core           0.5.11             pyhd8ed1ab_0    conda-forge
bioimageio.spec           0.5.1              pyhd8ed1ab_0    conda-forge

Code:

def run_cebra_net(
        input_data
):

    # Set up the model
    try:
        rdf_path = os.path.join(
            get_repo_path(), 'models', 'cebranet-cellular-membranes-in-volume-sem_torchscript.zip'
        )
        print(f'rdf_path = {rdf_path}')
        model_resource = bioimageio.core.load_resource_description(rdf_path)
    except TypeError:
        rdf_doi = "10.5281/zenodo.7274275"
        model_resource = bioimageio.core.load_resource_description(rdf_doi)

    # Prepare the data
    input_array = xr.DataArray(input_data[None, None, :], dims=tuple(model_resource.inputs[0].axes))

    # Set up the prediction pipeline
    devices = None
    weight_format = None
    prediction_pipeline = bioimageio.core.create_prediction_pipeline(
        model_resource, devices=devices, weight_format=weight_format
    )

    # Make sure the input and output shapes match
    prediction_pipeline.input_specs[0].shape = prediction_pipeline.output_specs[0].shape
    tiling = True

    # Predict with tiling
    result = bioimageio.core.predict_with_tiling(prediction_pipeline, input_array, tiling=tiling, verbose=True)[0]

    return (result[0, 0, :] * 255).astype('uint8')

Error:

/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
  backends.update(_get_backends("networkx.backends"))
Traceback (most recent call last):
  File "/media/julian/Data/tmp/test_cebra_em/proj/.snakemake/scripts/tmpehogir6d.run_task.py", line 17, in <module>
    from cebra_em_core.bioimageio.cebra_net import run_cebra_net
  File "/media/julian/Data/src/github/jhennies/CebraEM/cebra-em-core/cebra_em_core/bioimageio/cebra_net.py", line 2, in <module>
    import bioimageio.core
  File "/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/core/__init__.py", line 6, in <module>
    from .resource_io import (
  File "/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/core/resource_io/__init__.py", line 2, in <module>
    from .io_ import (
  File "/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/core/resource_io/io_.py", line 11, in <module>
    from bioimageio.core.resource_io.nodes import ResourceDescription
  File "/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/core/resource_io/nodes.py", line 9, in <module>
    from bioimageio.spec.model import raw_nodes as model_raw_nodes
ImportError: cannot import name 'raw_nodes' from 'bioimageio.spec.model' (/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/spec/model/__init__.py)

[FynnBe]

How did you install spec and core? Those versions are not compatible

[jhennies]

conda create -n bioimageio-test-env -c conda-forge bioimageio.core (I also had other packages in the create command, but this also yields the versions specified above)

[FynnBe]

hmm.. really good to know that the pin we set here does not make it into the conda-recipe...

[jhennies]

So in other words, I would need to downgrade the specs to 0.4.9?

[FynnBe]

currently yes, the new core release is coming very soon though, so maybe wait just a few days if you can

[jhennies]

Ok, that gets rid of the import error. But I'm back to a core dump, looking like this:

prediction with tiling:   0%|                                                                                                                                                                                                                                       | 0/1000 [00:00<?, ?it/s]terminate called after throwing an instance of 'c10::Error'
  what():  isTuple() INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/core/ivalue_inl.h":931, please report a bug to PyTorch. Expected Tuple but got String
Exception raised from toTuple at /pytorch/aten/src/ATen/core/ivalue_inl.h:931 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f2ed5dc88b2 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x2f850b0 (0x7f2f6333a0b0 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x2f873a5 (0x7f2f6333c3a5 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #3: torch::jit::SourceRange::highlight(std::ostream&) const + 0x38 (0x7f2f60fc4d68 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #4: torch::jit::ErrorReport::what() const + 0x2c5 (0x7f2f60fa42d5 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x2b11ce (0x7f2f7137e1ce in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x2c4fa6 (0x7f2f71391fa6 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x14e6b6 (0x55caf49286b6 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #8: _PyObject_MakeTpCall + 0x2ec (0x55caf49117ac in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #9: _PyEval_EvalFrameDefault + 0x4fa0 (0x55caf490df10 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #10: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #11: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #12: _PyEval_EvalFrameDefault + 0x4abe (0x55caf490da2e in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #13: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #14: <unknown function> + 0x14c9fb (0x55caf49269fb in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #15: _PyEval_EvalFrameDefault + 0x11f2 (0x55caf490a162 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #16: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #17: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #18: _PyEval_EvalFrameDefault + 0x696 (0x55caf4909606 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #19: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #20: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #21: <unknown function> + 0x14cadc (0x55caf4926adc in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #22: _PyEval_EvalFrameDefault + 0x45a6 (0x55caf490d516 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #23: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #24: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #25: <unknown function> + 0x14cadc (0x55caf4926adc in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #26: _PyEval_EvalFrameDefault + 0x45a6 (0x55caf490d516 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #27: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #28: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #29: <unknown function> + 0x14cadc (0x55caf4926adc in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #30: _PyEval_EvalFrameDefault + 0x45a6 (0x55caf490d516 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #31: <unknown function> + 0x13f802 (0x55caf4919802 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #32: _PyEval_EvalFrameDefault + 0x3e2 (0x55caf4909352 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #33: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #34: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #35: _PyEval_EvalFrameDefault + 0x3e2 (0x55caf4909352 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #36: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #37: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #38: _PyEval_EvalFrameDefault + 0x11f2 (0x55caf490a162 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #39: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #40: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #41: _PyEval_EvalFrameDefault + 0x11f2 (0x55caf490a162 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #42: <unknown function> + 0x13f802 (0x55caf4919802 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #43: _PyEval_EvalFrameDefault + 0x3e2 (0x55caf4909352 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #44: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #45: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #46: _PyEval_EvalFrameDefault + 0x3e2 (0x55caf4909352 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #47: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #48: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #49: _PyEval_EvalFrameDefault + 0x11f2 (0x55caf490a162 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #50: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #51: _PyEval_EvalCodeWithName + 0x48 (0x55caf4907e58 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #52: PyEval_EvalCodeEx + 0x39 (0x55caf4907e09 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #53: PyEval_EvalCode + 0x1b (0x55caf49b54ab in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #54: <unknown function> + 0x20887a (0x55caf49e287a in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #55: <unknown function> + 0x204c03 (0x55caf49dec03 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #56: <unknown function> + 0x98175 (0x55caf4872175 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #57: PyRun_SimpleFileExFlags + 0x1b1 (0x55caf49d8841 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #58: Py_RunMain + 0x38d (0x55caf49d5c6d in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #59: Py_BytesMain + 0x37 (0x55caf49a92e7 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #60: __libc_start_main + 0xf3 (0x7f2f88108083 in /lib/x86_64-linux-gnu/libc.so.6)
frame #61: <unknown function> + 0x1cf1e1 (0x55caf49a91e1 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)

/usr/bin/bash: line 1: 26594 Aborted                 (core dumped) /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9 /media/julian/Data/tmp/test_cebra_em/proj/.snakemake/scripts/tmp8ml04ipe.run_task.py

Have you seen this before? Or any idea where I might need to look into?

[FynnBe]

looks like you're trying to run a model traced on GPU on CPU (or vice versa??) (libtorch_cpu.so) makes sure inputs and model are on the same device and try GPU as well as CPU.. maybe that will get you somewhere

[jhennies]

Thanks @FynnBe, it was some pytorch installation issue, so a me-problem. From my side the issue is solved, meaning everything works for me (respective the import error). Do you want to keep the issue open as it requires manually specifying the spec version?