torchio
torchio copied to clipboard
fepegar
FileNotFoundError: File not found: "data"
Is there an existing issue for this?
- [X] I have searched the existing issues
Bug summary
I have searched issue, there is only one similar issue about file not found path. I followed the guide in that issue, update my torchio and torch, but this error still be there.
Code for reproduction
class iosampler(torch.utils.data.Dataset):
def __init__(self,input_list,label_list):
self.input_list=input_list
self.label_list=label_list
self.probalities={0:0,19:1,20:1,21:1,22:1,23:1,24:1}
Subject=[]
for item1,item2 in zip(self.input_list,self.label_list):
Subject.append(
tio.Subject(
input=tio.ScalarImage(item1),
seg=tio.LabelMap(item2))
)
sampler=tio.data.LabelSampler(
patch_size=128,
label_probabilities=self.probalities,
)
self.training_Set=tio.SubjectsDataset(Subject)
self.Queue=tio.Queue(
self.training_Set,
4,
4,
sampler,
num_workers=8,
shuffle_subjects=True,
shuffle_patches=True,
)
.....
TrainDataset=iosampler(Tr_data_list,Tr_label_list)
DataLoader=torch.utils.data.DataLoader(TrainDataset.Queue,shuffle=True,batch_size=8)
Actual outcome
raise FileNotFoundError(f'File not found: "{path}"') The path value is a posixpath value: data
Error messages
Traceback (most recent call last):**
File "../main.py", line 119, in main
for i,batch in enumerate(DataLoader):
File "../torch/utils/data/dataloader.py", line 530, in __next__
data = self._next_data()
File "../torch/utils/data/dataloader.py", line 570, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "../torch/utils/data/_utils/fetch.py", line 52, in fetch
return self.collate_fn(data)
File "../torch/utils/data/_utils/collate.py", line 157, in default_collate
return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})
File "../torch/utils/data/_utils/collate.py", line 157, in <dictcomp>
return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})
File "../torch/utils/data/_utils/collate.py", line 157, in default_collate
return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})
File "../torchio/data/image.py", line 750, in __init__
super().__init__(*args, **kwargs)
File "../torchio/data/image.py", line 161, in __init__
self.path = self._parse_path(path)
File "../torchio/data/image.py", line 431, in _parse_path
return [self._parse_single_path(p) for p in path]
File "../torchio/data/image.py", line 431, in <listcomp>
return [self._parse_single_path(p) for p in path]
File "../torchio/data/image.py", line 419, in _parse_single_path
raise FileNotFoundError(f'File not found: "{path}"')
FileNotFoundError: File not found: "data"
Expected outcome
normal dataloader
System info
Ubuntu: Ubuntu 20.04.3 LTS
torch 1.12.1+cu113
torchaudio 0.12.1+cu113
torchio 0.18.84
torchvision 0.13.1+cu113
Hi, @Fuzhiyuan. Could you please share a minimal reproducible example that I can try on my machine?
I am struggling with this error as well.
Here is the code I execute
I usually just use tio.Subject(tio.ScalarImage(tensor=array) but added the extra bits to be sure
I tried to feed a torch tensor and a numpy array
subject_dict = {'scan' : tio.ScalarImage(tensor=array, type=tio.INTENSITY, path=None),}
subject = tio.Subject(subject_dict) # use torchio subject to enable using grid sampling
grid_sampler = tio.inference.GridSampler(subject, patch_size=roiSize, patch_overlap=patch_overlap,
padding_mode='mean')
patch_loader = torch.utils.data.DataLoader(grid_sampler, batch_size=1)
aggregator = tio.inference.GridAggregator(grid_sampler, overlap_mode='crop') # average for bc
model.eval()
with torch.no_grad():
for i, patch_batch in tqdm(enumerate(patch_loader)): # This is the line that gives me the error
Here is the error
Traceback (most recent call last):
File "/home/ak18001/code/colloidoscope/scripts/training/scampi.py", line 232, in <module>
train(config, name, dataset_path=dataset_path, dataset_name=dataset_name,
File "/home/ak18001/code/colloidoscope/scripts/training/scampi.py", line 184, in train
losses = test(model, dataset_path, dataset_name, test_data, run=run,
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/colloidoscope/train_utils.py", line 414, in test
df, pred_positions, label = dc.detect(d['array'], diameter = detection_diameter, model=model, debug=True)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/colloidoscope/deepcolloid.py", line 43, in detect
return detect(*args, **kwargs)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/colloidoscope/predict.py", line 115, in detect
for i, patch_batch in tqdm(enumerate(patch_loader)):
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/tqdm/std.py", line 1180, in __iter__
for obj in iterable:
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
data = self._next_data()
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
return self.collate_fn(data)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
return collate(batch, collate_fn_map=default_collate_fn_map)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 128, in collate
return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 128, in <dictcomp>
return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 128, in collate
return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 625, in __init__
super().__init__(*args, **kwargs)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 149, in __init__
self.path = self._parse_path(path)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 404, in _parse_path
return [self._parse_single_path(p) for p in path]
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 404, in <listcomp>
return [self._parse_single_path(p) for p in path]
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 394, in _parse_single_path
raise FileNotFoundError(f'File not found: "{path}"')
FileNotFoundError: File not found: "data"
System info
It FAILS on this machine:
Platform: Linux-5.4.0-131-generic-x86_64-with-glibc2.17
TorchIO: 0.18.67
PyTorch: 1.13.0+cu116
SimpleITK: 2.1.1 (ITK 5.2)
NumPy: 1.20.0
Python: 3.8.12 (default, Oct 12 2021, 13:49:34)
[GCC 7.5.0]
But WORKS on this machine:
Platform: Linux-5.3.0-7648-generic-x86_64-with-glibc2.17
TorchIO: 0.18.67
PyTorch: 1.10.0+cu102
SimpleITK: 2.1.1 (ITK 5.2)
NumPy: 1.20.0
Python: 3.8.12 (default, Oct 12 2021, 13:49:34)
[GCC 7.5.0]
I tried various tio versions from 0.17 to most recent
Closing as I can't reproduce with the code you have shared. Feel free to share a minimal working example I can run on my machine and reopen the issue.
Hello @fepegar thank you for your reply, I have found the source of the error but unsure of how to fix it
Minimum working example
import numpy as np
import torch
import torchio as tio
roiSize = (1,64,64,64)
array = np.zeros(roiSize)
subject_dict = {'array' : tio.ScalarImage(tensor=array, type=tio.INTENSITY, path=None),}
subject = tio.Subject(subject_dict)
grid_sampler = tio.inference.GridSampler(subject, patch_size=(32,32,32), patch_overlap=(16,16,16), padding_mode='mean')
patch_loader = torch.utils.data.DataLoader(grid_sampler, batch_size=1)
aggregator = tio.inference.GridAggregator(grid_sampler, overlap_mode='crop')
for i, patch_batch in enumerate(patch_loader): # <- this raises the error
tensor = patch_batch['array'][tio.DATA]
locations = patch_batch[tio.LOCATION]
aggregator.add_batch(tensor, locations)
output_tensor = aggregator.get_output_tensor()
print(output_tensor.shape)
Error trace
Traceback (most recent call last):
File "/home/ak18001/code/colloidoscope/tests/test_tio_grid.py", line 31, in <module>
for i, patch_batch in enumerate(patch_loader):
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
data = self._next_data()
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
return self.collate_fn(data)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
return collate(batch, collate_fn_map=default_collate_fn_map)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 128, in collate
return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 128, in <dictcomp>
return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 128, in collate
return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 712, in __init__
super().__init__(*args, **kwargs)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 157, in __init__
self.path = self._parse_path(path)
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 420, in _parse_path
return [self._parse_single_path(p) for p in path]
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 420, in <listcomp>
return [self._parse_single_path(p) for p in path]
File "/home/ak18001/miniconda3/envs/col/lib/python3.8/site-packages/torchio/data/image.py", line 410, in _parse_single_path
raise FileNotFoundError(f'File not found: "{path}"')
FileNotFoundError: File not found: "data"
The source of the error is when using
PyTorch: 1.13.0+cu116
While it works with
PyTorch: 1.10.0+cu102
I require PyTorch 1.13.0+cu116 because I have a new GPU that needs this version of torch and cuda
If you have a recommendation for a fix I'd be happy to make a PR, for the life of me I can't figure out why _parse_single_path() is reading the path "data"
The only fix I can think of is to add this to line 157 of /torchio/data/image
if tensor is None:
self.path = self._parse_path(path)
Thanks for the MWE, @wahabk. I've tried reproducing on PyTorch 1.11 and 1.13, but I don't get the error. Can you please pip install --upgrade torchio and try again?
Hi @fepegar , I can't use a more modern tio since my project uses numba which relies on numpy<=1.20 :man_facepalming: , I think tio needs new numpy for its typing features
My requirements are quite niche so I will just use the monai patch sampler, feel free to close the issue
Ah, good catch. I've investigated and it turns out that NumPy typing was added in 1.20, so the requirements are wrong. Given that it's used only once, I think we can just replace the typing hint with simply np.ndarray, and remove the corresponding import. Would you like to contribute with a PR? We can then release a new version of TorchIO that will be compatible with your NumPy requirement.
@fepegar Looks like you beat me to it and incorporated this fix in this commit 38ae6c1efd00685a1c7bec4bbfff54f14f2a29a1
I will pull it and test it
Looks all good from my side :+1: thanks so much for the fix :)
It now works on this machine
Platform: Linux-5.4.0-131-generic-x86_64-with-glibc2.17
TorchIO: 0.18.85
PyTorch: 1.13.0+cu117
SimpleITK: 2.2.0 (ITK 5.3)
NumPy: 1.20.0
Python: 3.8.12 (default, Oct 12 2021, 13:49:34)
[GCC 7.5.0]
I was just testing whether the CI would be happy with the changes. Then I thought I give you the chance to contribute. I'll just make you a co-author of the merge commit, if that's ok.