Docker images throw error for inference

[michaelroyzen]

I tried running both Docker images as described and got the same error each time. From the conceptual-captions directory, I ran python /conceptual-captions/generate_caption.py test_images.txt test_images/ my_output.txt. My hardware is 2x 2080 Ti (11G VRAM), but I also ran it on Tesla T4 (16G VRAM) on another system and received the same error.

It resulted in this error: F0228 00:44:42.409458 41 cudnn_conv_layer.cu:28] Check failed: status == CUDNN_STATUS_SUCCESS (8 vs. 0) CUDNN_STATUS_EXECUTION_FAILED *** Check failure stack trace: *** @ 0x7f3f56f2e5cd google::LogMessage::Fail() @ 0x7f3f56f30433 google::LogMessage::SendToLog() @ 0x7f3f56f2e15b google::LogMessage::Flush() @ 0x7f3f56f30e1e google::LogMessageFatal::~LogMessageFatal() @ 0x7f3f578d3c83 caffe::CuDNNConvolutionLayer<>::Forward_gpu() @ 0x7f3f5784abb1 caffe::Net<>::ForwardFromTo() @ 0x7f3f583822bb boost::python::objects::caller_py_function_impl<>::operator()() @ 0x7f3f5737f5cd boost::python::objects::function::call() @ 0x7f3f5737f7c8 (unknown) @ 0x7f3f57387613 boost::python::handle_exception_impl() @ 0x7f3f5737c999 (unknown) @ 0x7f3f66878b73 (unknown) @ 0x7f3f6690f119 PyType_Ready @ 0x7f3f66914a99 best_base @ 0x7f3f6689d7c7 (unknown) @ 0x7f3f66878b73 (unknown) @ 0x7f3f6690da6e PyType_Modified @ 0x7f3f66914a99 best_base @ 0x7f3f66911f68 slot_tp_repr @ 0x7f3f66914a99 best_base @ 0x7f3f66911f68 slot_tp_repr @ 0x7f3f6691354e slot_nb_xor @ 0x7f3f66914a99 best_base @ 0x7f3f66914cba type_new @ 0x7f3f6692e01d _PyUnicodeUCS4_ToNumeric @ 0x7f3f6692f1c8 _PyUnicodeUCS4_ToNumeric @ 0x7f3f669303e8 warn_explicit @ 0x7f3f6694267c ast_for_comprehension @ 0x7f3f65b13830 __libc_start_main @ 0x55fdf1f4507f (unknown) Aborted (core dumped) mv: cannot stat 'feats_att': No such file or directory mv: cannot stat 'feats_fc': No such file or directory mv: cannot stat 'feats_box': No such file or directory /opt/conda/lib/python2.7/site-packages/smart_open/ssh.py:34: UserWarning: paramiko missing, opening SSH/SCP/SFTP paths will be disabled. pip install paramikoto suppress warnings.warn('paramiko missing, opening SSH/SCP/SFTP paths will be disabled. pip install paramiko` to suppress') DataLoader loading json file: testdata/testtalk.json DataLoader loading h5 file: testdata/feats_fc testdata/feats_att testdata/feats_box none read 18 image features assigned 18 images to split train assigned 18 images to split val assigned 18 images to split test Traceback (most recent call last): File "eval_ensemble.py", line 96, in vars(opt)) File "/conceptual-captions/captioning/eval_utils.py", line 123, in eval_split data = loader.get_batch(split) File "/conceptual-captions/captioning/dataloader.py", line 219, in get_batch ix, tmp_wrapped = self._prefetch_process[split].get() File "/conceptual-captions/captioning/dataloader.py", line 425, in get tmp = self.split_loader.next() File "/opt/conda/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 637, in next return self._process_next_batch(batch) File "/opt/conda/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch raise batch.exc_type(batch.exc_msg) IOError: Traceback (most recent call last): File "/opt/conda/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop samples = collate_fn([dataset[i] for i in batch_indices]) File "/conceptual-captions/captioning/dataloader.py", line 318, in getitem att_feat = self.att_loader.get(str(self.info['images'][ix]['id'])) File "/conceptual-captions/captioning/dataloader.py", line 56, in get feat = self.loader(f_input) File "/conceptual-captions/captioning/dataloader.py", line 31, in self.loader = lambda x: np.load(x)['feat'] File "/opt/conda/lib/python2.7/site-packages/numpy/lib/npyio.py", line 370, in load fid = open(file, "rb") IOError: [Errno 2] No such file or directory: 'testdata/feats_att/0006400c1c224e19.jpg.npz'

Terminating BlobFetcher`

Please let me know how I can fix this error. Thanks!