tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open:

[StewartTsai]

I have problem with my generate_tfrecord.py How can I solve it?

Error below here

Traceback (most recent call last): File "generate_tfrecord.py", line 100, in tf.app.run() File "D:\Anconda-Python_IDE\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run _sys.exit(main(argv)) File "generate_tfrecord.py", line 91, in main tf_example = create_tf_example(group, path) File "generate_tfrecord.py", line 46, in create_tf_example encoded_jpg = fid.read() File "D:\Anconda-Python_IDE\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 125, in read self._preread_check() File "D:\Anconda-Python_IDE\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 85, in _preread_check compat.as_bytes(self.__name), 1024 * 512, status) File "D:\Anconda-Python_IDE\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 526, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: D:\Tensorflow\models\research\object_detection\image\train\1 : \udca8t\udcb2Χ䤣\udca8\udcec\udcab\udcfc\udca9w\udcaa\udcba\udcc0ɮסC ; No such file or directory

I don't know what it means "No such file or directory" But my directory is correct

generate_tfrecord.py source code here

import pandas as pd import tensorflow as tf

from PIL import Image from object_detection.utils import dataset_util from collections import namedtuple, OrderedDict

flags = tf.app.flags flags.DEFINE_string('csv_input', '', 'Path to the CSV input') flags.DEFINE_string('image_dir', '', 'Path to the image directory') flags.DEFINE_string('output_path', '', 'Path to output TFRecord') FLAGS = flags.FLAGS

TO-DO replace this with label map def class_text_to_int(row_label): if row_label == 'Bactrocera_dorsalis': return 1 else: None

def split(df, group): data = namedtuple('data', ['filename', 'object']) gb = df.groupby(group) return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]

def create_tf_example(group, path): with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid: encoded_jpg = fid.read() encoded_jpg_io = io.BytesIO(encoded_jpg) image = Image.open(encoded_jpg_io) width, height = image.size

filename = group.filename.encode('utf8')
image_format = b'jpg'
xmins = []
xmaxs = []
ymins = []
ymaxs = []
classes_text = []
classes = []

for index, row in group.object.iterrows():
    xmins.append(row['xmin'] / width)
    xmaxs.append(row['xmax'] / width)
    ymins.append(row['ymin'] / height)
    ymaxs.append(row['ymax'] / height)
    classes_text.append(row['class'].encode('utf8'))
    classes.append(class_text_to_int(row['class']))

tf_example = tf.train.Example(features=tf.train.Features(feature={
    'image/height': dataset_util.int64_feature(height),
    'image/width': dataset_util.int64_feature(width),
    'image/filename': dataset_util.bytes_feature(filename),
    'image/source_id': dataset_util.bytes_feature(filename),
    'image/encoded': dataset_util.bytes_feature(encoded_jpg),
    'image/format': dataset_util.bytes_feature(image_format),
    'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
    'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
    'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
    'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
    'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
    'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_example

def main(_): writer = tf.python_io.TFRecordWriter(FLAGS.output_path) path = os.path.join(os.getcwd(), FLAGS.image_dir) examples = pd.read_csv(FLAGS.csv_input) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString())

writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))

if name == 'main': tf.app.run()

---

[srinibujjay]

i am also facing similar error as below: tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: C:\tensorflow1\models\research\object_detection\images\test\Broken rail013.jpg : The system cannot find the file specified. ; No such file or directory

[Dev-HJYoo]

i think your file'size is small, don't you?

[srinibujjay]

The error is rectified by matching jpg file and xml file total count (one xml file for each jpg file) For me the issue is closed now

[sumannelli]

I am also facing the same issue. I checked the quantities of images and .xml files both having the equal count. Provided the valid path. Please help me.

[AyazYousafxai]

make sure in xml file the filename extension is exist or not like train cam_image1.jpg

[AyazYousafxai]

train cam_image1.jpg

[ChamodyaDias]

Check your train_labels.csv & test_labels.csv files and check the "Filename" column. It should have the name of the image as "yourImgName.png / yourImgName.jgp" etc. The extension should be there. Unless, change the code of "xml_to_csv.py" as follows:

def xml_to_csv(path):
    xml_list = []
    for xml_file in glob.glob(path + '/*.xml'):
        tree = ET.parse(xml_file)
        root = tree.getroot()
        for member in root.findall('object'):
            value = (root.find('filename').text + '.png', #Put your file extension here
                     int(root.find('size')[0].text),
                     int(root.find('size')[1].text),
                     member[0].text,
                     int(member[4][0].text),
                     int(member[4][1].text),
                     int(member[4][2].text),
                     int(member[4][3].text)
                     )
            xml_list.append(value)
    column_name = ['filename', 'width', 'height',
               'class', 'xmin', 'ymin', 'xmax', 'ymax']
    xml_df = pd.DataFrame(xml_list, columns=column_name)
    return xml_df

[daniablank]

My CSV has the extensions in it, and I can verify that the image files that it is claiming do not exist, do exist. I can copy the exact filepath in the error and check if it is a file, and it is. In another instance of this issue, I've been told that this error is thrown if the boxes are under 33 pixels on a side, but that isn't the case for the image it is failing to detect. Are there other conditions apart from the filepath not leading to an actual file that cause this error?

[radheyasarode]

Copy the images and xml files into the directory in which your code is. OR

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

this will work `too.```

[solomonvimal]

I have the same issue - but my files are not in .jpg format. Do .tif files work in tensorflow?

[muhammadaamir123]

When I execute this command: python3 train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config

WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons
• https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From train.py:55: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From train.py:55: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

WARNING:tensorflow:From train.py:167: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From train.py:89: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W1212 03:17:10.382213 6232 deprecation_wrapper.py:119] From train.py:89: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From c:\users\aamir\desktop\models\research\object_detection\utils\config_util.py:86: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W1212 03:17:10.384212 6232 deprecation_wrapper.py:119] From c:\users\aamir\desktop\models\research\object_detection\utils\config_util.py:86: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From train.py:94: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.

W1212 03:17:10.388208 6232 deprecation_wrapper.py:119] From train.py:94: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.

WARNING:tensorflow:From c:\users\aamir\desktop\models\research\object_detection\trainer.py:228: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.create_global_step W1212 03:17:10.411310 6232 deprecation.py:323] From c:\users\aamir\desktop\models\research\object_detection\trainer.py:228: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.create_global_step WARNING:tensorflow:From c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py:104: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W1212 03:17:10.418304 6232 deprecation_wrapper.py:119] From c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py:104: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

WARNING:tensorflow:From c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py:119: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

W1212 03:17:10.419303 6232 deprecation_wrapper.py:119] From c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py:119: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

Traceback (most recent call last): File "train.py", line 167, in tf.app.run() File "C:\Users\Aamir\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\Aamir\Anaconda3\lib\site-packages\absl\app.py", line 299, in run _run_main(main, args) File "C:\Users\Aamir\Anaconda3\lib\site-packages\absl\app.py", line 250, in _run_main sys.exit(main(argv)) File "train.py", line 163, in main worker_job_name, is_chief, FLAGS.train_dir) File "c:\users\aamir\desktop\models\research\object_detection\trainer.py", line 235, in train train_config.prefetch_queue_capacity, data_augmentation_options) File "c:\users\aamir\desktop\models\research\object_detection\trainer.py", line 59, in create_input_queue tensor_dict = create_tensor_dict_fn() File "train.py", line 120, in get_next dataset_builder.build(config)).get_next() File "c:\users\aamir\desktop\models\research\object_detection\builders\dataset_builder.py", line 138, in build label_map_proto_file=label_map_proto_file) File "c:\users\aamir\desktop\models\research\object_detection\data_decoders\tf_example_decoder.py", line 195, in init use_display_name) File "c:\users\aamir\desktop\models\research\object_detection\utils\label_map_util.py", line 149, in get_label_map_dict label_map = load_labelmap(label_map_path) File "c:\users\aamir\desktop\models\research\object_detection\utils\label_map_util.py", line 129, in load_labelmap label_map_string = fid.read() File "C:\Users\Aamir\Anaconda3\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 122, in read self._preread_check() File "C:\Users\Aamir\Anaconda3\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 84, in _preread_check compat.as_bytes(self.__name), 1024 * 512) tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: data/object-detection.pbtxt : The system cannot find the file specified. ; No such file or directory

Any solution?

[SUFEHeisenberg]

tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: : ϵͳ\udcd5Ҳ\udcbb\udcb5\u dcbdָ\udcb6\udca8\udcb5\udcc4·\udcbe\udcb6\udca1\udca3 ; No such process I met with the same questions yesterday which confused me all day along 0.0 But today I successed with two changes:

1. Change all path into GetFullPath forms
2. delete the 'space' between the equal and path e.g. --input_file='F:/PycharmProjects/data/tf_train.tfrecord' And then made it. Dunno the reason but you can have a try.

[ajeeth-b]

Your csv file have all the images name under filename column without its type i.e extension, you have to add the format of all image under the filename column in csv file.

[harshita897]

give an alternative for this problem

[JustinsDevAccount]

What helped me in this scenario is to specify the full path, ex. C:\Users\justi\Deep_Learning\Object_Recognition\research\object_detection\images\train INSTEAD OF: images\train Also, keep an eye on if your folder specifiers are '/' or '\' depending on your OS.

[ajithvcoder]

try to print the path and see

def create_tf_example(group, path):
    print(path)
    print(os.path.join(path, '{}'.format(group.filename)))

whether the path is correct or not

[arunkumar27karthi]

whether the code is correct or not

''' Usage:

From tensorflow/models/

Create train data:

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record

Create test data:

python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=test.record ''' from future import division from future import print_function from future import absolute_import

import os import io import pandas as pd import tensorflow.compat.v1 as tf

from PIL import Image from object_detection.utils import dataset_util from collections import namedtuple, OrderedDict

flags = tf.app.flags flags.DEFINE_string('csv_input', '', 'Path to the CSV input') flags.DEFINE_string('output_path', '', 'Path to output TFRecord') flags.DEFINE_string('image_dir', '', 'Path to images') FLAGS = flags.FLAGS

TO-DO replace this with label map

def class_text_to_int(row_label): if row_label == 'men': return 1 elif row_label == 'women': return 2 elif row_label == 'cat': return 3 else: None

def split(df, group): data = namedtuple('data', ['filename', 'object']) gb = df.groupby(group) return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]

def create_tf_example(group, path): with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid: encoded_jpg = fid.read() encoded_jpg_io = io.BytesIO(encoded_jpg) image = Image.open(encoded_jpg_io) width, height = image.size

filename = group.filename.encode('utf8')
image_format = b'jpg'
xmins = []
xmaxs = []
ymins = []
ymaxs = []
classes_text = []
classes = []

for index, row in group.object.iterrows():
    xmins.append(row['xmin'] / width)
    xmaxs.append(row['xmax'] / width)
    ymins.append(row['ymin'] / height)
    ymaxs.append(row['ymax'] / height)
    classes_text.append(row['class'].encode('utf8'))
    classes.append(class_text_to_int(row['class']))

tf_example = tf.train.Example(features=tf.train.Features(feature={
    'image/height': dataset_util.int64_feature(height),
    'image/width': dataset_util.int64_feature(width),
    'image/filename': dataset_util.bytes_feature(filename),
    'image/source_id': dataset_util.bytes_feature(filename),
    'image/encoded': dataset_util.bytes_feature(encoded_jpg),
    'image/format': dataset_util.bytes_feature(image_format),
    'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
    'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
    'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
    'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
    'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
    'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_example

def main(_): writer = tf.python_io.TFRecordWriter(FLAGS.output_path) path = os.path.join(FLAGS.image_dir) examples = pd.read_csv(FLAGS.csv_input) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString())

writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))

if name == 'main': tf.app.run()

[ajithvcoder]

@arunkumar27karthi what is the output u are getting ? any erro

[arunkumar27karthi]

i got this error for the above code

2020-03-08 21:22:29.141659: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll Traceback (most recent call last): File "generate_tf_records.py", line 104, in tf.app.run() File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 299, in run _run_main(main, args) File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 250, in _run_main sys.exit(main(argv)) File "generate_tf_records.py", line 95, in main tf_example = create_tf_example(group, path) File "generate_tf_records.py", line 50, in create_tf_example encoded_jpg = fid.read() File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\lib\io\file_io.py", line 122, in read self._preread_check() File "C:\Users\prem kumar\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\lib\io\file_io.py", line 84, in _preread_check compat.as_bytes(self.__name), 1024 * 512) tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: images/train\cat1.jpg : The system cannot find the path specified. ; No such process

[arunkumar27karthi]

is i should download the racoon dataset and work on it ?

[ajithvcoder]

u can use any dataset . see this path . NewRandomAccessFile failed to Create/Open: images/train\cat1.jpg . its wrong .

give the path manually in below mentioned path variable

def main(_):
writer = tf.python_io.TFRecordWriter(FLAGS.output_path)
path = os.path.join(FLAGS.image_dir)          

you can check but adding print in below function

def create_tf_example(group, path):
    print(path)
    print(os.path.join(path, '{}'.format(group.filename)))

[shivprasad94]

i had same issue, but this had something to do with my images/train folder and respective xml files miss match and label.csv

even if there is small issue/ miss match in images and xml you will get error...

step 1 : check - image/train/cat1.jpg step 2 : check the file name col data in your- train_labels.csv the name should be exactly matching in images/train and train_labels.csv (verify with all images,xml)

once everything is done run below command..

python generate_tfrecord.py --csv_input=images/train_labels.csv --image_dir=images/train --output_path=train.record

[niraj3819]

Copy the images and xml files into the directory in which your code is. OR

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

this will work `too.```

i tried but it didn't worked for me. can you please suggest some new method.

[arunkumar27karthi]

Can I recognise object using tensorflow from matplotlib 3d view

On Tue, 28 Apr 2020, 5:39 pm niraj3819, [email protected] wrote:

Copy the images and xml files into the directory in which your code is. OR

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

this will work `too.```

i tried but it didn't worked for me. can you please suggest some new method.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/EdjeElectronics/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10/issues/225#issuecomment-620565341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANORDGLDIAHJ7736BP6LWNTRO3BPNANCNFSM4GUAWCZA .

[CHERYZ]

Hi, I solved this question just modified generat_tfrecord.py code like this: Change the path to your own file path. And then just run the generate_tfrecord.py

[SrihariKalyanaraman]

I got this error and I made a small change in the generate_tfRecords.py code

While mentioning the path in main function instead of images give '/images/train ' while generating tfRecords for Train data and give '/images/test ' while generating tfRecords for test data

Few Warnings might come due to tensorflow version compatibility , but the records are generated successfully.

def main(_): writer = tf.python_io.TFRecordWriter(FLAGS.output_path) path = os.path.join(os.getcwd(), 'images/train/') # I have changed in this line examples = pd.read_csv(FLAGS.csv_input) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString())

writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))

[Armand-Eicker-Dev]

@SrihariKalyanaraman i was searching this thread looking for answers, the fix you posted above SOLVED my problem. Just wanted to say thanks...sometimes the fix can be right under our noses.

[ljubantomic01]

I got this error and I made a small change in the generate_tfRecords.py code

While mentioning the path in main function instead of images give '/images/train ' while generating tfRecords for Train data and give '/images/test ' while generating tfRecords for test data

Few Warnings might come due to tensorflow version compatibility , but the records are generated successfully.

def main(_): writer = tf.python_io.TFRecordWriter(FLAGS.output_path) path = os.path.join(os.getcwd(), 'images/train/') # I have changed in this line examples = pd.read_csv(FLAGS.csv_input) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString())

writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))

Thank you very much!!!

[Petros626]

try to print the path and see

def create_tf_example(group, path):
    print(path)
    print(os.path.join(path, '{}'.format(group.filename)))

whether the path is correct or not

where i must run these commands? in anaconda prompt as admin?

[alwaysaditi]

Copy the images and xml files into the directory in which your code is. OR

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record --image_dir=(image directory)

this will work `too.```

thanks a lot for this!it worked for me