error in generate tfrecord in Anaconda Tensorflow

[aaa12328]

when I use this command python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record and python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=data/test.record

both of them have this error

Traceback (most recent call last): File "generate_tfrecord.py", line 20, in from object_detection.utils import dataset_util ImportError: No module named 'object_detection'

how to fix it ?

[555timer]

Any luck? I have the exact same issue.

[AdityaVallabh]

This file is essentially what you need. You could copy it to your directory and import dataset_util.

Or if you already have the Tensorflow Object Detection API repo cloned, you could go to tensorflow/models/research/ and execute

# From tensorflow/models/research/
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

[rahulramesh3321]

@aaa12328, @AdityaVallabh Hi , I am facing issue in creating TFrecords and i am actually stuck here from last 2 days. Please help me. Below are the error :

File "D:\anaconda\lib\site-packages\absl\flags_flagvalues.py", line 473, in getattr raise AttributeError(name)

AttributeError: output_path

[AdityaVallabh]

@rahulramesh3321 which line exactly in generate_tfrecord.py is raising the error?

Did you run the file with the appropriate flags? python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record

[rahulramesh3321]

Hi Aditya,

I am able to solve it. I found that i was doing something wrong with syntax.

On Thu, Feb 7, 2019 at 12:16 AM AdityaVallabh [email protected] wrote:

@rahulramesh3321 https://github.com/rahulramesh3321 which line exactly in generate_tfrecord.py https://github.com/datitran/raccoon_dataset/blob/master/generate_tfrecord.py is raising the error?

Did you run the file with the appropriate flags? python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/datitran/raccoon_dataset/issues/81#issuecomment-461139784, or mute the thread https://github.com/notifications/unsubscribe-auth/AoNw3kb6BehPbnmNjUDzAAlOSz7wrC5Wks5vKyMDgaJpZM4Zkac5 .

-- Thanks and Regards Rahul Singh Chandel " "

[Aashiq24]

hi, i am having the same issue as @rahulramesh3321. i have been struggling to find a solution but to no avail. i am using pycharm with conda environment, python 3.7, tensorflow 1.14.

i get the following error message each time i try to run the program.

**Traceback (most recent call last): File "C:/Users/MSI-GF63/Desktop/raccoon_dataset-master/generate_tfrecord.py", line 100, in tf.compat.v1.app.run() File "C:\Users\MSI-GF63\anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\MSI-GF63\anaconda3\lib\site-packages\absl\app.py", line 299, in run _run_main(main, args) File "C:\Users\MSI-GF63\anaconda3\lib\site-packages\absl\app.py", line 250, in _run_main sys.exit(main(argv)) File "C:/Users/MSI-GF63/Desktop/raccoon_dataset-master/generate_tfrecord.py", line 86, in main writer = tf.compat.v1.python_io.TFRecordWriter(FLAGS.out_path) File "C:\Users\MSI-GF63\anaconda3\lib\site-packages\tensorflow\python\platform\flags.py", line 85, in getattr return wrapped.getattr(name) File "C:\Users\MSI-GF63\anaconda3\lib\site-packages\absl\flags_flagvalues.py", line 473, in getattr raise AttributeError(name) AttributeError: output_path

Process finished with exit code 1**

can someone please help me with this?

[Homayra22]

Hi I am having the same problem,but with tensor flow. Whenever I try to generate tf record with python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record

I get the error

Traceback (most recent call last): File "C:\Users\homay\OneDrive\Documents\Object API\Glass\generate_tfrecord.py", line 17, in import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow'

[tanishkthomas]

when I use this command python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=data/train.record and python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=data/test.record

both of them have this error

Traceback (most recent call last): File "generate_tfrecord.py", line 20, in from object_detection.utils import dataset_util ImportError: No module named 'object_detection'

how to fix it ?

You just need to write "from utils import dataset_util" instead of "from object_detection.utils import dataset_util" in the generate_tfrecord,py file.

Because the object_detection library consists of the utils package, hence you can access it directly..

[Homayra22]

Thanks 😌

[vaibhavthapli]

Hi, I got an error when creating TFRecord

code : !python {files['TF_RECORD_SCRIPT']} -x {os.path.join(paths['IMAGE_PATH'], 'validation')} -l {files['LABELMAP']} -o {os.path.join(paths['ANNOTATION_PATH'], 'train.record')}

Error; Traceback (most recent call last): File "Tensorflow\scripts\generate_tfrecord.py", line 168, in tf.app.run() File "C:\Users\vaibh\miniconda3\envs\CV\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\vaibh\miniconda3\envs\CV\lib\site-packages\absl\app.py", line 312, in run _run_main(main, args) File "C:\Users\vaibh\miniconda3\envs\CV\lib\site-packages\absl\app.py", line 258, in _run_main sys.exit(main(argv)) File "Tensorflow\scripts\generate_tfrecord.py", line 155, in main examples = xml_to_csv(args.xml_dir) File "Tensorflow\scripts\generate_tfrecord.py", line 88, in xml_to_csv int(member[4][0].text), IndexError: child index out of range

[rubalsharma0mr0]

That's already written

[akshikamde21]

Hi I am having the same problem,but with tensor flow. Whenever I try to generate tf record with python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record

I get the error

Traceback (most recent call last): File "C:\Users\homay\OneDrive\Documents\Object API\Glass\generate_tfrecord.py", line 17, in import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow'

hi what did you do to resolve this error? pls help me. i am facing same issue.

[rubalsharma0mr0]

Yes, it got resolved. Actually, there were some changes in the code: Refer to this link: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html#create-tensorflow-records It resolved my problem.

Replace the code in your generate_tfrecord.py file with this:

""" Sample TensorFlow XML-to-TFRecord converter

usage: generate_tfrecord.py [-h] [-x XML_DIR] [-l LABELS_PATH] [-o OUTPUT_PATH] [-i IMAGE_DIR] [-c CSV_PATH]

optional arguments: -h, --help show this help message and exit -x XML_DIR, --xml_dir XML_DIR Path to the folder where the input .xml files are stored. -l LABELS_PATH, --labels_path LABELS_PATH Path to the labels (.pbtxt) file. -o OUTPUT_PATH, --output_path OUTPUT_PATH Path of output TFRecord (.record) file. -i IMAGE_DIR, --image_dir IMAGE_DIR Path to the folder where the input image files are stored. Defaults to the same directory as XML_DIR. -c CSV_PATH, --csv_path CSV_PATH Path of output .csv file. If none provided, then no file will be written. """

import os import glob import pandas as pd import io import xml.etree.ElementTree as ET import argparse

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # Suppress TensorFlow logging (1) import tensorflow.compat.v1 as tf from PIL import Image from object_detection.utils import dataset_util, label_map_util from collections import namedtuple

Initiate argument parser

parser = argparse.ArgumentParser( description="Sample TensorFlow XML-to-TFRecord converter") parser.add_argument("-x", "--xml_dir", help="Path to the folder where the input .xml files are stored.", type=str) parser.add_argument("-l", "--labels_path", help="Path to the labels (.pbtxt) file.", type=str) parser.add_argument("-o", "--output_path", help="Path of output TFRecord (.record) file.", type=str) parser.add_argument("-i", "--image_dir", help="Path to the folder where the input image files are stored. " "Defaults to the same directory as XML_DIR.", type=str, default=None) parser.add_argument("-c", "--csv_path", help="Path of output .csv file. If none provided, then no file will be " "written.", type=str, default=None)

args = parser.parse_args()

if args.image_dir is None: args.image_dir = args.xml_dir

label_map = label_map_util.load_labelmap(args.labels_path) label_map_dict = label_map_util.get_label_map_dict(label_map)

def xml_to_csv(path): """Iterates through all .xml files (generated by labelImg) in a given directory and combines them in a single Pandas dataframe.

Parameters:
----------
path : str
    The path containing the .xml files
Returns
-------
Pandas DataFrame
    The produced dataframe
"""

xml_list = []
for xml_file in glob.glob(path + '/*.xml'):
    tree = ET.parse(xml_file)
    root = tree.getroot()
    for member in root.findall('object'):
        value = (root.find('filename').text,
                 int(root.find('size')[0].text),
                 int(root.find('size')[1].text),
                 member[0].text,
                 int(member[4][0].text),
                 int(member[4][1].text),
                 int(member[4][2].text),
                 int(member[4][3].text)
                 )
        xml_list.append(value)
column_name = ['filename', 'width', 'height',
               'class', 'xmin', 'ymin', 'xmax', 'ymax']
xml_df = pd.DataFrame(xml_list, columns=column_name)
return xml_df

def class_text_to_int(row_label): return label_map_dict[row_label]

def split(df, group): data = namedtuple('data', ['filename', 'object']) gb = df.groupby(group) return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]

def create_tf_example(group, path): with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid: encoded_jpg = fid.read() encoded_jpg_io = io.BytesIO(encoded_jpg) image = Image.open(encoded_jpg_io) width, height = image.size

filename = group.filename.encode('utf8')
image_format = b'jpg'
xmins = []
xmaxs = []
ymins = []
ymaxs = []
classes_text = []
classes = []

for index, row in group.object.iterrows():
    xmins.append(row['xmin'] / width)
    xmaxs.append(row['xmax'] / width)
    ymins.append(row['ymin'] / height)
    ymaxs.append(row['ymax'] / height)
    classes_text.append(row['class'].encode('utf8'))
    classes.append(class_text_to_int(row['class']))

tf_example = tf.train.Example(features=tf.train.Features(feature={
    'image/height': dataset_util.int64_feature(height),
    'image/width': dataset_util.int64_feature(width),
    'image/filename': dataset_util.bytes_feature(filename),
    'image/source_id': dataset_util.bytes_feature(filename),
    'image/encoded': dataset_util.bytes_feature(encoded_jpg),
    'image/format': dataset_util.bytes_feature(image_format),
    'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
    'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
    'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
    'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
    'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
    'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_example

def main(_):

writer = tf.python_io.TFRecordWriter(args.output_path)
path = os.path.join(args.image_dir)
examples = xml_to_csv(args.xml_dir)
grouped = split(examples, 'filename')
for group in grouped:
    tf_example = create_tf_example(group, path)
    writer.write(tf_example.SerializeToString())
writer.close()
print('Successfully created the TFRecord file: {}'.format(args.output_path))
if args.csv_path is not None:
    examples.to_csv(args.csv_path, index=None)
    print('Successfully created the CSV file: {}'.format(args.csv_path))

if name == 'main': tf.app.run()

[vaibhavthapli]

if we use Kaggle data we have to change the index value of member make it 5. and try.

On Wed, Feb 9, 2022 at 4:43 PM Rubal Sharma @.***> wrote:

Yes, it got resolved. Actually, there were some changes in the code:

Replace the code in your generate_tfrecord.py file with this:

""" Sample TensorFlow XML-to-TFRecord converter

usage: generate_tfrecord.py [-h] [-x XML_DIR] [-l LABELS_PATH] [-o OUTPUT_PATH] [-i IMAGE_DIR] [-c CSV_PATH]

optional arguments: -h, --help show this help message and exit -x XML_DIR, --xml_dir XML_DIR Path to the folder where the input .xml files are stored. -l LABELS_PATH, --labels_path LABELS_PATH Path to the labels (.pbtxt) file. -o OUTPUT_PATH, --output_path OUTPUT_PATH Path of output TFRecord (.record) file. -i IMAGE_DIR, --image_dir IMAGE_DIR Path to the folder where the input image files are stored. Defaults to the same directory as XML_DIR. -c CSV_PATH, --csv_path CSV_PATH Path of output .csv file. If none provided, then no file will be written. """

import os import glob import pandas as pd import io import xml.etree.ElementTree as ET import argparse

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # Suppress TensorFlow logging (1) import tensorflow.compat.v1 as tf from PIL import Image from object_detection.utils import dataset_util, label_map_util from collections import namedtuple Initiate argument parser

parser = argparse.ArgumentParser( description="Sample TensorFlow XML-to-TFRecord converter") parser.add_argument("-x", "--xml_dir", help="Path to the folder where the input .xml files are stored.", type=str) parser.add_argument("-l", "--labels_path", help="Path to the labels (.pbtxt) file.", type=str) parser.add_argument("-o", "--output_path", help="Path of output TFRecord (.record) file.", type=str) parser.add_argument("-i", "--image_dir", help="Path to the folder where the input image files are stored. " "Defaults to the same directory as XML_DIR.", type=str, default=None) parser.add_argument("-c", "--csv_path", help="Path of output .csv file. If none provided, then no file will be " "written.", type=str, default=None)

args = parser.parse_args()

if args.image_dir is None: args.image_dir = args.xml_dir

label_map = label_map_util.load_labelmap(args.labels_path) label_map_dict = label_map_util.get_label_map_dict(label_map)

def xml_to_csv(path): """Iterates through all .xml files (generated by labelImg) in a given directory and combines them in a single Pandas dataframe.

Parameters:

path : str The path containing the .xml files Returns

Pandas DataFrame The produced dataframe """

xml_list = [] for xml_file in glob.glob(path + '/*.xml'): tree = ET.parse(xml_file) root = tree.getroot() for member in root.findall('object'): value = (root.find('filename').text, int(root.find('size')[0].text), int(root.find('size')[1].text), member[0].text, int(member[4][0].text), int(member[4][1].text), int(member[4][2].text), int(member[4][3].text) ) xml_list.append(value) column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax'] xml_df = pd.DataFrame(xml_list, columns=column_name) return xml_df

def class_text_to_int(row_label): return label_map_dict[row_label]

def split(df, group): data = namedtuple('data', ['filename', 'object']) gb = df.groupby(group) return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]

def create_tf_example(group, path): with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid: encoded_jpg = fid.read() encoded_jpg_io = io.BytesIO(encoded_jpg) image = Image.open(encoded_jpg_io) width, height = image.size

filename = group.filename.encode('utf8') image_format = b'jpg' xmins = [] xmaxs = [] ymins = [] ymaxs = [] classes_text = [] classes = []

for index, row in group.object.iterrows(): xmins.append(row['xmin'] / width) xmaxs.append(row['xmax'] / width) ymins.append(row['ymin'] / height) ymaxs.append(row['ymax'] / height) classes_text.append(row['class'].encode('utf8')) classes.append(class_text_to_int(row['class']))

tf_example = tf.train.Example(features=tf.train.Features(feature={ 'image/height': dataset_util.int64_feature(height), 'image/width': dataset_util.int64_feature(width), 'image/filename': dataset_util.bytes_feature(filename), 'image/source_id': dataset_util.bytes_feature(filename), 'image/encoded': dataset_util.bytes_feature(encoded_jpg), 'image/format': dataset_util.bytes_feature(image_format), 'image/object/bbox/xmin': dataset_util.float_list_feature(xmins), 'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs), 'image/object/bbox/ymin': dataset_util.float_list_feature(ymins), 'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs), 'image/object/class/text': dataset_util.bytes_list_feature(classes_text), 'image/object/class/label': dataset_util.int64_list_feature(classes), })) return tf_example

def main(_):

writer = tf.python_io.TFRecordWriter(args.output_path) path = os.path.join(args.image_dir) examples = xml_to_csv(args.xml_dir) grouped = split(examples, 'filename') for group in grouped: tf_example = create_tf_example(group, path) writer.write(tf_example.SerializeToString()) writer.close() print('Successfully created the TFRecord file: {}'.format(args.output_path)) if args.csv_path is not None: examples.to_csv(args.csv_path, index=None) print('Successfully created the CSV file: {}'.format(args.csv_path))

if name == 'main': tf.app.run()

— Reply to this email directly, view it on GitHub https://github.com/datitran/raccoon_dataset/issues/81#issuecomment-1033647077, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALHGAPVLXBHBTRCABR72V3LU2JD53ANCNFSM4GMRU44Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

[HridoyHazard]

i solved this issue by rename correctly the xml file (inside of it) and the image file. both have to be in same name otherwise you can't generate tfrecord.

[nishabaruwal576]

could you elaborate more on this?