eager_few_shot_od_training_tflite / how to add 2 new classes

[shazz]

Prerequisites

Please answer the following question for yourself before submitting an issue.

• [X] I checked to make sure that this issue has not been filed already.

1. The entire URL of the documentation with the issue

https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/eager_few_shot_od_training_tflite.ipynb

2. Describe the issue

Hi. Could you explain a little bit more the data preparation ? Particularly how to add not only 1 new class but 2 (rubber_ducky and dog for example). This part is not that clear:

# By convention, our non-background classes start counting at 1.  Given
# that we will be predicting just one class, we will therefore assign it a
# `class id` of 1.
duck_class_id = 1
num_classes = 1

category_index = {duck_class_id: {'id': duck_class_id, 'name': 'rubber_ducky'}}

# Convert class labels to one-hot; convert everything to tensors.
# The `label_id_offset` here shifts all classes by a certain number of indices;
# we do this here so that the model receives one-hot labels where non-background
# classes start counting at the zeroth index.  This is ordinarily just handled
# automatically in our training binaries, but we need to reproduce it here.
label_id_offset = 1
train_image_tensors = []
gt_classes_one_hot_tensors = []
gt_box_tensors = []
for (train_image_np, gt_box_np) in zip(
    train_images_np, gt_boxes):
  train_image_tensors.append(tf.expand_dims(tf.convert_to_tensor(
      train_image_np, dtype=tf.float32), axis=0))
  gt_box_tensors.append(tf.convert_to_tensor(gt_box_np, dtype=tf.float32))
  zero_indexed_groundtruth_classes = tf.convert_to_tensor(
      np.ones(shape=[gt_box_np.shape[0]], dtype=np.int32) - label_id_offset)
  gt_classes_one_hot_tensors.append(tf.one_hot(
      zero_indexed_groundtruth_classes, num_classes))

Thanks !

[srjoglekar246]

If you wanted to add two new classes:

You would need to change num_classes to 3 (1 + 2 more) Add ids for each class starting from 1 (Like duck_class_id = 1) Then in category_index just map each class' ID to a dict containing id & name keys.

Some of the later code for inference might also need to change (if it assumes there are only two classes). Some of the Python errors there will be decently verbose to debug.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[srjoglekar246]

@shazz Did your issue get resolved?

[shazz]

Unfortunately no, later parts in the code also expect only one class. I'll post the details on Monday.

[shazz]

Hi @srjoglekar246,

That we have tried as suggested:

# By convention, our non-background classes start counting at 1.  Given
# that we will be predicting just one class, we will therefore assign it a
# `class id` of 1.
duck_class_id = 1
pigeon_class_id = 2
num_classes = 2

category_index = {duck_class_id: {'id': duck_class_id, 'name': 'rubber_ducky'}, {pigeon_class_id : {'id': pigeon_class_id , 'name': 'rubber_pigeon'}}

But in the next cell, the sanity check fails as all boxes are tagged rubber_ducky. So I guess there is something missing to map which gt_boxes are rubber_pigeon and which gt_boxes are rubber_ducky in:

label_id_offset = 1
train_image_tensors = []
gt_classes_one_hot_tensors = []
gt_box_tensors = []
for (train_image_np, gt_box_np) in zip(train_images_np, gt_boxes):
  train_image_tensors.append(tf.expand_dims(tf.convert_to_tensor(train_image_np, dtype=tf.float32), axis=0))
  gt_box_tensors.append(tf.convert_to_tensor(gt_box_np, dtype=tf.float32))

  # in this one ?
  zero_indexed_groundtruth_classes = tf.convert_to_tensor(np.ones(shape=[gt_box_np.shape[0]], dtype=np.int32) - label_id_offset)

  gt_classes_one_hot_tensors.append(tf.one_hot(zero_indexed_groundtruth_classes, num_classes))
print('Done prepping data.')

How do you make this distinction ?

Thanks!

[srjoglekar246]

Yup. Looking at the documentation for tf.one_hot, the zero_indexed_groundtruth_classes should be a list of all class indices that are true for the training image (after subtracting label_id_offset).

It might be helpful to print out what some of these tensors are, to manipulate them accordingly. IIUC, gt_box_np.shape[0] is always 1, so zero_indexed_groundtruth_classes always equals something like [0]. You can probably just have another list of ground-truth classes, and assign zero_indexed_groundtruth_classes accordingly.

So if its a pigeon, zero_indexed_groundtruth_classes will be [1] (not 2, since label_id_offset is subtracted) in tensor format, which will then be converted to one-hot representation.

[ghost]

Hello, I'm working with @shazz. Based on your answer this is what we changed. Does it make sense to you?

# By convention, our non-background classes start counting at 1.  Given
# that we will be predicting just one class, we will therefore assign it a
# `class id` of 1.
gt_boxes = []

# X_train is a pd.DataFrame containing the boxes information and their classes
for id, row in X_train.iterrows():
  array = np.array([[float(row["top"]), float(row["left"]), float(row["bottom"]), float(row["right"])]], dtype=np.float32)
  gt_boxes.append(array)

classes = []
for id, row in X_train.iterrows():
  array = np.ones(shape=[gt_boxes[idx].shape[0]], dtype=np.int32)*row["class"]
  classes.append(array)

person_class_id = 1
hardhat_class_id = 2

num_classes = 2
category_index = {person_class_id: {'id': person_class_id, 'name': 'Person'},
                  hardhat_class_id: {'id': hardhat_class_id, 'name': 'HardHat'}}

# Convert class labels to one-hot; convert everything to tensors.
# The `label_id_offset` here shifts all classes by a certain number of indices;
# we do this here so that the model receives one-hot labels where non-background
# classes start counting at the zeroth index.  This is ordinarily just handled
# automatically in our training binaries, but we need to reproduce it here.
label_id_offset = 1
train_image_tensors = []
gt_classes_one_hot_tensors = []
gt_box_tensors = []
# print(len(train_images_np),len(gt_boxes), len(classes))
for (train_image_np, gt_box_np, c) in zip(train_images_np, gt_boxes, classes):
  train_image_tensors.append(tf.expand_dims(tf.convert_to_tensor(train_image_np, dtype=tf.float32), axis=0)) #(x,y,z) becomes (1,x,y,z)
  gt_box_tensors.append(tf.convert_to_tensor(gt_box_np, dtype=tf.float32))
  zero_indexed_groundtruth_classes = tf.convert_to_tensor(c - label_id_offset)
  gt_classes_one_hot_tensors.append(tf.one_hot(zero_indexed_groundtruth_classes, num_classes))

print('Done prepping data.')

Still waiting for the result as it takes time to train. Thanks :) !

[srjoglekar246]

I think this should work, atleast for the correct class labels part :-)

[omar16100]

@lorynebissuel @shazz Is it working now?

[nkulkarni3297]

@lorynebissuel @shazz Were you guys able to resolve this issue? Also I want to know something here regarding the passing of row["class"] part. Was it of type object/ string or was it integer index? Because when I am trying with object/string type, it shows me error at zero_indexed_groundtruth_classes = tf.convert_to_tensor(c - label_id_offset). And when I am trying with index values, it takes it without issue but even my output shows only one class instead of multiple classes.

Interestingly I am referring the tf2 colab file linked below https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/eager_few_shot_od_training_tf2_colab.ipynb

It would be great if anyone can help me with this.

[Leci37]

here, you have a good example of preditions with eager_few_shot_od_training_tf2_colab.ipynb multiclasses Allomyrina vs Lucanidae object box detection label_map= { 0: "Allomyrina dichotomus" 1: "Lucanidae"}

Full example: MultiClasses in eager_few_shot_od_training_tf2_colab.zip

It is anwser : https://stackoverflow.com/questions/73905337/how-to-perform-object-detection-model-training-on-more-than-1-class/75310788#75310788

Origin autor : https://github.com/tensorflow/models/issues/8862#issuecomment-997397188

[Leci37]

We are looking for multiclass detection with starting with: https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/eager_few_shot_od_training_tflite.ipynb

And I find 3 differents ways to see the for loop for (train_image_np, gt_box_np, c) in zip(train_images_np, gt_boxes, classes):

@Leci37 I explain , in this question they make it slightly different than you (the way to introduce the multiclasses detection to the model ) https://stackoverflow.com/questions/73905337/how-to-perform-object-detection-model-training-on-more-than-1-class/75310788#75310788 @nkulkarni3297 Read here could help you

@wayne931121 And then in the [eager_few_shot_training.zip](https://github.com/tensorflow/models/files/7740892/eager_few_shot_training.zip) of this answer, for example they don't use the label_id_offset = 1 to subtract one to the id_classes https://github.com/tensorflow/models/issues/8862#issuecomment-997397188

@lorynebissuel your respont https://github.com/tensorflow/models/issues/9655#issuecomment-771198843

Do you know which one works ? do you have good experiences with those solutions ?

In relation of @tombstone , @jch1, @pkulzc , @jvishnuvardhan , @kumariko of https://github.com/tensorflow/models/issues/10239 . Are both the same question