How to use class weight when detecting object?

[cam4ani]

Hello everyone,

First a big thanks for this incredible work!!! I am really impressed by the segmentation part! I am trying to use this implementation of MASK-RCNN on unbalanced dataset. I would have liked to use 'class_weight' parameter in the fit_generator() function from the mrcnn/model.py file. But, I think this wont be correct, as It should not be influencing the segmentation network for example. Hence, I am a bit confused, how to use during training only a weight value for each class in the loss function when doing the detection?

I hope to have some help here :) in any case thank you for all!

[cam4ani]

Anyone would have some suggestion please?

[lucascannelloni]

Hi,

I have the same issue with unbalanced training data. Did you find a way to weigh the classes? @cam4ani

[HjalteMann]

Same issue here.

[Amrimn]

@cam4ani @cam4ani @lucascannelloni Hello, does anyone find a solution? I am facing the same problem.

[bboug17]

Hello, I have the same problem. Can anyone help? thank you.

[Altimis]

Hello, did anyone find a solution for this please ?

[KazuhideMimura]

Hi, I have the same problem. If anyone has actually used "class_weight", I want to know whether the results improved, or how you set the weight of each class. (I'm not sure about theoretical part...)

I defined class_weight dictionary in model.py and added "class_weight = class_weight" at the parameters of self.keras_model.fit_generator (around line 2400 of model.py). I could run the program, but the result was not so much improved.

I suppose that class_weight[0] is for the weight of class "background", but the exact number of background is unknown. So I guessed approximate number of background to determine the weight of class[0].

[Altimis]

@KazuhideMimura

Hey friend. I have succeeded at using the class_weight and the results were impoved significantly. To use it you need to do further steps :

1. In your training script, create a dictonary containing the id and the number of occurence of each class. example :

CLASS_WEIGHTS = { 1:600, #A 2:10, #B 3:5, #C }

2. create a function that computes the weights :

`def compute_class_weights(CLASS_WEIGHTS=CLASS_WEIGHTS):

mean = np.array(list(CLASS_WEIGHTS.values())).mean() # sum_class_occurence / nb_classes
max_weight = np.array(list(CLASS_WEIGHTS.values())).max()
CLASS_WEIGHTS.update((x, float(max_weight/(y))) for x, y in CLASS_WEIGHTS.items())
CLASS_WEIGHTS=dict(sorted(CLASS_WEIGHTS.items()))

return CLASS_WEIGHTS`

3. when calling the train function, you need to add a new parameter class_weight in the main train function localized in your model.py code :

def train(self, train_dataset, val_dataset, learning_rate, epochs, layers, class_weight=None, augmentation=imgaug.augmenters.fliplr(0.5), custom_callbacks=None, no_augmentation_sources=None)

when calling fit_generator, add the class_weight parameter :

if class_weight==None: self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=1, #modified from workers to 1 (besuace the training stucks in epoch 1 or 2 or 3) use_multiprocessing=False, #modified from true to false (besuace the training stucks in epoch 1 or 2 or 3) )

else : self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=1, #modified from workers to 1 (besuace the training stucks in epoch 1 or 2 or 3) use_multiprocessing=False, #modified from true to false (besuace the training stucks in epoch 1 or 2 or 3) class_weight = class_weight #modified : added to the fit_generator for class_wieghts ) self.epoch = max(self.epoch, epochs)

Note that if the ratio between your classes distributions is too big (like in this case), this solution doesnt change much. You need to have at most a ratio from 1 to 10 (based on my experience)

[Altimis]

@KazuhideMimura As far as i know, you cant guess the number of of background, it could be an infinit number. The number of the background depends on the choice of the number of anchors and ROI to train. This implementation of Mask R-CNN takes a rato of 1:3 of ROI to be considered as positif (objects) and the others as negatif (background). So when using class_weight, you need to care only about the classes of your object, (All these informations arent 100% sure, they are just based on my personal experience)

[KazuhideMimura]

@Altimis Thank you for quick response and kind advises. I am trying one class detection from images including a huge number of objects I don't want to detect, and I was wondering whether changing class_weight of background might be some help.

I think I need to search for another way. Thank you!!

[Altimis]

In this case setting a class weight will not help. You need to decrease the false positives (undesirable detections) . There are many ways to do so :

1. increase detection_min_confidence (0.93 in my case works fine)

2. increase NMS_tresh (0.9 in my case works fine)

3. increase the number of anchors and ROI to train. in this way, you will provide your model more informations about the negatif class (background)

4. use data augmentation to make your model more robust

5. you can also regularise the loss_weights. by increasing the rpn_class_loss and rpn_bbox_loss that are responsible for distinguishing between the background (negatif label) and the object (positif label), at the level of the RPN.

LOSS_WEIGHTS = { "rpn_class_loss": 5., "rpn_bbox_loss": 5., "mrcnn_class_loss": 1., "mrcnn_bbox_loss": 1., "mrcnn_mask_loss": 1 }

[acivit]

Hi! @Altimis @KazuhideMimura

I did the same changes you proposed in the model.py file, but i got the following error:

ValueError: Provided class_weight was a list of 3 elements, but the model has 0 outputs. You should provide one class_weightarray per model output.

I tried using class weight as a dict and as a list but any of them worked for me. I wonder if someone faced the same issue and how you solved it :)

Thanks!

[KazuhideMimura]

@acivit Hi, I haven't seen such error code, so I just write a change I've made in my code.

I changed L. 2364-2375 of model.py as follows. self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=workers, use_multiprocessing=True, class_weight={1: 0.5, 2: 5.0}, # added )

Isn't it work?

[KazuhideMimura]

@Altimis Thank you for your kind advice (last year). I labeled all the noise and set a class_weight. It took so much time, but it was effective.

[acivit]

@acivit Hi, I haven't seen such error code, so I just write a change I've made in my code.

I changed L. 2364-2375 of model.py as follows. self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=workers, use_multiprocessing=True, class_weight={1: 0.5, 2: 5.0}, # added )

Isn't it work?

Hi @KazuhideMimura thanks for your reply.

My 'model.py' file from l 2358 to 2371 is:

self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=1, #verbose=1, use_multiprocessing=False, class_weight=class_weight )

And i got the following error:

raise ValueError('Unknown entries in {} dictionary: {}. Only expected '

ValueError: Unknown entries in class_weight dictionary: [1, 2, 3, 4]. Only expected following keys: []

Any ideas? Did you change somehow the model? Thanks!

[KazuhideMimura]

@acivit I didn't change any other part for setting class_weight.

Where's the class_weight dictionary defined in your code? I once tried to define the weight dictionary at outside of model.py, which didn't work. So I calculate weights of each class beforehand and directly write as an argument of self.keras_model.fit_generator(

Although this may not be the best solution, it worked.

[acivit]

@KazuhideMimura

I define the class_weight in a custom main function where i call the model.train ... take a look:

The function get_class_weights returns a dict in form {1: weight1, 2: weight2, 3: weight3 , ...} and I've tried aswell a list like: [weight1, weight2, weight3 ...]

    class_weights = dataset_train.get_class_weights()

    dataset_train.prepare()
    
    # Validation dataset
    dataset_val.load_dataset()
    dataset_val.prepare()

    # Training - Stage 1
    print("Training network heads")
    model.train(
        dataset_train,
        dataset_val,
        learning_rate=config.LEARNING_RATE,
        epochs=60,
        class_weight=class_weights,
        layers="heads",
    )

    # Training - Stage 2
    # Finetune layers from ResNet stage 4 and up
    print("Fine tune Resnet stage 4 and up")
    model.train(
        dataset_train,
        dataset_val,
        learning_rate=config.LEARNING_RATE,
        epochs=130,
        class_weight=class_weights,
        layers="4+",
    )

    # Training - Stage 3
    # Fine tune all layers
    print("Fine tune all layers")
    model.train(
        dataset_train,
        dataset_val,
        learning_rate=config.LEARNING_RATE / 10,
        epochs=220,
        class_weight=class_weights,
        layers="all",
    )

I will try defining the dictionary inside model.py, lets see what happens :)

Thanks!

######### EDIT

Tried defining the dict inside model.py and... :

raise ValueError('Unknown entries in {} dictionary: {}. Only expected '

ValueError: Unknown entries in class_weight dictionary: [1, 2, 3, 4]. Only expected following keys: []

Same error!

[MahBadran93]

@acivit Did you find any solution to the weight class problem ? If so, please can you put the solution.

[acivit]

Hi @MahBadran93, I couldn't solve the problem yet, I think I will try to implement the class weights when defining the class_losses in the model.py file directly.

[MahBadran93]

Thanks @acivit, How to do that ? you mean changing the LOSS_WEIGHTS ?

[acivit]

@MahBadran93 If you check the model.py file https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/model.py , at the graph_loss functions (for example at the line 1076)

def mrcnn_class_loss_graph(target_class_ids, pred_class_logits, active_class_ids):

At the end of the function it computes the loss for each image for the active classes in that image

loss = loss * pred_active

Then it computes the mean of each loss

loss = tf.reduce_sum(loss) / tf.reduce_sum(pred_active)

If between those lines you multiply loss * class_weights (adapting the format of the data to be able to compute the operation, I haven't tested yet but in my case loss is a tensor and class_weights a dict or list, so there must be some change done)

There you prioritize the loss for each class... or you should.

Does it make any sense? What do you think? If you test it please tell me how did it work :D

In the following week I will be doing some tests with those changes to check how it works.

Hope it works!

[MahBadran93]

Thank you very much @acivit , I will see if it works. I will update you. I opened another issue describing for my case here: https://github.com/matterport/Mask_RCNN/issues/2545

[acivit]

Hi! Any update?

Some days ago I opened an issue aswell: #2535 But still have no answer (neither hope that someones answers with the perfect answer :D)

[MahBadran93]

Hi @acivit. Sorry, I still have no solution. I am trying and if I have a solution I will put it here.

[acivit]

Hello @MahBadran93

I tested with those changes and both precision and recall for an imbalanced object increased! (The magnitude of the data was: 200k, 200k, 100k, 4k)

For the object of 100k it increased from 50% to 80%. And the other (4k) is detected most of the times!

[MahBadran93]

Hello @acivit. Thank you very much. Did you make the same changes you explained in your previous answer right? I will test them and I hope it works. Can I ask you how you evaluate precision and recall for each class?

[acivit]

Hi @MahBadran93, exactly, I did those changes :)

It is complicated to explain how do I compute the precision and recall, but in the end i just make predictions with the trained model with new data (or data that wasn't used for training / validation) and I compare it with the grountruth data.

[MahBadran93]

Hi @acivit. Thank you very much. For precision and recall. am trying to implement for each class but the only thing I was able to find is the mean Average precision for all the images not per class. If you have the time to explain how it is done I would really appreciated it.

[acivit]

Hello @MahBadran93, sorry for the late answer.

My implementation to compute the precision and recall is done outside the package. I just have a validation dataset (that wasn't use to train the model) with the boxes and masks (to use it as a groundtruth), then i run the model and use those images at the validation dataset.

Imagine you are working just with the bounding boxes, I compute the intersection area between the results from the model and the groundtruth, if they match a threshold (like 60% intersection area) that bounding box is assigned to the grountruth, and then i check for the true positives, false negatives and false positives from all the blocks assigned (comparing the class).

I may have explained kinda awful, but that's the concept :)

Best of luck!

[MahBadran93]

Thank you very much @acivit. Can I ask if it is possible to show your implementation? Another question: I am trying to add class weights as you described before. I multiplied the mrcnn class loss with class weights(dictionary) but I get an error that the value should be string or list. What was your class weight type?

Thank you again @acivit