MoViNet-stream-classification

[Nithin-Supani]

Prerequisites

Please answer the following question for yourself before submitting an issue.

• [x] I checked to make sure that this issue has not been filed already.
• [x] I checked to make sure that this issue has not answered properly to address the issue.

1. The entire URL of the documentation with the issue

https://tfhub.dev/tensorflow/movinet/a2/stream/kinetics-600/classification/3

2. Describe the issue

In the above tensorflow-hub page at the end it's written "This model can be used with the hub.KerasLayer and wrapped in a tf.keras.Model as shown in the example. Calling model.fit() will enable fine-tuning using the pretrained weights."

But my question is how can the model be fin-tunned. Because the last layer of the model gives outputs, states.

I have understood that the outputs are set of probabilities of all the classes. The states should be fed to the model input along with the image and labels.

What kind of functional layer should i write to feed the output form one layer to the models first layer as on of its input?

Here is my colab link: https://colab.research.google.com/drive/1Dnhu_gYstEu3IPzZgNlptuLJutnaK9EY?usp=share_link.

Please help.

Thank you

[fcientymeh]

Create and train base model, next create stream model (with proper params like enabling output states..) and assign weights from trained base model to stream model.

[Nithin-Supani]

@fcientymeh It dosen't work. I followed your method form this link: https://github.com/tensorflow/models/issues/10730#issuecomment-1383180908

The architectures of both the models are different. I got an error when i load the weights to the stream model form base model.

I have attached the screen shot.

[fcientymeh]

You cannot simply save saved model, next load saved model, and read weights from it. You can use weights from trained model (yours: model from trainded a few seconds ago- at this time you can) or save weighs after training (creating checkpoing). In your code you should do: new_model.set_weights(model.set_weights()) and not try to load weights from saved model previously saved on disk. If you want get weights later, after training model, you should save and load weights using proper functions (look at: https://www.tensorflow.org/guide/keras/save_and_serialize?hl=pl).

[fcientymeh]

Look at my beautifull function wrapping everything ;-)

def AIPCreateBackboneAndClassifierModel(model_id, num_classes, frames_number, batch_size, resolution, 
                                        train_whole_model, dropout,
                                        conv_type: str = '3d', se_type: str = '3d', activation: str = 'swish',
                                        gating_activation: str = 'sigmoid', stream_mode=False, input_specs=None, load_pretrained_weights=True, training = True):
  '''
  Create video analysis model
  Return: movinet model
  //Andrzej Mąka, Aiseclab Sp. z o.o /POLAND //
  '''
  tf.keras.backend.clear_session()
  backbone = movinet.Movinet(model_id=model_id,
                            causal=stream_mode,
                            conv_type=conv_type,
                            se_type=se_type,
                            #input_specs=input_specs,
                            activation=activation,
                            gating_activation=gating_activation,
                            #use_sync_bn=True,
                            #use_positional_encoding=True,
                            use_external_states=stream_mode
                            )

  backbone.trainable = train_whole_model
  
  model = None
  if load_pretrained_weights:
    model = movinet_model.MovinetClassifier(backbone=backbone, num_classes=600)
    model.build([None, None, None, None, 3])
    checkpoint_dir = PRETRAINED_WEIGHTS[model_id] # some hashmap with pair: model_id and local checkpoint dir
    checkpoint_path = tf.train.latest_checkpoint(checkpoint_dir)
    checkpoint = tf.train.Checkpoint(model=model)
    status = checkpoint.restore(checkpoint_path)
    status.assert_existing_objects_matched()  
 
  model = movinet_model.MovinetClassifier(
        backbone=backbone, 
        #input_specs=input_specs,
        activation=activation,
        num_classes=num_classes, 
        output_states=stream_mode,
        dropout_rate = dropout
        )
  
  model.build([batch_size, frames_number, resolution, resolution, 3])
  return model

Create base model using param stream_mode=False. Train model. Next create second stream model using param stream_mode=True. Assign weighs as I wrote before. Buy me a beer ;-)

[nileshspringml]

I tried above code provided by @fcientymeh and wanted to check it with conv_type='2plus1d', se_type='2d' but it's giving an error. Please find below error with screenshot.

[laxmareddyp]

Hi @Nithin-Supani,

Sorry for delay in response , we created the gist which is working fine. kindly please make sure that while training keep external_states=False and for inference make sure to enable external_states=True.Please let me know if it solves your problem.

Thanks.

[nileshspringml]

Hi @laxmareddyp

Thank you so much for your reply. We tried notebook which provided by you but it's not able to help us to solve our code issue which is related to 2d and still getting the error.

We want to keep conv_type='2plus1d', se_type='2plus3d' in below code. As we want to convert it into tflite and use it with 2d streaming input.

It will be really helpful for us if get any solution around this.

[fcientymeh]

This checkpoint, that you trying to load, is for conv se_type = 3d only. So - use 3d , or do not load this checkpoint beacause it is for 3d. I don't know - i remember that I tried to use 2dplus1 and load checkpoint and I have had similar error.

[nileshspringml]

Thank you @fcientymeh for this valuable information.

[laxmareddyp]

Hi @nileshspringml,

The models are trained using the following configurations and the weights which are listed in the project are by using these configurations. So, it's not possible to load weights using se_type='2d', instead you can try without loading any weights. If you wish to use pretrained weights set se_type='2plus3d' and load the pre trained weights and then convert to tflite.

Thanks.

[nileshspringml]

Thank you @laxmareddyp for this information. As of now we are not loading checkpoints and it's working fine now.

[laxmareddyp]

We are working on colab notebook for movinet streaming model for training and inference, most probably it will be published by next week.you can refer that for more in-depth insight's.

Thanks.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[Nithin-Supani]

Look at my beautifull function wrapping everything ;-)

def AIPCreateBackboneAndClassifierModel(model_id, num_classes, frames_number, batch_size, resolution, 
                                        train_whole_model, dropout,
                                        conv_type: str = '3d', se_type: str = '3d', activation: str = 'swish',
                                        gating_activation: str = 'sigmoid', stream_mode=False, input_specs=None, load_pretrained_weights=True, training = True):
  '''
  Create video analysis model
  Return: movinet model
  //Andrzej Mąka, Aiseclab Sp. z o.o /POLAND //
  '''
  tf.keras.backend.clear_session()
  backbone = movinet.Movinet(model_id=model_id,
                            causal=stream_mode,
                            conv_type=conv_type,
                            se_type=se_type,
                            #input_specs=input_specs,
                            activation=activation,
                            gating_activation=gating_activation,
                            #use_sync_bn=True,
                            #use_positional_encoding=True,
                            use_external_states=stream_mode
                            )

  backbone.trainable = train_whole_model
  
  model = None
  if load_pretrained_weights:
    model = movinet_model.MovinetClassifier(backbone=backbone, num_classes=600)
    model.build([None, None, None, None, 3])
    checkpoint_dir = PRETRAINED_WEIGHTS[model_id] # some hashmap with pair: model_id and local checkpoint dir
    checkpoint_path = tf.train.latest_checkpoint(checkpoint_dir)
    checkpoint = tf.train.Checkpoint(model=model)
    status = checkpoint.restore(checkpoint_path)
    status.assert_existing_objects_matched()  
 
  model = movinet_model.MovinetClassifier(
        backbone=backbone, 
        #input_specs=input_specs,
        activation=activation,
        num_classes=num_classes, 
        output_states=stream_mode,
        dropout_rate = dropout
        )
  
  model.build([batch_size, frames_number, resolution, resolution, 3])
  return model

Create base model using param stream_mode=False. Train model. Next create second stream model using param stream_mode=True. Assign weighs as I wrote before. Buy me a beer ;-)

Thank you this method worked for me.

[Nithin-Supani]

We are working on colab notebook for movinet streaming model for training and inference, most probably it will be published by next week.you can refer that for more in-depth insight's.

Thanks.

Thank you for the update about the upcoming collab notebook for the Movinet streaming model. I appreciate your efforts and look forward to exploring the notebook for a more in-depth understanding of training the stream model.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[laxmareddyp]

Hi @Nithin-Supani

Thank you for the update about the upcoming collab notebook for the Movinet streaming model. I appreciate your efforts and look forward to exploring the notebook for a more in-depth understanding of training the stream model.

Here is the colab notebook published recently in github models/official repo.

This tutorial trains a MoViNet with A0 configuration from the TensorFlow Model Garden package (tensorflow-models).

This tutorial demonstrates how to:

Use models from the TensorFlow Models package. Train/Fine-tune a pre-built MoViNet for Video Classification. Export the trained/tuned MoViNet-A0-Stream model

Thanks.

[abduldattijo]

Hello i keep getting this error when using the movinet ao stream model, I am using vs code

{ "name": "ValueError", "message": "Trying to load a model of incompatible/unknown type. 'C:\Users\abdul\AppData\Local\Temp\tfhub_modules\0b1707d9329e1a1c18f429bc07101281a3ed23ce' contains neither 'saved_model.pb' nor 'saved_model.pbtxt'.", "stack": "--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Untitled-1.ipynb Cell 39 line 1 7 # MoViNet encoder from Kaggle 9 hub_url = "https://www.kaggle.com/models/google/movinet/frameworks/TensorFlow2/variations/a0-stream-kinetics-600-classification/versions/2" ---> 10 encoder = hub.KerasLayer(hub_url, trainable=True) 12 # Initialize states 13 init_states_fn = encoder.resolved_object.signatures['init_states']

File c:\Users\abdul\anaconda3\Lib\site-packages\tensorflow_hub\keras_layer.py:157, in KerasLayer.init(self, handle, trainable, arguments, _sentinel, tags, signature, signature_outputs_as_dict, output_key, output_shape, load_options, **kwargs) 153 self._output_shape = data_structures.NoDependency( 154 _convert_nest_to_shapes(output_shape)) 156 self._load_options = load_options --> 157 self._func = load_module(handle, tags, self._load_options) 158 self._is_hub_module_v1 = getattr(self._func, "_is_hub_module_v1", False) 160 # Update with the defaults when using legacy TF1 Hub format.

File c:\Users\abdul\anaconda3\Lib\site-packages\tensorflow_hub\keras_layer.py:459, in load_module(handle, tags, load_options) 457 except ImportError: # Expected before TF2.4. 458 set_load_options = load_options --> 459 return module_v2.load(handle, tags=tags, options=set_load_options)

File c:\Users\abdul\anaconda3\Lib\site-packages\tensorflow_hub\module_v2.py:107, in load(handle, tags, options) 102 saved_model_pbtxt_path = os.path.join( 103 tf.compat.as_bytes(module_path), 104 tf.compat.as_bytes(tf.saved_model.SAVED_MODEL_FILENAME_PBTXT)) 105 if (not tf.io.gfile.exists(saved_model_path) and 106 not tf.io.gfile.exists(saved_model_pbtxt_path)): --> 107 raise ValueError("Trying to load a model of incompatible/unknown type. " 108 "'%s' contains neither '%s' nor '%s'." % 109 (module_path, tf.saved_model.SAVED_MODEL_FILENAME_PB, 110 tf.saved_model.SAVED_MODEL_FILENAME_PBTXT)) 112 if options: 113 if not hasattr(getattr(tf, "saved_model", None), "LoadOptions"):

ValueError: Trying to load a model of incompatible/unknown type. 'C:\Users\abdul\AppData\Local\Temp\tfhub_modules\0b1707d9329e1a1c18f429bc07101281a3ed23ce' contains neither 'saved_model.pb' nor 'saved_model.pbtxt'." }

[abduldattijo]

this is the code i am using on vs code https://colab.research.google.com/drive/1ulgZaoUSE40WuOzzYClex0PmzpRdQkcy