[BUG]-Network error while importing flava_model_for_classification from torchmultimodal

[sumitdhungana14]

Add Link

https://pytorch.org/tutorials/beginner/flava_finetuning_tutorial.html

Describe the bug

In the following code snippet:

from torchmultimodal.models.flava.model import flava_model_for_classification
model = flava_model_for_classification(num_classes=len(vocab))

I am getting the following error:

Failed to download https://download.pytorch.org/models/multimodal/flava/flava_for_pretraining_unified_text_encoder.pt

Here is the error trace:

SSLCertVerificationError                  Traceback (most recent call last)
File [/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py:1348](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py:1348), in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
   1347 try:
-> 1348     h.request(req.get_method(), req.selector, req.data, headers,
   1349               encode_chunked=req.has_header('Transfer-encoding'))
   1350 except OSError as err: # timeout error

File [/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1286](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1286), in HTTPConnection.request(self, method, url, body, headers, encode_chunked)
   1285 """Send a complete request to the server."""
-> 1286 self._send_request(method, url, body, headers, encode_chunked)

File [/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1332](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1332), in HTTPConnection._send_request(self, method, url, body, headers, encode_chunked)
   1331     body = _encode(body, 'body')
-> 1332 self.endheaders(body, encode_chunked=encode_chunked)

File [/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1281](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1281), in HTTPConnection.endheaders(self, message_body, encode_chunked)
   1280     raise CannotSendHeader()
-> 1281 self._send_output(message_body, encode_chunked=encode_chunked)

File [/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1041](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1041), in HTTPConnection._send_output(self, message_body, encode_chunked)
   1040 del self._buffer[:]
-> 1041 self.send(msg)
   1043 if message_body is not None:
   1044 
...
-> 1351         raise URLError(err)
   1352     r = h.getresponse()
   1353 except:

URLError:

Describe your environment

Platform: macOS Python version: 3.11.6 CUDA: no PyTorch version: 2.0.1

cc @kartikayk @ankitade

[ebsmothers]

Hi @sumitdhungana14 I've tried this on a few different envs and I'm unable to reproduce the error. Not an exact match to your env, but I was able to download on macOS with Python 3.11.5 and PyTorch 2.1.0, which should be close enough. Are you behind a proxy server or anything like that?

[sumitdhungana14]

I'm not aware of any proxy server, but I did try on different networks and I'm still getting the same error. I tried upgrading PyTorch to 2.1.0 too, still no luck.

[ebsmothers]

If you're on macOS can you try something like this to download the pretrained weights to your local?

curl https://download.pytorch.org/models/multimodal/flava/flava_for_pretraining_unified_text_encoder.pt -o flava_weights.pt

[sumitdhungana14]

I downloaded the weights, and have the following code:

from torchmultimodal.models.flava.model import flava_model_for_classification
model = flava_model_for_classification(num_classes=len(vocab), pretrained=False)

weights_path = "./flava_weights.pt"
state_dict = torch.load(weights_path, map_location=torch.device('cpu'))


# Add a prefix to each key in the state_dict
prefix = "model."
prefixed_state_dict = {prefix + key: value for key, value in state_dict.items()}

model.load_state_dict(prefixed_state_dict)

The last line is causing the following error:

RuntimeError: Error(s) in loading state_dict for FLAVAForClassification:
	Missing key(s) in state_dict: "classifier.model.0.weight", "classifier.model.0.bias", "classifier.model.3.weight", "classifier.model.3.bias". 

This mightn't be a PyTorch issue but any direction would be helpful.

[ebsmothers]

@sumitdhungana14 I gave the wrong URL for download, sorry about that! The one I shared was just the FLAVA model class but you want the one that includes the classifier head. I updated my previous comment to give the correct path, pasting it here as well:

curl https://download.pytorch.org/models/multimodal/flava/flava_for_pretraining_unified_text_encoder.pt -o flava_weights.pt

[sumitdhungana14]

@ebsmothers I'm still getting the missing keys and unexpected keys error.

	Missing key(s) in state_dict: "classifier.model.0.weight", "classifier.model.0.bias", "classifier.model.3.weight", "classifier.model.3.bias". 

The list of unexpected keys is pretty long.

[ebsmothers]

@sumitdhungana14 it's expected that not all the keys will match, this is why we load the checkpoint with strict=False here. Normally this is handled on our end to make sure you're loading the proper keys, but given you had to load the weights manually it's worth doing a sanity check to ensure that the mismatched keys are the same ones you get when you load directly from TorchMultimodal. So you can add strict=False to your load_state_dict call and confirm that the mismatched keys are the same set as in https://gist.github.com/ebsmothers/e7c616cb8d2fb5aea4186ae32a1fb22d (this is the output printed when loading directly from TorchMultimodal).

[sumitdhungana14]

Yes, I get the same warning message on adding strict=False to load_state_dict. Does this mean the weights have been successfully loaded? On running the toy loop I get the following log:

Loss at step 0 = 8.350113868713379
Loss at step 1 = 8.267955780029297
Loss at step 2 = 8.209989547729492
Loss at step 3 = 8.242615699768066
Loss at step 4 = 8.45287799835205
Loss at step 5 = 8.243292808532715
Loss at step 6 = 8.035395622253418
Loss at step 7 = 8.378912925720215
Loss at step 8 = 8.532682418823242
Loss at step 9 = 7.696856498718262
Loss at step 10 = 7.680078029632568
Loss at step 11 = 7.2484660148620605
Loss at step 12 = 6.959850311279297
Loss at step 13 = 6.572226047515869
Loss at step 14 = 8.489222526550293
Loss at step 15 = 5.906797409057617
Loss at step 16 = 8.543415069580078
Loss at step 17 = 8.443180084228516
Loss at step 18 = 8.175264358520508
Loss at step 19 = 8.317131042480469
Loss at step 20 = 4.268733024597168
Loss at step 21 = 8.244979858398438
Loss at step 22 = 3.961406707763672
Loss at step 23 = 3.5651602745056152
Loss at step 24 = 8.665459632873535
...
Loss at step 46 = 8.4642333984375
Loss at step 47 = 5.553901672363281
Loss at step 48 = 5.022511005401611
Loss at step 49 = 3.2713959217071533

[ebsmothers]

@sumitdhungana14 this generally looks reasonable. Note that you should not expect strictly deterministic results as the remaining parameters not loaded (e.g. the classifier weights) will be randomly initialized. But you can plot the loss curve over many iterations and confirm if it is decreasing to check that the model is training properly.