Adds CLIP to models exportable with ONNX

[unography]

This isn't currently working, getting an error while validating the model -

onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(float)) , expected: (tensor(int64))

Environment: Pytorch: 1.13.0.dev20220806 onnxruntime: 1.12.0

Would love some guidance here!

@ChainYo @patrickvonplaten

[chainyo]

Hi, @unography. Could you give us a more detailed traceback, please?

It's hard to say without the script command and the full traceback.

[unography]

Hi, @unography. Could you give us a more detailed traceback, please?

It's hard to say without the script command and the full traceback.

this is the full traceback -

(transformers) ➜  transformers git:(main) python -m transformers.onnx --model=openai/clip-vit-base-patch32 onnx/
vocab_file vocab.json
merges_file merges.txt
tokenizer_file tokenizer.json
added_tokens_file added_tokens.json
special_tokens_map_file special_tokens_map.json
tokenizer_config_file tokenizer_config.json
Using framework PyTorch: 1.13.0.dev20220806
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:222: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:262: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:680: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask.fill_(torch.tensor(torch.finfo(dtype).min))
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:230: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:239: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4592: UserWarning: Exporting aten::index operator of advanced indexing in opset 14 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
Validating ONNX model...
Traceback (most recent call last):
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 107, in <module>
    main()
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 100, in main
    validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/convert.py", line 405, in validate_model_outputs
    onnx_outputs = session.run(onnx_named_outputs, onnx_inputs)
  File "/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(float)) , expected: (tensor(int64))

[regisss]

@unography You need to pass the ONNX inputs in the same order as they are registered in the forward method of the model. We can see here that pixel_values comes before attention_mask, so in the ONNX config you must return:

OrderedDict(
  [
      ("input_ids", {0: "batch", 1: "sequence"}),
      ("pixel_values", {0: "batch"}),
      ("attention_mask", {0: "batch", 1: "sequence"}),
  ]
)

Note that it is an OrderedDict so the order matters :)

To explain a bit the error message, what was happening is that it expected the second input to be int64 since that is how you defined it in the ONNX config. But it actually got a float tensor because pixel_values is passed before attention_mask in the forward method.

[unography]

@unography You need to pass the ONNX inputs in the same order as they are registered in the forward method of the model. We can see here that pixel_values comes before attention_mask, so in the ONNX config you must return:

OrderedDict(
  [
      ("input_ids", {0: "batch", 1: "sequence"}),
      ("pixel_values", {0: "batch"}),
      ("attention_mask", {0: "batch", 1: "sequence"}),
  ]
)

Note that it is an OrderedDict so the order matters :)

To explain a bit the error message, what was happening is that it expected the second input to be int64 since that is how you defined it in the ONNX config. But it actually got a float tensor because pixel_values is passed before attention_mask in the forward method.

ah yes, understood. able to resolve this issue, getting an error on the output values now

(transformers) ➜  transformers git:(main) python -m transformers.onnx --model=openai/clip-vit-base-patch32 onnx/
vocab_file vocab.json
merges_file merges.txt
tokenizer_file tokenizer.json
added_tokens_file added_tokens.json
special_tokens_map_file special_tokens_map.json
tokenizer_config_file tokenizer_config.json
Using framework PyTorch: 1.13.0.dev20220806
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:222: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:262: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:680: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask.fill_(torch.tensor(torch.finfo(dtype).min))
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:230: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:239: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4592: UserWarning: Exporting aten::index operator of advanced indexing in opset 14 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
Validating ONNX model...
Traceback (most recent call last):
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 107, in <module>
    main()
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 100, in main
    validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/convert.py", line 405, in validate_model_outputs
    onnx_outputs = session.run(onnx_named_outputs, onnx_inputs)
  File "/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(float)) , expected: (tensor(int64))
(transformers) ➜  transformers git:(clip_onnx) ✗ xx
(transformers) ➜  transformers git:(clip_onnx) ✗ python -m transformers.onnx --model=openai/clip-vit-base-patch32 onnx/
vocab_file vocab.json
merges_file merges.txt
tokenizer_file tokenizer.json
added_tokens_file added_tokens.json
special_tokens_map_file special_tokens_map.json
tokenizer_config_file tokenizer_config.json
Using framework PyTorch: 1.13.0.dev20220806
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:222: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:262: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:680: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask.fill_(torch.tensor(torch.finfo(dtype).min))
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:230: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:239: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4592: UserWarning: Exporting aten::index operator of advanced indexing in opset 14 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
Validating ONNX model...
        -[x] ONNX model output names {'last_hidden_state'} do not match reference model {'text_embeds', 'logits_per_image', 'text_model_output', 'logits_per_text', 'image_embeds', 'vision_model_output'}
Traceback (most recent call last):
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 107, in <module>
    main()
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 100, in main
    validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/convert.py", line 414, in validate_model_outputs
    raise ValueError(
ValueError: Outputs doesn't match between reference model and ONNX exported model: {'last_hidden_state'}

I'm guessing in the onnx config I have to define a separate function for outputs as well?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[regisss]

@unography Yes, you have to define outputs the same way you did for inputs. Inputs are not defined in the parent class because they usually vary from one model to another, which is why it is mandatory to define them for each model. However, some default outputs are already defined depending on the tasks. For the default task, you can see here that it expects last_hiddent_state as the output, which is not returned by CLIP. So you can override this to specify the outputs you want.

Not sure though which outputs we would like to have here. Maybe text_embeds and image_embeds since this is basically feature extraction?

[unography]

@unography Yes, you have to define outputs the same way you did for inputs. Inputs are not defined in the parent class because they usually vary from one model to another, which is why it is mandatory to define them for each model. However, some default outputs are already defined depending on the tasks. For the default task, you can see here that it expects last_hiddent_state as the output, which is not returned by CLIP. So you can override this to specify the outputs you want.

Not sure though which outputs we would like to have here. Maybe text_embeds and image_embeds since this is basically feature extraction?

so if I define in the onnx config, say text_embeds and image_embeds, but the model is actually returning more outputs, like vision_model_output, will these extra outputs create any conflict or will it get handled by the onnxconfig automatically?

[regisss]

@unography Yes, you have to define outputs the same way you did for inputs. Inputs are not defined in the parent class because they usually vary from one model to another, which is why it is mandatory to define them for each model. However, some default outputs are already defined depending on the tasks. For the default task, you can see here that it expects last_hiddent_state as the output, which is not returned by CLIP. So you can override this to specify the outputs you want. Not sure though which outputs we would like to have here. Maybe text_embeds and image_embeds since this is basically feature extraction?

so if I define in the onnx config, say text_embeds and image_embeds, but the model is actually returning more outputs, like vision_model_output, will these extra outputs create any conflict or will it get handled by the onnxconfig automatically?

Not sure about this. I think it should work the same as inputs, i.e. the order will matter and the inputs that are not specified in the config will just be skipped.

[unography]

@regisss sure, i'll try it out, thank you so much for your help!

[unography]

how do I make the test cases pass? Is it only formatting issues or something else?

[unography]

@regisss I made the changes, apart from the code formatting. How do I format my code correctly? And do I need to run make fix-copies ?

[regisss]

@regisss I made the changes, apart from the code formatting. How do I format my code correctly? And do I need to run make fix-copies ?

I don't think make fix-copies is necessary anymore because you already updated the doc.

[regisss]

@unography Not sure why modeling_groupvit.py is still in the changes.

Also, can you make sure that the test pytest tests/onnx/test_onnx_v2.py -v -k "clip" pass?

[unography]

@regisss i reverted changes to groupvit, and when I'm running the test (on Colab) the tests are being skipped -

pytest tests/onnx/test_onnx_v2.py -v -k "clip"

============================= test session starts ==============================
platform linux -- Python 3.7.13, pytest-3.6.4, py-1.11.0, pluggy-0.7.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /content/transformers, inifile: setup.cfg
plugins: typeguard-2.7.1
collected 398 items / 396 deselected                                           

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default <- ../../usr/lib/python3.7/unittest/case.py SKIPPED [ 50%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default <- ../../usr/lib/python3.7/unittest/case.py SKIPPED [100%]

is there some issue with my changes that its skipping these tests?

[regisss]

@regisss i reverted changes to groupvit, and when I'm running the test (on Colab) the tests are being skipped -

pytest tests/onnx/test_onnx_v2.py -v -k "clip"

============================= test session starts ==============================
platform linux -- Python 3.7.13, pytest-3.6.4, py-1.11.0, pluggy-0.7.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /content/transformers, inifile: setup.cfg
plugins: typeguard-2.7.1
collected 398 items / 396 deselected                                           

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default <- ../../usr/lib/python3.7/unittest/case.py SKIPPED [ 50%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default <- ../../usr/lib/python3.7/unittest/case.py SKIPPED [100%]

is there some issue with my changes that its skipping these tests?

@unography My bad I forgot the environment variable in the command I gave you, sorry. Here it is:

RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -v -k "clip"

Some tests are skipped by default because they can take some time to complete, which is why we need to specify this env variable when running them.

[unography]

@regisss for some reason on google colab the tests are still being skipped, so I'm not able to test on GPU

this is on my local machine -

(transformers) ➜  transformers git:(clip_onnx) RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -v -k "clip"
=========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.8.12, pytest-7.1.2, pluggy-1.0.0 -- /Users/dhruv/Documents/code/transformers/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/dhruv/Documents/code/transformers, configfile: setup.cfg
collected 398 items / 396 deselected / 2 selected                                                                                                                                                         

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default PASSED                                                                                                       [ 50%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default PASSED                                                                                               [100%]

============================================================================================ warnings summary =============================================================================================
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default
  /Users/dhruv/Documents/code/transformers/src/transformers/image_utils.py:223: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
    def resize(self, image, size, resample=PIL.Image.BILINEAR, default_to_square=True, max_size=None):

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default
  /Users/dhruv/Documents/code/transformers/src/transformers/models/clip/feature_extraction_clip.py:67: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
    resample=Image.BICUBIC,

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default
  /Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:222: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default
  /Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:262: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default
  /Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:681: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
    mask.fill_(torch.tensor(torch.finfo(dtype).min))

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default
  /Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:230: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default
  /Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:239: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if attention_mask.size() != (bsz, 1, tgt_len, src_len):

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_029_clip_default
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_029_clip_default
  /Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4592: UserWarning: Exporting aten::index operator of advanced indexing in opset 14 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= 2 passed, 396 deselected, 14 warnings in 53.51s =============================================================================

[regisss]

@unography I'm going to checkout your branch and run the test on GPU.

[unography]

@regisss do let me know if there are any further changes needed!

[regisss]

@regisss do let me know if there are any further changes needed!

@unography I had to change two small things in modeling_clip.py to make the tests pass:

• replace similarity.T by similarity.t()
• replace logits_per_text.T by logits_per_text.t()

It seems ONNX does not like .T, I got the issue mentioned here. Have you encountered the same issue?

[unography]

@regisss do let me know if there are any further changes needed!

@unography I had to change two small things in modeling_clip.py to make the tests pass:

• replace similarity.T by similarity.t()
• replace logits_per_text.T by logits_per_text.t()

It seems ONNX does not like .T, I got the issue mentioned here. Have you encountered the same issue?

Oh I think this got fixed in pytorch's latest release. I'll verify this once, and test on older versions of Pytorch as well, and make the change and push

[unography]

@regisss .T is working for me while using pytorch's nightly release, but it fails on pytorch 1.12.1, the stable version. I've made the change to make it .t(), this is working in the nightly release version as well

[regisss]

Thanks @unography, it looks good to me!

Looking at the failed tests, it seems you need to run make fix-copies one more time. Could you do it please?

[unography]

@regisss ah, my mistake. pushed. there are now additional changes to owlvit, groupvit and vision_text_dual_encoder, I'm assuming we copy over code to these files from the actual CLIP model?

[regisss]

@regisss ah, my mistake. pushed. there are now additional changes to owlvit, groupvit and vision_text_dual_encoder, I'm assuming we copy over code to these files from the actual CLIP model?

Yes that is what happens. Actually you removed those changes because they were among all the formatting changes that you got the first time you ran black, and I told you to remove them, sorry. I did not pay attention to those.

It looks good to me @unography :)

[regisss]

Gently pinging @sgugger for approval

[unography]

@sgugger sure, removed the comment and pushed, but some tests are failing right now, I can't understand why.

[sgugger]

This is a flaky test, don't worry. Thanks again for your contribution!

[regisss]

Congrats @unography for this PR!

[unography]

Thanks @regisss for all the help!

[chainyo]

Huge contribution! That's awesome!

[NielsRogge]

Awesome work @unography - are you planning to add support for GroupViT and OWL-ViT as well?

[unography]

@NielsRogge sure, if they're open I'll add a draft PR for them and get started!