optimum
optimum copied to clipboard
Published
20 hours ago β’
huggingface
huggingface
Support BLIP ONNX export & runtime
Feature request
I want to export my model to onnxοΌbut an error was happend, like, xxx is not support. such as, if I want export a blip model "Salesforce/blip-image-captioning-large" from the huggingface, it can't export it yet.
Motivation
export model to onnx
Your contribution
submitting a PR
I found it from here: https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model#customize-the-export-of-official-transformers-models
Thank you, BLIP is in transformers so we could support the ONNX export though. Feel free to submit a PR if you define a config that works!
@fxmarty , config as bellow
from optimum.exporters.onnx import export
from transformers.models.blip import BlipForConditionalGeneration
from optimum.exporters.onnx.model_configs import NormalizedTextAndVisionConfig, TextAndVisionOnnxConfig
from typing import Dict
from pathlib import Path
class BLIPNormalizedConfig(NormalizedTextAndVisionConfig):
TEXT_CONFIG = "text_config"
VISION_CONFIG = "vision_config"
# config
class BLIPOnnxConfig(TextAndVisionOnnxConfig):
NORMALIZED_CONFIG_CLASS = BLIPNormalizedConfig
@property
def inputs(self) -> Dict[str, Dict[int, str]]:
return {
"pixel_values": {0: "image_batch_size", 1: "num_channels", 2: "height", 3: "width"},
"input_ids": {0: "text_batch_size", 1: "sequence_length"},
"attention_mask": {0: "text_batch_size", 1: "sequence_length"},
}
# export to ONNX
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
# print("config:",model.config)
export(
model=model,
config=BLIPOnnxConfig(config=model.config, task="image-to-text"),
output=Path("onnx_model/blip.onnx"),
device="cuda",
input_shapes={
"width":384,
"height":384
}
)
an error happend: graph.outputs() is [], stack is flow, can you help me?
/blip/transformers2onnx.py:32 in <module> β
β β
β 29 # export to ONNX β
β 30 model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-i β
β 31 # print("config:",model.config) β
β β± 32 export( β
β 33 β model=model, β
β 34 β config=BLIPOnnxConfig(config=model.config, task="image-to-text"), β
β 35 β output=Path("onnx_model/blip.onnx"), β
β β
β /lib/python3.8/site-packages/optimum/exporters/onn β
β x/convert.py:855 in export β
β β
β 852 β β elif dtype is not None: β
β 853 β β β raise ValueError("Unsupported dtype, supported dtypes are: β
β 854 β β β
β β± 855 β β export_output = export_pytorch( β
β 856 β β β model, β
β 857 β β β config, β
β 858 β β β opset, β
β β
β /lib/python3.8/site-packages/optimum/exporters/onn β
β x/convert.py:575 in export_pytorch β
β β
β 572 β β β with config.patch_model_for_export(model, model_kwargs=mod β
β 573 β β β β # Export can work with named args but the dict contain β
β 574 β β β β # tuple. β
β β± 575 β β β β onnx_export( β
β 576 β β β β β model, β
β 577 β β β β β (dummy_inputs,), β
β 578 β β β β β f=output.as_posix(), β
β β
β /lib/python3.8/site-packages/torch/onnx/utils.py:5 β
β 04 in export β
β β
β 501 β β β All errors are subclasses of :class:`errors.OnnxExporterE β
β 502 β """ β
β 503 β β
β β± 504 β _export( β
β 505 β β model, β
β 506 β β args, β
β 507 β β f, β
β β
β /lib/python3.8/site-packages/torch/onnx/utils.py:1 β
β 529 in _export β
β β
β 1526 β β β β dynamic_axes = {} β
β 1527 β β β _validate_dynamic_axes(dynamic_axes, model, input_names, β
β 1528 β β β β
β β± 1529 β β β graph, params_dict, torch_out = _model_to_graph( β
β 1530 β β β β model, β
β 1531 β β β β args, β
β 1532 β β β β verbose, β
β β
β /lib/python3.8/site-packages/torch/onnx/utils.py:1 β
β 161 in _model_to_graph β
β β
β 1158 β β β β is_script, β
β 1159 β β β ) β
β 1160 β β
β β± 1161 β _set_input_and_output_names(graph, input_names, output_names) β
β 1162 β params_dict = _get_named_param_dict(graph, params) β
β 1163 β β
β 1164 β if training is None or training == _C_onnx.TrainingMode.EVAL: β
β β
β /lib/python3.8/site-packages/torch/onnx/utils.py:1 β
β 705 in _set_input_and_output_names β
β β
β 1702 β β β if node.debugName() != name: β
β 1703 β β β β node.setDebugName(name) β
β 1704 β set_names(list(graph.inputs()), input_names, "input") β
β β± 1705 β set_names(list(graph.outputs()), output_names, "output") β
β 1706 β
β 1707 β
β 1708 @_beartype.beartype β
β β
β /lib/python3.8/site-packages/torch/onnx/utils.py:1 β
β 683 in set_names β
β β
β 1680 β β if name_list is None: β
β 1681 β β β return β
β 1682 β β if len(name_list) > len(node_list): β
β β± 1683 β β β raise RuntimeError( β
β 1684 β β β β "number of %s names provided (%d) exceeded number of β
β 1685 β β β β % (descriptor, len(name_list), descriptor, len(node_l β
β 1686 β β β )
@fxmarty I'd be interested in adding this to transformers.js - do you think it's worth revisiting?
@Arrkwen the config you provided works for me. Happy to review it in a PR.
