Exporter failing due to output shape

[Janus289]

When trying to export the Huggingface models Deeppavlov/rubert-base-cased and ckiplab/bert-base-chinese-ner using the command line, it fails with the output

Some weights of the model checkpoint at Deeppavlov/rubert-base-cased were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.predictions.decoder.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Using framework PyTorch: 1.12.1
Overriding 1 configuration item(s)
        - use_cache -> False
Skipping token_type_ids input
Tuple detected at graph output. This will be flattened in the converted model.
Converting PyTorch Frontend ==> MIL Ops:   0%|                                                                                                           | 0/630 [00:00<?, ? ops/s]CoreML embedding (gather) layer does not support any inputs besides the weights and indices. Those given will be ignored.
Converting PyTorch Frontend ==> MIL Ops: 100%|██████████████████████████████████████████████████████████████████████████████████████████████▋| 628/630 [00:00<00:00, 4660.63 ops/s]
Running MIL Common passes:   0%|                                                                                                                       | 0/39 [00:00<?, ? passes/s]/Users/starlight/NERConversion/.venv/lib/python3.9/site-packages/coremltools/converters/mil/mil/passes/name_sanitization_utils.py:135: UserWarning: Output, '1020', of the source model, has been renamed to 'var_1020' in the Core ML model.
  warnings.warn(msg.format(var.name, new_name))
Running MIL Common passes: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [00:00<00:00, 47.87 passes/s]
Running MIL Clean up passes: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 31.91 passes/s]
/Users/starlight/NERConversion/.venv/lib/python3.9/site-packages/coremltools/models/model.py:145: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error:  Encountered an error while compiling a neural network model: validator error: Model output 'pooler_output' has a different shape than its corresponding return value to main.".
  _warnings.warn(
Validating Core ML model...
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/starlight/exporters/src/exporters/coreml/__main__.py", line 166, in <module>
    main()
  File "/Users/starlight/exporters/src/exporters/coreml/__main__.py", line 154, in main
    convert_model(
  File "/Users/starlight/exporters/src/exporters/coreml/__main__.py", line 65, in convert_model
    validate_model_outputs(coreml_config, preprocessor, model, mlmodel, args.atol)
  File "/Users/starlight/exporters/src/exporters/coreml/validate.py", line 108, in validate_model_outputs
    coreml_outputs = mlmodel.predict(coreml_inputs)
  File "/Users/starlight/NERConversion/.venv/lib/python3.9/site-packages/coremltools/models/model.py", line 545, in predict
    raise self._framework_error
  File "/Users/starlight/NERConversion/.venv/lib/python3.9/site-packages/coremltools/models/model.py", line 143, in _get_proxy_and_spec
    return (_MLModelProxy(filename, compute_units.name), specification, None)
RuntimeError: Error compiling model: "compiler error:  Encountered an error while compiling a neural network model: validator error: Model output 'pooler_output' has a different shape than its corresponding return value to main.".
Exception ignored in: <function MLModel.__del__ at 0x11ebe1ee0>
Traceback (most recent call last):
  File "/Users/starlight/NERConversion/.venv/lib/python3.9/site-packages/coremltools/models/model.py", line 369, in __del__
ImportError: sys.meta_path is None, Python is likely shutting down

It runs correctly with --model=distillbert-base-uncased. Using python 3.9.13, coremltools 6.1 torch 1.12.1

A .mlpackage file is created, but I can't use one I can't call predict() on.

[hollance]

Thanks for the report. Apparently the pooler_output output has an incorrect shape in the converted model. This is likely a bug in the exporter. As a workaround, you can try removing this output from the model.

Something like this (not tested so may have bugs):

del mlmodel._spec.description.output[-1]
mlmodel.save(...)

[Janus289]

Hi,

I tried that workaround and am now getting the error RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Encountered an error while compiling a neural network model: validator error: Model and main function must have same number of outputs.

[hollance]

Ah, too bad. One thing that might work is to create your own CoreMLConfig object and override the function that creates the output definitions to remove the pooler_output, and then use that config to do the conversion.