[Example] Yolo12 Detection sample with OpenVINO/XNNPACK backend

[daniil-lyakhov]

Summary

Ultralytics Yolo12 Detection sample to expand example models with the detection task:

1. Export script to get executorch yolo model lowered to OpenVINO/XNNPACK
2. Demo app which gets a model and a video as an input and outputs an annotated video

Changes in the OpenVINO backend:

• OpenVINO repo branch is updated to support yolo12 models (Commit to Openvino master WIP)
• Minor fixes in quantizer.py

Test plan

Warning:

OpenCV should be installed in the environment.

OpenVINO:

./.ci/scripts/test_yolo12.sh  -model yolo12n -mode openvino -pt2e_quantize OFF -upload tmp_ov_run -video_path  path/to/mp4/file

XNNPACK:

./.ci/scripts/test_yolo12.sh  -model yolo12n -mode xnnpack -pt2e_quantize OFF -upload tmp_xnnpack_run -video_path  path/to/mp4/file

Issues:

• Quantization does not work in both backends, issues WIP. Marked by NonImplemented error by now
• OpenVINO is being build twice because it is not present in the executorch-config.cmake file. Perhaps we could add the OpenVINO backend to the config file?

CC: @ynimmaga @suryasidd @alexsu52

[pytorch-bot[bot]]

:link: Helpful Links

:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10156

• :page_facing_up: Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

:x: 1 New Failure

As of commit 9e1d95f187e94b507c6859b81f02da01ee71bd4f with merge base f0a7d10b33fff712a0eb7307de4c830e64e5e845 ():

NEW FAILURE - The following job has failed:

• pull / unittest-editable / linux / linux-job (gh) backends/xnnpack/test/ops/test_gelu.py::TestGelu::test_fp16_gelu

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[GregoryComer]

FYI On the mlock error, you can try switching the load mode in the Module constructor (or data loader) to Mmap (from Mlock). IMO that should be the default.

[daniil-lyakhov]

FYI On the mlock error, you can try switching the load mode in the Module constructor (or data loader) to Mmap (from Mlock). IMO that should be the default.

This helped, thanks!

[daniil-lyakhov]

@lucylq, @jackzhxng, could you please take a look?

[kimishpatel]

@guangy10 to see if this can be done also via optimum-et

[guangy10]

@guangy10 to see if this can be done also via optimum-et

HF Transformers seem to have a different one yolos https://github.com/huggingface/transformers/tree/main/src/transformers/models/yolos. Should we try that in optimum-et?

[daniil-lyakhov]

Left some comments. Didnt review in details since this is largely in examples folder. At high level can you include some performance numbers in the summary

Please take a look https://github.com/pytorch/executorch/pull/10156/files#diff-a2182fd228e1b70702adceb8eacfcb42462b65dab17a66113a43ba511ee6988aR15-R22

[kimishpatel]

WIll take a look. Please do ping in case this slips through. I apologize that this has been waiting for so long

[daniil-lyakhov]

lets wait till the fix is landed so that we dont have to patch to your personal repo + branch

The fix is finally merged to the openvino master! The README and the test script are updated accordingly

[kimishpatel]

lets wait till the fix is landed so that we dont have to patch to your personal repo + branch

The fix is finally merged to the openvino master! The README and the test script are updated accordingly

Thank you. Accepted. Will wait for rest of the CI and merge

[kimishpatel]

Just realized, i didnt approve the workflows. Just did

[daniil-lyakhov]

Linters and some other jobs were failed most likely due to the old base commit. Rebased the PR and fixed the linter

[kimishpatel]

kicked off the run

[kimishpatel]

some tests are failing. not sure if they are realted

[daniil-lyakhov]

some tests are failing. not sure if they are realted

Looks like some sporicidal problems with the XNNPACKQuantizer

FAILED backends/xnnpack/test/ops/test_exp.py::TestExp::test_fp16_exp - AssertionError: Output 0 does not match reference output.
FAILED backends/xnnpack/test/ops/test_conv1d.py::TestConv1d::test_qs8_conv1d_batchnorm_seq - AssertionError: Output 0 does not match reference output.

and a problem with access to a json file for the llama tests on a mac os

Starting to run llama runner at 23:25:16
+ cmake-out/examples/models/llama/llama_main --model_path=tinyllama_coreml_fp32.pte --tokenizer_path=tokenizer.bin --prompt=Once --temperature=0 --seq_len=10 --warmup=1
I tokenizers:regex.cpp:27] Registering override fallback regex
Error parsing json file: [json.exception.parse_error.101] parse error at line 1, column 1: attempting to parse an empty input; check that your input string or stream contains the expected JSON
invalid charcouldn't load tokenizer.bin. 
Error message: 
could not parse ModelProto from tokenizer.bin
It is likely that the tokenizer artifact is broken or of a different format.++ date +%H:%M:%S

Most likely this commits will fix the problems https://github.com/pytorch/executorch/pull/12096 https://github.com/pytorch/executorch/pull/12089. Merging the main branch to my PR

[kimishpatel]

Unrelated failures. merging