Qualcomm AI Engine Direct - GA Static Granite3.3-2b

[chenweng-quic]

Summary

Add Granite3.3-2b support.

Source model:

Static llama: python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -H mlgtw-linux -s c3b39f15 -m SM8650 --temperature 0 --model_mode kv --max_seq_len 1024 --prefill_ar_len 128 --decoder_model granite_3_3-2b_instruct --prompt "I would like to learn python, could you teach me with a simple example?" --run_lm_eval --task hellaswag --limit 10 --artifact llama_qnn --kv_updater shift_pointer

Accuracy(hellaswag)(limit=10)

prepare_pt2e: {'acc_norm,none': 0.5} convert_pt2e: {'acc_norm,none': 0.3} device: {'acc_norm,none': 0.2}

Statistics on SM8650(16a4w_block64)

Statistics on SM8750(16a4w_block64)

Test plan

python backends/qualcomm/tests/test_qnn_delegate.py -k TestExampleLLMScript.test_granite_3_3_2b_instruct --device c3b39f15 --host mlgtw-linux --model SM8650 --build_folder build-android --executorch_root . --artifact_dir ./llama_qnn --llama_artifacts llama_qnn

cc @cccclai @cbilgin

[pytorch-bot[bot]]

:link: Helpful Links

:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15808

• :page_facing_up: Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

:x: 1 New Failure

As of commit 33576ac73516d2ae597fb230bbbd3b96949bd5ea with merge base 3e90b446f70f5f40435034f272ebf6b56dfb93ef ():

NEW FAILURE - The following job has failed:

• pull / test-moshi-linux / linux-job (gh) RuntimeError: Could not load libtorchcodec. Likely causes:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[pytorch-bot[bot]]

The label module: qnn is only applicable to issues and has been removed. Please only use this label on issues.

[github-actions[bot]]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example @pytorchbot label "release notes: none"

For more information, see https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[meta-codesync[bot]]

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D87034494.