Koan-Sin Tan comments

Results 251 comments of


                                            Koan-Sin Tan

"Accuracy" metric for LLM model(s)

MMLU (0-shot, 5-shot): - @freedomtan try test 0-shot, 1-shot and find out the context and sequence length.

"Accuracy" metric for LLM model(s)

running MMLU with `lm_eval` | num_fewshot| context length| |--|--| |0| < 1024 | |1| could > 1024| |5| could > 3072| with something like ``` python -m executorch.examples.models.llama.eval_llama -c "${LLAMA_CHECKPOINT:?}"...

"Accuracy" metric for LLM model(s)

> evaluated with `lm_eval --model hf --model_args pretrained=meta-llama/... --tasks mmlu --num_fewshot 5` on Colab (w/ L4 GPU) > > model MMLU (5-shot) > 3.2 1B Instruct 0.4557 ± 0.0041 >...

Check if we can leverage Google Play's "Play for On-device AI"

[NNAPI is now considered to be formally deprecated](https://developer.android.com/ndk/guides/neuralnetworks). @dylanzika-google: will Google update the Pixel backend, which uses NNAPI.

Check if we can leverage Google Play's "Play for On-device AI"

On Pixel 10 Pro XL, NNAPI isn't working well.

Check if we can leverage Google Play's "Play for On-device AI"

@swasson488 @petermattson FYI: Pixel 10 numbers are clearly worse than Pixel 9 numbers. We need Google colleagues to fix it.

Check if we can leverage Google Play's "Play for On-device AI"

Why Pixel 10 numbers are not good. 1. for Mobilenet V4 and EDSR, fully-delegated, but clearly slower 2. for MobileDet, more ops not delegated (on Pixel 9, only the post...

Check if we can leverage Google Play's "Play for On-device AI"

with TFLite CLI benchmark_model: `benchmark_model --graph=${MODEL_PATH} --use_nnapi=1 --nnapi_allow_fp16=1 --enable_op_profiling=1`

LLM IFEval Dataset Implementation

for performance (and quick): tinymmlu (or ifeval maybe) for accuracy: tinymmly and ifeval

LLM Dataset Implementation

> [@freedomtan](https://github.com/freedomtan) I found a way to use different dataset files per `normal`, `quick`, and `rapid` runs, but they all have to use the same dataset (AKA `mlperf::QuerySampleLibrary`). Please let...