Memory use for MobileNetV2 UINT8 2-3x larger than TFlite

[mariecwhite]

For MobileNetV2 UINT8, peak resident set size on IREE is 2-3x larger on both mobile and server CPU compared with TFLite. For all other models under test, IREE memory use is usually 1-2x smaller than TFLite.

[hanhanW]

What compilation flags do you use?

[ScottTodd]

In addition, please fill in the "version information" field when filing issues like this. Code size, performance, memory use, and program compatibility can all change between commits/versions, and issues are much more difficult to triage and resolve without knowing what version of the software was used when the issue was noticed. If we're missing an easy way to get version information with whatever tools you are using, we should also be fixing that.

[mariecwhite]

Thanks Scott. Git commit hash is 223d02f16.

Compiler flags for mobile:

"${IREE_COMPILE_PATH}" \
    --iree-input-type=tosa \
    --iree-hal-target-backends=llvm-cpu \
    --iree-llvm-target-triple=aarch64-none-linux-android29 \
    --iree-llvm-debug-symbols=false \
    --iree-vm-bytecode-module-strip-source-map=true \
    --iree-vm-emit-polyglot-zip=false \
    "${IREE_MODEL_DIR}/${MODEL_NAME}.mlir" \
    --o "${IREE_MODEL_DIR}/llvm-cpu/${MODEL_NAME}.vmfb" || true

  echo -e "\tCompiling ${MODEL_NAME}_mmt4d.vmfb for aarch64..."
  "${IREE_COMPILE_PATH}" \
    --iree-input-type=tosa \
    --iree-hal-target-backends=llvm-cpu \
    --iree-llvm-target-triple=aarch64-none-linux-android29 \
    "--iree-flow-mmt4d-target-options=arch=aarch64 features=+dotprod" \
    --iree-llvm-target-cpu-features=+dotprod \
    --iree-llvm-debug-symbols=false \
    --iree-vm-bytecode-module-strip-source-map=true \
    --iree-vm-emit-polyglot-zip=false \
    "${IREE_MODEL_DIR}/${MODEL_NAME}.mlir" \
    --o "${IREE_MODEL_DIR}/llvm-cpu/${MODEL_NAME}_mmt4d.vmfb" || true

  echo -e "\tCompiling ${MODEL_NAME}_im2col_mmt4d.vmfb for aarch64..."
  "${IREE_COMPILE_PATH}" \
    --iree-input-type=tosa \
    --iree-hal-target-backends=llvm-cpu \
    --iree-llvm-target-triple=aarch64-none-linux-android29 \
    "--iree-flow-mmt4d-target-options=arch=aarch64 features=+dotprod" \
    --iree-llvm-target-cpu-features=+dotprod \
    --iree-flow-enable-conv-img2col-transform \
    --iree-llvm-debug-symbols=false \
    --iree-vm-bytecode-module-strip-source-map=true \
    --iree-vm-emit-polyglot-zip=false \
    "${IREE_MODEL_DIR}/${MODEL_NAME}.mlir" \
    --o "${IREE_MODEL_DIR}/llvm-cpu/${MODEL_NAME}_im2col_mmt4d.vmfb" || true

Flags for desktop:

echo "Compiling ${MODEL_NAME}.vmfb for llvm-cpu..."
  "${IREE_COMPILE_PATH}" \
    --iree-input-type=tosa \
    --iree-llvm-target-cpu-features=host \
    --iree-hal-target-backends=llvm-cpu \
    --iree-llvm-debug-symbols=false \
    --iree-vm-bytecode-module-strip-source-map=true \
    --iree-vm-emit-polyglot-zip=false \
    "${IREE_MODEL_DIR}/${MODEL_NAME}.mlir" \
    --o "${IREE_MODEL_DIR}/llvm-cpu/${MODEL_NAME}.vmfb"
 done