onnxruntime
onnxruntime copied to clipboard
microsoft
[java] Adding ability to load a model from a memory mapped byte buffer
Description
Adds support for constructing an OrtSession from a java.nio.ByteBuffer. These buffers can be memory mapped from files which means there doesn't need to be copies of the model protobuf held in Java, reducing peak memory usage during session construction.
Motivation and Context
Reduces memory usage on model construction by not requiring as many copies on the Java side. Should help with #19599.
Hello, are there any updates on this PR? This would be a very valuable addition to the Java integration of ORT.
@yuslepukhin please could you review this after the 1.18 release? And #18998 which is also a small straightforward improvement.
/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline
/azp run Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline
/azp run Big Models,Linux Android Emulator QNN CI Pipeline,Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline
![azure-pipelines[bot] avatar](https://avatars.githubusercontent.com/in/9426?v=4 "Published by a pub.dev verified publisher"){kind=small}
@craigacp this might need a merge with main due to the pipeline definitions being updated. I can't run the 'Python format' CI with the current set of changes and it's required to be able to checkin.
@Craigacp this might need a merge with main due to the pipeline definitions being updated. I can't run the 'Python format' CI with the current set of changes and it's required to be able to checkin.
Ok, I've rebased the branch on main, should be good now.
/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline
/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline
/azp run Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline
/azp run Windows GPU CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline
/azp run Windows GPU CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-ortmodule-distributed,Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline
/azp run Windows GPU CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-ortmodule-distributed,Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline