Error when trying to run vision model onnx

[elephantpanda]

Describe the issue

I am trying to run the phi-3-v-128k-instruct-vision.onnx model using the script:

        int height = 224;
        int width = 224;
        int max_num_crops = 1;
        int num_images = 1;
        var imageSizes = OrtValue.CreateTensorValueFromMemory<long>(new long[] { height, width }, new long[] { num_images, 2 });
        var pixelValues = OrtValue.CreateTensorValueFromMemory<Float16>(new Float16[3* max_num_crops* height * width ], new long[] { num_images, max_num_crops, 3,height,width });

        var inputs = new Dictionary<string, OrtValue>
        {
            {"pixel_values",pixelValues },
            {"image_sizes",imageSizes}
        };
        var runOptions = new RunOptions();
        var output = visionSession.Run(runOptions, inputs, new[] { "visual_features" });

and it's giving me the error:

OnnxRuntimeException: [ErrorCode:RuntimeException] Non-zero status code returned while running Add node. Name:'/img_processor/vision_model/embeddings/Add' Status Message: D:\a\_work\1\s\onnxruntime\core/providers/cpu/math/element_wise_ops.h:560 onnxruntime::BroadcastIterator::Append axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 577 by 1297

Microsoft.ML.OnnxRuntime.NativeApiStatus.VerifySuccess (System.IntPtr nativeStatus) (at D:/a/_work/1/s/csharp/src/Microsoft.ML.OnnxRuntime/NativeApiStatus.shared.cs:33)
Microsoft.ML.OnnxRuntime.InferenceSession.RunImpl (Microsoft.ML.OnnxRuntime.RunOptions options, System.IntPtr[] inputNames, System.IntPtr[] inputValues, System.IntPtr[] outputNames) (at D:/a/_work/1/s/csharp/src/Microsoft.ML.OnnxRuntime/InferenceSession.shared.cs:982)
Microsoft.ML.OnnxRuntime.InferenceSession.Run (Microsoft.ML.OnnxRuntime.RunOptions runOptions, System.Collections.Generic.IReadOnlyDictionary`2[TKey,TValue] inputs, System.Collections.Generic.IReadOnly

It fails for all execution providers CPU/DirectML

To reproduce

As above.

Urgency

No response

Platform

Windows

OS Version

10

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

Microsoft.ML.OnnxRuntime.Managed.1.20.0-dev-20240823-1101-4c4ae1e490

ONNX Runtime API

C#

Architecture

X64

Execution Provider

DirectML Microsoft.AI.DirectML.1.15.1

Execution Provider Library Version

No response

[elephantpanda]

Actually looking at the genai code it seems I need to set:

width=height=336 
max_num_crops=4

Not really sure what these numbers do bit it seems to work. If there's any documentation about this it would be welcome. However if I set max_num_crops>8 then I get the error:

OnnxRuntimeException: [ErrorCode:RuntimeException] Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_5' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2557)\onnxruntime.dll!00007FF84DAE4A75: (caller: 00007FF84E12918D) Exception(24) tid(6b8) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

Even though it is using hardly any of my VRAM. (P5000 GPU 32GB VRAM) This is the same error I get when trying to run the phi vision model with genai.

[github-actions[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[elephantpanda]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

Just bumping this because it is not fixed yet but the bot is trying to close it! 😬

[microsoft-github-policy-service[bot]]

Applying stale label due to no activity in 30 days

[microsoft-github-policy-service[bot]]

Applying stale label due to no activity in 30 days