Invalid result of running U2Net on Vulkan

[MaksimDanilov]

🐛 Describe the bug

I try to get rid of multiple nn formats (CoreML and Tensorflow-Lite) for U2Net. Exported program for CoreML and XNNPack gives correct result, but If I partition with Vulkan it gives corrupted image after processing. Do I need to test Vulkan only on android devices? How can I understand what happened?

Versions

Build from source exectorch v1.0.0 on Apple MacBook Pro M1.

cc @SS-JIA @manuelcandales @digantdesai @cbilgin

[MaksimDanilov]

Vulkan

XNNPack

[MaksimDanilov]

Also run this model on Android (build from source executorch with Vulkan and XNNPack enabled and ship as static library to my library) and the same result. GPU seems not invoking on Pixel 9 Pro 'cause neither profiler gives stats on GPU, nor inference time (XNNPack vs Vulkan) doesn't change. On Mac Vulkan do run on GPU.

[SS-JIA]

@MaksimDanilov would you be able to provide an export script + model run instructions for your model? I will try to reproduce and figure out the issue!

[MaksimDanilov]

I will try to make projects to reproduce the problem to this issue :-)

[MaksimDanilov]

https://github.com/MaksimDanilov/executorch_vulkan_problem - done :-)

[MaksimDanilov]

@SS-JIA just in case comment above is not get lost

[SS-JIA]

@MaksimDanilov sorry, just seeing this - will take a look and keep you updated!

[SS-JIA]

@MaksimDanilov thanks for putting up the repo. I was able to reproduce the issue - however, strangely I found that the issue was not in the Vulkan backend.

In your app code, I found that if I just convert the input image to float and normalize it to the 0, 1 range, the model seems to work fine.

    cv::Mat image_orig = cv::imread(if_, cv::IMREAD_COLOR);
    if (image_orig.empty())
        return -1;

    cv::Mat image;
    image_orig.convertTo(image, CV_32F, 1.0 / 255.0); // Convert

For example, I see this result for the horse image:

I created a fork of your repo with my changes: https://github.com/SS-JIA/executorch_vulkan_problem

I added a section in the README with the full instructions for how to repro.

Please try it out and lmk if you are still having issues!

[MaksimDanilov]

@SS-JIA Hi. Yeap. I know that model should be converted to that range, however forgot about it. I'll come back later with reply. Thank you

[MaksimDanilov]

Surprisingly, with xnnpack and coreml this code works.