opencv_contrib icon indicating copy to clipboard operation
opencv_contrib copied to clipboard

Add additional image processing operators for Ascend NPU by utilizing DVPP

Open MengqingCao opened this issue 1 year ago • 10 comments

The user base for Ascend NPU and programming with CANN is increasing rapidly, with a growing number of users joining each day. To facilitate the use of these users, this PR provides more support for Ascend backend operators. All operators this PR offers are using use DVPP as the computational unit. Digital Vision Pre-Processing (DVPP) is an image processing unit built into the Ascend AI processor. Its main functions include image and video encoding/decoding, as well as image cropping and scaling.

The high-frequency operators with NPU as the backend and basic data structure AscendMat has been provided in #3552, while it still lacks many image processing operators. Moreover, only two interpolation algorithms for the resize operator are supported in #3552. In this PR, the bilinear interpolation algorithm and nearest neighbour interpolation algorithm are implemented for the resize operator, as well as the Ascend implementation of the copyMakeBorder operator.

In addition, the serialization of image processing operations is widely used in the preprocessing and post-processing stages of computer vision deep learning methods. Therefore, providing integrated operators is very meaningful for improving the convenience of use for OpenCV and deep learning crossover users. For example, torchvision also provides similar operators: RESIZED_CROP. Thus, this PR also provides two serialization processing operators: cropResize and cropResizeMakeBorder.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • [x] I agree to contribute to the project under Apache 2 License.
  • [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • [x] The PR is proposed to the proper branch
  • [N/A] There is a reference to the original bug report and related work
  • [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name.
  • [x] The feature is well documented and sample code can be built with the project CMake

MengqingCao avatar Dec 13 '23 09:12 MengqingCao

@opencv-alalek Happy New Year! This PR relies on CANN backends, could you trigger the test by labelling it? BTW, it may fail until the library it needs is linked. (Link libacl_dvpp_mpi library when building with CANN backend #24720)

The build on win64 is scheduled, and I have no idea with how to deal this.

MengqingCao avatar Jan 02 '24 07:01 MengqingCao

The build on win64 is scheduled, and I have no idea with how to deal this.

Windows builder worker was paused due to maintenance. Ignore.


I added "cann" label but GHA doesn't launch CANN build configuration due to some reason.

If you want to test both PRs/patches from opencv and opencv_contrib repositories then they must have the same "source" branch name (currently names are different). You need to push branch to opencv under the name dvpp_support.

opencv-alalek avatar Jan 02 '24 07:01 opencv-alalek

I added "cann" label but GHA doesn't launch CANN build configuration due to some reason.

Thx, are there any errors with the CI machine or bugs on GHA?

If you want to test both PRs/patches from opencv and opencv_contrib repositories then they must have the same "source" branch name (currently names are different). You need to push branch to opencv under the name dvpp_support.

Got it, I will resubmit a PR in opencv repo.

MengqingCao avatar Jan 02 '24 08:01 MengqingCao

Good Day! @opencv-alalek, Thanks for your help, now CANN's CI is working properly and this PR passed all the checks. Could you please help review this PR or assign someone else to review it? This PR is mainly based on #3552 with the new operators utilizing DVPP for Ascend NPU, providing more interpolation methods for resize, adding copyMakeBorder and 2 serialization operators. Looking forward to suggestions! Thanks!

MengqingCao avatar Jan 11 '24 07:01 MengqingCao

/cc @fengyuentau

opencv-alalek avatar Jan 11 '24 07:01 opencv-alalek

Thanks for the review! All the fixes, as well as the cherry-pick have been done now.

MengqingCao avatar Jan 19 '24 03:01 MengqingCao

@asmorkalov Good Day! Should the code in this PR be included in the branch?

MengqingCao avatar Jan 30 '24 07:01 MengqingCao

@asmorkalov Sorry to bother you, could you check if this project is now mergeable?

MengqingCao avatar Feb 20 '24 02:02 MengqingCao

Hello @MengqingCao , could you help resolve comments in https://github.com/opencv-infrastructure/opencv-gha-dockerfile/pull/28?

fengyuentau avatar Feb 20 '24 03:02 fengyuentau

Hello @MengqingCao , could you help resolve comments in opencv-infrastructure/opencv-gha-dockerfile#28?

sure, I'll check it soon.

MengqingCao avatar Feb 20 '24 04:02 MengqingCao

@asmorkalov Sorry to bother you again, could you please review whether this pull request should be included? thx 👍

MengqingCao avatar Feb 29 '24 08:02 MengqingCao

@fengyuentau @asmorkalov Good day! I noticed that the CI passed but was not updated in the build check on this page. Should the CI be retriggered or anything else to do with it?

image

MengqingCao avatar Mar 25 '24 06:03 MengqingCao