Bugfix radon transform

[Dugnom]

This PR solves #3846, see description of the problems and proposed fixes there.

Pull Request Readiness Checklist

• [x] I agree to contribute to the project under Apache 2 License.
• [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
• [x] The PR is proposed to the proper branch
• [x] There is a reference to the original bug report and related work
• [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name.
• [ ] The feature is well documented and sample code can be built with the project CMake