[CodeCamp2023-305] Improve Visualizer Rendering Speed with OpenCV Backend

[KerwinKai]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

As https://github.com/open-mmlab/mmengine/issues/1101 described.

Modification

I have shown the design ideas in functions such as __init__, get_image, draw_lines, etc. The basic idea is to add a parameter called backend to the Visualizer to control whether the rendering backend used for drawing is cv2 or matplotlib. The design idea of the drawing function is basically the same as that of draw_lines, that is, the original parameters of the function can be input into the drawing function of cv2 as much as possible. If it is not possible, a prompt will be thrown. This is my first draft, which shows the overall implementation idea of this task. If my idea have any problems, please correct me in time. I will complete the remaining code, documents and test cases as soon as possible.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[KerwinKai]

Oki, I will update my code and docstring mentioned above as soon as posible.

[HAOCHENYE]

Please present the visualization results using both the cv2 and matplotlib backends to verify their correctness.

[KerwinKai]

Please present the visualization results using both the cv2 and matplotlib backends to verify their correctness.

Hi, here is my vis result:https://github.com/KerwinKai/cv2_backend_for_mmengine/tree/main/cv2_backend_result

[KerwinKai]

I want to ask a question about mypy check. Seeing the error report, it seems that there is still str in the output list, but the color_str2rgb function returns only one tuple, and I feel that str should not be obtained.

[KerwinKai]

I want to ask a question about mypy check. Seeing the error report, it seems that there is still str in the output list, but the color_str2rgb function returns only one tuple, and I feel that str should not be obtained.

fix it by adding # type:ignore

[zhouzaida]

Hi @KerwinKai , please resolve conflicts

[KerwinKai]

The tests failed in ci/circleci may not caused by the code I submit.

=========================== short test summary info ============================
FAILED tests/test_dataset/test_base_dataset.py::TestConcatDataset::test_getitem - assert tensor(False)
 +  where tensor(False) = <built-in method all of Tensor object at 0x7fe67ae26810>()
 +    where <built-in method all of Tensor object at 0x7fe67ae26810> = tensor([[[[0.2660, 0.9773, 0.0876,  ..., 0.8606, 0.3239, 0.8624],\n          [0.6684, 0.5893, 0.0743,  ..., 0.3921, 0.9...40, 0.4949, 0.4402,  ..., 0.4223, 0.8890, 0.5576],\n          [0.2576, 0.5780, 0.3324,  ..., 0.8408, 0.6389, 0.3608]]]]) != tensor([[[[0.0053, 0.7860, 0.3569,  ..., 0.9242, 0.8792, 0.5923],\n          [0.9889, 0.2432, 0.1560,  ..., 0.6497, 0.6...69, 0.2710, 0.3506,  ..., 0.2588, 0.6569, 0.0595],\n          [0.9705, 0.4193, 0.7484,  ..., 0.4690, 0.7984, 0.0703]]]]).all
===== 1 failed, 886 passed, 115 skipped, 236 warnings in 294.68s (0:04:54) =====

[HAOCHENYE]

Approved!!!

[KerwinKai]

Here is my new result, the self._image order rgb in matplotlib backend and bgr in cv2 backend:https://github.com/KerwinKai/cv2_backend_for_mmengine/tree/main/cv2_backend_result

Here is an example to draw and save point image in two backend:

image = np.zeros((500, 500, 3))
    backends = ['matplotlib', 'cv2']
    # test draw_points
    function = 'draw_points'
    for backend in backends:
        vis = Visualizer(image=image,
                         backend=backend)
        vis.draw_points(
            np.array([[100, 300], [200, 400]]),
            colors=['g', 'r']
        )
        result_img = vis.get_image()
        save_name = f'./images/{function}_{backend}.png'
        if backend == 'matplotlib':
            plt.imshow(result_img)
            plt.axis('off')
            plt.savefig(save_name, bbox_inches='tight', pad_inches = 0)
        else:
            cv2.imwrite(save_name, result_img)

See more details in: https://github.com/KerwinKai/cv2_backend_for_mmengine/blob/main/cv2_backend_result/generate.py

[zhouzaida]

Hi, how about standardizing the input format to be RGB for all interfaces? This would facilitate interface consistency. If necessary in the future, we can provide an additional parameter to specify the color space. Therefore, set_image does not need to perform color space conversion.

[KerwinKai]

Hi, how about standardizing the input format to be RGB for all interfaces? This would facilitate interface consistency. If necessary in the future, we can provide an additional parameter to specify the color space. Therefore, set_image does not need to perform color space conversion.

Does that mean that all functions such as draw_text input are sorted by rgb. For example：

vis = Visualizer(image=image,
                         backend='cv2')
vis.draw_points(
            np.array([[100, 300], [200, 400]]),
            colors=[(255, 0, 0)])

We will get a red line because colors inputs are all in RGB order.

[KerwinKai]

Got it, standard the input format to be RGB for all interfaces now and update relate docstring.

[zhouzaida]

Could you also update the images in https://github.com/KerwinKai/cv2_backend_for_mmengine/tree/main/cv2_backend_result/images using the latest commit?

[KerwinKai]

Could you also update the images in https://github.com/KerwinKai/cv2_backend_for_mmengine/tree/main/cv2_backend_result/images using the latest commit?

Oki, already update it.

[zhouzaida]

It seems like that https://github.com/KerwinKai/cv2_backend_for_mmengine/blob/main/cv2_backend_result/images/draw_binary_masks_cv2.png is different from https://github.com/KerwinKai/cv2_backend_for_mmengine/blob/main/cv2_backend_result/images/draw_binary_masks_matplotlib.png.

[KerwinKai]

It seems like that https://github.com/KerwinKai/cv2_backend_for_mmengine/blob/main/cv2_backend_result/images/draw_binary_masks_cv2.png is different from https://github.com/KerwinKai/cv2_backend_for_mmengine/blob/main/cv2_backend_result/images/draw_binary_masks_matplotlib.png.

I fix my code to orgin version in main mmengine and update https://github.com/KerwinKai/cv2_backend_for_mmengine/tree/main/cv2_backend_result/images again. The different now I thick may cause by line 1194.

        if self.backend == 'matplotlib':
            self.ax_save.imshow(
                img,
                extent=(0, self.width, self.height, 0),
line1194  interpolation='nearest')
        else:
            self._image = img

[KerwinKai]

It seems like that https://github.com/KerwinKai/cv2_backend_for_mmengine/blob/main/cv2_backend_result/images/draw_binary_masks_cv2.png is different from https://github.com/KerwinKai/cv2_backend_for_mmengine/blob/main/cv2_backend_result/images/draw_binary_masks_matplotlib.png.

I fix my code to orgin version in main mmengine and update https://github.com/KerwinKai/cv2_backend_for_mmengine/tree/main/cv2_backend_result/images again. The different now I thick may cause by line 1194.

        if self.backend == 'matplotlib':
            self.ax_save.imshow(
                img,
                extent=(0, self.width, self.height, 0),
line1194  interpolation='nearest')
        else:
            self._image = img

Here is a use example in larger size image: https://github.com/KerwinKai/cv2_backend_for_mmengine/tree/main/cv2_backend_result/test_binary_masks_in_larger_image

[qianyizhang]

this seems like a solid enhancement, why not getting merged?