Added paint brush tools

[bsekachev]

Motivation and context

How has this been tested?

Checklist

• [x] I submit my changes into the develop branch
• [ ] I have added a description of my changes into CHANGELOG file
• [ ] I have updated the documentation accordingly
• [ ] I have added tests to cover my changes
• [ ] I have linked related issues (read github docs)
• [ ] I have increased versions of npm packages if it is necessary (cvat-canvas, cvat-core, cvat-data and cvat-ui)

License

• [x] I submit my code changes under the same MIT License that covers the project. Feel free to contact the maintainers if that's a concern.
• [x] I have updated the license header for each file (see an example below)

# Copyright (C) 2022 Intel Corporation
#
# SPDX-License-Identifier: MIT

[kkmehta03]

i tried checking out this branch but couldn't find the brush tool for some reason. i did

docker-compose rm
docker-compose up

am i missing something here?

[bsekachev]

/check

[github-actions[bot]]

:x: Some checks failed :page_facing_up: See logs here

[bsekachev]

/check

[github-actions[bot]]

:x: Some checks failed :page_facing_up: See logs here

[bsekachev]

/check

[github-actions[bot]]

:x: Some checks failed :page_facing_up: See logs here

[bsekachev]

/check

[github-actions[bot]]

:heavy_check_mark: All checks completed successfully :page_facing_up: See logs here

[bsekachev]

@klakhov

I couldn't figure out how to use dragging feature, when I'm using this shortcut nothing happens

You can drag the panel using this element:

Do you think we can do it more clear way for a user?

Upd: if you mean changing brush size, it is implemented the same way is in photoshop: hover canvas, hold alt, press right click and drag mouse left/right.

[klakhov]

Okay, I see. Also one more question: when I'm done working with mask it becomes almost invisible, should it really be this way? What if i want to create another mask along the border of the first one?

[bsekachev]

@klakhov

Also one more question

Hmm.. In general it depends on Shape opacity value.

And according to my idea it should increase automatically to "appropriate" value, when a new mask created or appeared another way. What is your use-case?

[bsekachev]

/check

[klakhov]

Yep. Opacity slider solves this problem :smile:

[github-actions[bot]]

:x: Some checks failed :page_facing_up: See logs here

[bsekachev]

Yep. Opacity slider solves this problem smile

But nevertheless, what was your use-case? Generally I expect that this slider is corrected automatically if masks are appear on a frame

[klakhov]

For example if I want to have mask of the car and a lamp post. If opacity is low, I almost cant see car, and if I set it to high value I can see everything. I dont quite understand when opacity will change automatically except when I activate/deactivate mask. Edit: I see after i finish working with mask, all masks have 30 opacity

[bsekachev]

Edit: I see after i finish working with mask, all masks have 30 opacity

Yes, it is expected. Buy my question is what have you done the first time when it was not changed automatically :) Just would like to know if there are unknown bugs

[klakhov]

Cant reproduce it now. For some reason after switching branches, first mask that i've drawn was not restoring it's opacity

[klakhov]

Found one more problem, if I use Plus button to create new mask, brush dissapears.

[zhiltsov-max]

Tried the patch. Overall looks good. Some comments:

1. Probably, it a little bit unfair to call this new instrument "Draw a mask" from the data scientist perspective. Polygons do really work well as masks.
2. A context menu button or a separate tool to convert shapes to masks and back could be very useful (rasterize shape / vectorize mask)
3. Masks could be smoothed right after drawing
4. A mask mode to draw a filled mask could be useful (very close to the point 2 here)
5. Little mask gaps could be filled right after drawing
6. A mode to stick to the underlying edges could be useful (in polygons too. Maybe, it's more to polygons).

[bsekachev]

@zhiltsov-max

Tried the patch. Overall looks good. Some comments:

1. Probably, it a little bit unfair to call this new instrument "Draw a mask" from the data scientist perspective. Polygons do really work well as masks.
2. A context menu button or a separate tool to convert shapes to masks and back could be very useful (rasterize shape / vectorize mask)
3. Masks could be smoothed right after drawing
4. A mask mode to draw a filled mask could be useful (very close to the point 2 here)
5. Little mask gaps could be filled right after drawing
6. A mode to stick to the underlying edges could be useful (in polygons too. Maybe, it's more to polygons).

A mode to stick to the underlying edges could be useful (in polygons too. Maybe, it's more to polygons).

It is implemented for polygons as automatic bordering mode. For masks we have Remove underlying pixels that allow to solve the use-case.

I'm not sure I understand items 3-5, we can discuss them.

[zhiltsov-max]

3. Masks could be smoothed right after drawing

In some editors I've seen that the scribble the user draws (while a mouse button is pressed) is not being rasterized right away. Instead, while the button is pressed, the preview is shown, but when the button is released, the scribble drawn is being smoothed (probably, some kind of a Bezier curve is created to fit the scribble), and this new curve is what is actually resterized. The result is that the scribble is more smooth, i.e. has lesser sharp edges, but it may be slightly different from what the user has drawn.

4. A mask mode to draw a filled mask could be useful (very close to the point 2 here)

It is similar to what we understand as polygons. But if we draw a polygon and then convert it to a mask, we need to do extra clicks. With such mode where we can choose to draw a filled contour, the mask we get is already filled without extra actions. Otherwise, in the current mode, in many cases the user needs to manually fill the interior of the drawn mask by mouse, and this leads us to the next problem:

5. Little mask gaps could be filled right after drawing

If we are trying to fill a big contour by mouse scribbles, we often get numerous little gaps in the mask. This is especially observable if the brush size is small. And it may be hard or time-consuming to find and fix these little holes that should be filled in the resulting mask.

[bsekachev]

@zhiltsov-max

It is similar to what we understand as polygons. But if we draw a polygon and then convert it to a mask, we need to do extra clicks. With such mode where we can choose to draw a filled contour, the mask we get is already filled without extra actions. Otherwise, in the current mode, in many cases the user needs to manually fill the interior of the drawn mask by mouse, and this leads us to the next problem:

As I understand what you are talking about is implemented already:

Speaking about others, I am not sure we should change user's input somehow, because they are manual tools. We should not decide for a users what they want.

[zhiltsov-max]

Yes, this looks similar. Maybe I've overlooked some features.

[nmanovic]

@bsekachev , several comments to improve UX:

• Need tooltips instead of notifications. Sometimes it is very difficult to understand what a function (button) does.
• In polygon tool some very important features are absent. For example, quick border drawing. If it is possible to unify drawing capabilities for polygons, it will be awesome.
• For brush it is necessary to hide the cursor. Otherwise I see some artifacts, which effect the usability.
• CTRL+1 and CTRL+2 doesn't change the label when I select the tool but before I click "draw mask" button.
• It is not very intuitive for new users to finish drawing a polygon by double click.
• Saving operation is extremely slow. I drew a complex mask, propagated it on 50 frames. It is saving results for several minutes.

[bsekachev]

Need tooltips instead of notifications. Sometimes it is very difficult to understand what a function (button) does. For brush it is necessary to hide the cursor. Otherwise I see some artifacts, which effect the usability.

Done.

CTRL+1 and CTRL+2 doesn't change the label when I select the tool but before I click "draw mask" button. Saving operation is extremely slow. I drew a complex mask, propagated it on 50 frames. It is saving results for several minutes.

Will investigate this..

In polygon tool some very important features are absent. For example, quick border drawing. If it is possible to unify drawing capabilities for polygons, it will be awesome. It is not very intuitive for new users to finish drawing a polygon by double click.

I will think if we can easily use drawing polygons implemented with SVG.js here. But it does not solve the second issue.

[bsekachev]

/check

[github-actions[bot]]

:x: Some checks failed :page_facing_up: See logs here

[bsekachev]

@nmanovic

Saving operation is extremely slow. I drew a complex mask, propagated it on 50 frames. It is saving results for several minutes.

Should be much faster now.