Deep Neural Network Filter

[yaitskov]

User selects pixels, which should belong to the filter and which should not. DT builds DNN to recognize selected pixels taking into account their neighbors.

After training the DNN is applied to select pixels.

Filter parameters related to DNN:

• list of layers with number of neurons in each
• seed for random number generator
• optimization algorithm (adam, sgd, etc)
• checkbox whether to take into account pixel coordinates besides color channel values

[MStraeten]

Which problem do you want to solve with that?

[yaitskov]

I am looking for a smooth way to select pixels on a complex photo, where same looking pixels are spread all over the photo, but belong to different objects of complicated form which are not separable by sane number of drawn curves.

e.g. black hair dyed in blond over white body skin and black hair roots and black top/jeans are mixed with black rock background.

[MStraeten]

do you have an idea how to train such DNN? Each module in the dt pixel pipe has different input data from raw data without rgb information over an unlimited to a limited colorspace. So you need to train for each of those intermediate data even those you cant display on a common display ;) keep in mind: in darktable you never see the output od a module - just the output of the whole pixelpipe

[jenshannoschwalm]

I also think there is room for having some sort of "content aware masking" in dt. First problems

1. Content would most likely involve whole image data and thus interfere with roi concept
2. Need for an interface to use those "content masks"

[spaceChRiS]

Probably related: #12295.

[jenshannoschwalm]

Yes, several issues basically on one topic

[yaitskov]

do you have an idea how to train such DNN? Each module in the dt pixel pipe has different input data from raw data without rgb information over an unlimited to a limited colorspace. So you need to train for each of those intermediate data even those you cant display on a common display ;) keep in mind: in darktable you never see the output od a module - just the output of the whole pixelpipe

DNN has short life time and incrementally build with user interaction - user clicks on pixels or draw lines to select them, marking them as green (part of filter)/ red (not part of filter), until it is satisfied with approximation - click is cheap. So DNN is rebuild after some module updates down the stack.

DNN is simple and training set is small (hundred click max), that why it is can be incremental.

DNN input is a matrix of adjacent pixels 3x3 (in main use case, but should be adjustable) => 9 pixels for every pixel all channels (3 channels). Particular color space doesn't matter, because the encoded color is just a number vector.

Color spaces are isomorphic, aren't they? The richest color space could be used to mitigate difference between color space used for training and used for running if it is presented.

Code for DNN is as complex as a student's lab of AI course. 100 lines of tensorflow API, which is C++

Additional inputs for DNN might be location on the image of the filtering pixel but it is just 2 ints.

UI of the filter panel keeps lists for red / green pixels and highlights them similar to yellow filter highlight.

[yaitskov]

Probably related: #12295.

wow - just 5 days ahead of me - must be some psychic connection ;)

[github-actions[bot]]

This issue did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[github-actions[bot]]

This issue was closed because it has been inactive for 300 days since being marked as stale. Please check if the newest release or nightly build has it fixed. Please, create a new issue if the issue is not fixed.