Add ability to use custom nodes as layer or mask

[ucupumar]

Using custom nodes should be possible with these rules:

• Should be in the form of a node group
• At least output a color
• Vector input and alpha output can be optional
• There can be an option to choose custom node output as channel override if the name of the output is the same as the channel