Feature Request: ColorFilterEffect based on ColorMatrix

[freddiesmartbox]

Is your feature request related to a problem? Please describe.

We are porting a WPF application which makes use of Greyscale, variable Contrast, and variable Brightness effects: the existing WPF uses Shaders for this, but they can be expressed as well with a ColorMatrix.

Describe the solution you'd like

Support a ColorFilterEffect, which applies a 4x5 color matrix. We reckon a simple ColorFilterEffect class with a single property of (new) type ColorMatrix would be a sensible API, and maps pretty directly onto a Skia implementation.

Convenience methods for building matrices for recognised operations could be provided, and could be parsable effects, e.g. greyscale and others (https://www.w3.org/TR/filter-effects-1/#attr-valuedef-type-matrix)

We have a proof of concept in https://github.com/Smartbox-Assistive-Technology/Avalonia/tree/ColorFilterEffect and are happy to open a PR and continue to work on it if desirable

• ColorFilterEffect is implemented based on the existing BlurEffect
• ColorMatrix is based on the existing Matrix class and the Skia ColorMatrix definition: it's not ready-to-go, having not been properly tested yet, and we'd need to work out what does/doesn't need to be on the API if we go this route (probably want to strip it right back to only the essentials).
• Two examples are added to the Animations page of the RenderDemo project.

Concerns:

• ColorMatrix is a big struct following Matrix: happy to change the implementation
• The bounds of the effect in the POC are set as zero: however, if the matrix changes the alpha component, it can end up visibly rendering over everything: we don't think there is a sensible solution for this, and it's probably just something that has to be accepted as being less-than-ideal (related comment: https://github.com/AvaloniaUI/Avalonia/pull/17981#discussion_r1918191893)
• Usage disagrees with the SkiaSharp documentation as to the constant components being in the range [0, 255] rather than [0, 1] (https://learn.microsoft.com/en-us/dotnet/api/skiasharp.skcolorfilter.createcolormatrix?view=skiasharp-2.88#remarks)

Describe alternatives you've considered

Possibility of adding such an effect was previously raised in https://github.com/AvaloniaUI/Avalonia/pull/17981#issuecomment-2595055907

Our current workaround for this is an approach close to that shown in https://gist.github.com/StefanKoell/b7dd1c847984a9a9ae75ed4a96fbc4b5 which involves capturing the render buffer and painting it back over itself. This isn't ideal:

• Implementation is cumbersome and means we have Skia specific logic which we'd rather not
• It doesn't play well with transparencies

The WPF we are working from uses custom shaders for various effects, and something like https://github.com/AvaloniaUI/Avalonia/pull/17981 would work for us; however, it seems excessive for the simple effects we need, and would be Skia specific; RGBA color matrices are fairly standard, won't have a large API surface, and works fine with Skia 2 and 3 (POC commit works fine on release/11.3.1 branch)

An alternative way of implementing the ColorMatrix would be to use a Matrix4x4 and a Color/Vector4 for the constant part, or it could be a 5x5 matrix like in the GDI

• Preferred to implement ColorMatrix in the style of Avalonia.Matrix for consistency
• 5x5 matrix seems excessive, but would be valuable if we want to support ColorMatrix products.

Additional context

No response

[freddiesmartbox]

@lukesmartbox rebased against master and 11.3.9

[maxkatz6]

We had plans for a similar idea a while ago, definitely something that would be great to have. Pinging cc @miloush @kekekeks

[kekekeks]

From the first glance the PoC looks fine since it just exposes the color matrix and only deals with managed resources, it basically mirrors what we have with existing effects.

We potentially want to support some kind of color filter composition later. E. g. a heatmap filter can be implemented with Skia like this:

            var moveAlphaToChannels = SKColorFilter.CreateColorMatrix(new float[]
            {
                0, 0, 0, 1, 0,
                0, 0, 0, 1, 0,
                0, 0, 0, 1, 0,
                0, 0, 0, 0, 1,
            });
        
            var r = new byte[256];
            var g = new byte[256];
            var b = new byte[256];
            var a = new byte[256];

            for (var c = 0; c < 256; c++)
            {
                var pixel = palette.GetPixel(c, 0);
                r[c] = pixel.Red;
                g[c] = pixel.Green;
                b[c] = pixel.Blue;
                a[c] = pixel.Alpha;
            }
            var table = SKColorFilter.CreateTable(a, r, g, b);
            return SKColorFilter.CreateCompose(table, moveAlphaToChannels);

[kekekeks]

I guess CSS allows multiple effect to be applied to an element, so we should probably allow multiple effects to be applied to a visual, so it's separate from the current proposal. The effect should be named ColorMatrixFilterEffect or something. Is there a precedent of something similar in UWP/WPF or other frameworks?

[miloush]

There is e.g. https://microsoft.github.io/Win2D/WinUI3/html/T_Microsoft_Graphics_Canvas_Effects_ColorMatrixEffect.htm

[freddiesmartbox]

Having taken a look at the WinUI API+ above, the old GDI+ APIs, and Flutter's ColorFilter, I don't think there's anything on those beyond the matrix that we can realistically implement within the existing DrawingContextImpl and Skia APIs. I note that the matrix element naming in WinUI is different from that in the PoC (which is consistent with Skia and Flutter).

I've pushed a commit to the working branch renaming ColorFilterEffect to ColorMatrixFilterEffect.

If this is a welcome feature and the PoC looks reasonable, do you want us to open a PR so it's easier to comment on specifics of the implementation? We are very keen to continue working on this!

Given its fairly self-contained nature, is this something that is likely to be in a V11 release?

[Athari]

Would love to see this feature implemented. It'll make using grayscale icons for buttons and previewing images in different modes much more convenient.

What about adding HslaColorMatrixFilterEffect? SKImageFilter contains not only CreateColorFilter for RGBA, but CreateHslaColorMatrix for HSLA as well. Considering the only difference between these effects is one method, I think it makes sense to support both.

In WinUI, it corresponds to RgbToHue+ColorMatrix+HueToRgb, iiuc, which is currently impossible in Avalonia. But even when multiple filters are implemented, having that as a single filter would be convenient.

[freddiesmartbox]

If an API can be agreed then I'm sure we'd be happy to do the work for that: can't imagine it would so far removed from the RGBA variant to add significantly to the development time.

[MrJul]

API diff for review:

Avalonia.Base (net10.0, net8.0)

  namespace Avalonia.Media
  {
+     public sealed class ColorMatrix
+     {
+         public ColorMatrix(double m11, double m12, double m13, double m14, double m15, double m21, double m22, double m23, double m24, double m25, double m31, double m32, double m33, double m34, double m35, double m41, double m42, double m43, double m44, double m45);
+         public ColorMatrix Append(ColorMatrix value);
+         public static Avalonia.Matrix CreateTranslation(double xPosition, double yPosition);
+         public bool Equals(ColorMatrix other);
+         public override bool Equals(object? obj);
+         public override int GetHashCode();
+         public static bool operator ==(ColorMatrix value1, ColorMatrix value2);
+         public static bool operator !=(ColorMatrix value1, ColorMatrix value2);
+         public static ColorMatrix operator *(ColorMatrix value1, ColorMatrix value2);
+         public static ColorMatrix Parse(string s);
+         public ColorMatrix Prepend(ColorMatrix value);
+         public Color Transform(Color c);
+         public static ColorMatrix Greyscale { get; }
+         public static ColorMatrix Identity { get; }
+         public static ColorMatrix Inversion { get; }
+         public bool IsIdentity { get; }
+         public double M11 { get; }
+         public double M12 { get; }
+         public double M13 { get; }
+         public double M14 { get; }
+         public double M15 { get; }
+         public double M21 { get; }
+         public double M22 { get; }
+         public double M23 { get; }
+         public double M24 { get; }
+         public double M25 { get; }
+         public double M31 { get; }
+         public double M32 { get; }
+         public double M33 { get; }
+         public double M34 { get; }
+         public double M35 { get; }
+         public double M41 { get; }
+         public double M42 { get; }
+         public double M43 { get; }
+         public double M44 { get; }
+         public double M45 { get; }
+     }
+     public sealed class ColorMatrixFilterEffect : Effect, IColorMatrixFilterEffect, IEffect, IMutableEffect
+     {
+         public static readonly Avalonia.StyledProperty<ColorMatrix> MatrixProperty;
+         public ColorMatrixFilterEffect();
+         public IImmutableEffect ToImmutable();
+         public ColorMatrix Matrix { get; set; }
+     }
+     public interface IColorMatrixFilterEffect : IEffect
+     {
+         ColorMatrix Matrix { get; }
+     }
+     public class ImmutableColorMatrixFilterEffect : IColorMatrixFilterEffect, IEffect, IImmutableEffect
+     {
+         public ImmutableColorMatrixFilterEffect(ColorMatrix matrix);
+         public bool? Equals(IEffect? other);
+         public ColorMatrix Matrix { get; }
+     }
  }

[MrJul]

Notes from the API review meeting:

• Rename ColorMatrixFilterEffect to ColorMatrixEffect, matching WinUI and Winforms.
• Remove the CreateTranslation method (leftover?).
• Use float instead of double here. Values will range from 0 to 1, the double precision is unnecessary. (Skia even uses half precision to implement that.)

[freddiesmartbox]

Made those changes, and rebased against master

Wasn't sure if it made sense to add the additional ReadFloat method variants (rather than using ReadDouble and converting); doesn't seem to be any precedent either way.