JavaFX Support: Enhancements to SVGPaint / Output API

[SonarSonic]

While testing the library I wrote some basic implementations of Output for Skija / JavaFX, I thought I'd mention a few things I found tricky to achieve with the current API.

• [ ] Rendering the custom paint types. - Currently it is not currently possible to render custom paint types without reflection. MaskedPaint, RGBColor, SVGMultipleGradientPaint, SVGRadialGradientPaint and TransformedPaint all have fields which need to be accessible to effectively render the paint in a renderer which is not AWT based. I'm happy to make a PR for this, just let me know in what way you'd prefer the variables to be accessible.

• [ ] Rendering filters and blend modes - currently the filters are drawn to a BufferedImage and then sent to the output, Unlike AWT, Skija and JavaFX provide BlendModes directly, and many of the filters have hardware-accelerated alternatives which prevents the need for this additional step, and allows outputs from Skija to still be cached as vectors to enable zooming without a loss in quality. Though some filters would be tricky to render without having access back to some of the original attributes, e.g. the stack of filters used for DropShadow, could also be replaced with a faster alternative.

• [ ] Specifically for blend modes perhaps the API could have a currentBlendMode() + applyBlendMode() and supportsBlendMode() - if it's supported the Output can use the faster blend mode method if it's not it can use the fallback method of rendering to a BufferedImage first

[weisJ]

Having other supported Outputs would certainly be very nice. I started the whole Output refactor with future support for JavaFX in mind, but never got around to deep dive into how rendering works there.

• For custom paint types it should be fine to "publicly" (as in public for the implementation api, the classes are not exposed as module public) expose the required fields. For gradients this might require passing an Output into the paintForBounds function to skip the hardware acceleration optimisation, so the jsvg version of the class is use (which can me modified to expose members).

  This would allow for a good first draft, which doesn't aggressively alter the API. I don't quite like that the awt Paints are still instantiated. But without insight into what would need to change I don't want to spend too much time coming up with an abstraction, that probably would need to change down the line anyway.

  The tricky part would definitely be MaskedPaint - It is quite invasive in its nature. Ideally the mask image itself would be painted into a for JavaFX/Skia suitable buffer, independent of BufferedImage.

• The filter pipeline is very much designed with AWT in mind. Maybe a FilterPipeline interface for Output could be utilised to build the suitable internal structure for applying filters. Having some (ugly) first version of filters for JavaFX would maybe help to see if there is a common abstraction here.

• I think that makes sense it would need to move the BlendMode enum into public api, so we should be careful. For example I wouldn't want to expose HasMatchName as it is very much an internal interface.

Thank you for tackling this. I would be very much interested in merging new backends for JSVG. Depending on what dependencies they have, they should probably be their own modules, but that can wait for a first implementation.

[SonarSonic]

• JavaFX Paints - JavaFX's approach to Paints is very similar to AWT, and almost always behind the scenes it boils down to some derivative of AWT with minor changes for JFX (though this does mean annoyingly even though they are basically identical you can't just pass the AWT paint instances) - though as they follow so closely, it's a relatively lightweight operation for example to hold temporary RadialPaint and LinearPaint instances in the JFX Output implementation, which are just updated from the AWT paints. What gets more tricky is when TexturePaint gets involved, which needs to be mapped to ImagePattern in JavaFX...then comes in Image Inter-Op

• Image inter-op - I have done some experimenting with ByteBuffers which allows AWT / JFX / Skija images to share the same ByteBuffer without needing to copy image data, but they must be stored in specific formats either as ARGB (int) or BGRA (bytes). Skija and JFX perform better with BGRA as they literally then just copy it to the draw buffer, annoyingly AWT only provides TYPE_4BYTE_ABGR and not TYPE_4BYTE_BGRA order so a custom ComponentColorModel is needed. This is not the most robust solution, as it is not entirely thread safe if you keep instances around if you have AWT / JFX / Skija renderers all in the same pipeline (so it's better to wrap the byte buffer, draw it, then dispose of the wrapper), and I haven't seen anyone else deploying this solution, but it is very fast. And it could allow for storing image data needed in a Platform agnostic way - and if for whatever reason the image can't be provided in a compatible format, we can have a fallback to perform a slower conversion.

• Skija Paints - Skija also has a very different philosophy when it comes to paints, it essentially has the paint as a state object which is very lightweight, which also means in most cases an on-the-fly conversion to a Skia paint from the AWT paints is not too intensive, so again we could keep the majority of the current API in tact.

• FilterPipeline - I really like your idea of a FilterPipeline, that makes a lot of sense and would allow for a lot of speed ups as JavaFX has implementations for most Effects directly which will be much faster than the buffered image approach, I also believe they follow the SVG spec so getting equivalent results shouldn't be too difficult. Skija also has implementations for most the required filters also.

• Blend Modes - I think BlendMode should definitely be a public enum as Skija and JFX both have public enums for it, and it is commonly used in their APIs. Perhaps if the blend mode just has a string for it's 'SVG Value' and in the AttributeParser there's just a parseBlendMode method to avoid needing the HasMatchName interface. Or alternatively just move everything related to parsing the blend mode to the AttributeParser?

• Skija's SVG Renderer - Finally I did want to add that Skija can render SVG directly, so it is less required, however I would personally still use the JSVG -> Skija variant as having everything abstracted away in the C++ wrapper is not always useful, and the Java side of the SVG API is much more limited. I've also already found a few instances where the JSVG -> Skija workflow delivers more accurate clipping results (but it's possible that's because Skija haven't updated the skia dependency in a while)

Anyway that's all my current thinking! Would be great to hear your thoughts and explore this some more.

Ollie

[weisJ]

Thanks for the in-depth information. Texture paints are also a pain-point in the AWT implementation currently e.g. specifying a pattern-transform completely breaks antialiasing. Ideally the pattern should be painted to a device pixel grid aligned rectangular region and filling in the resulting gaps "by hand" through manual repetition. Just never got around to implementing it. I think this is an issue we could tackle later on, if the rest seems feasible enough.

For the image inter-op we should make sure that we can keep the current storage type for the AWT implementation. Even though the color model is implemented in Java the painting pipeline of AWT is very much aware of known classes and optimises it under the hood, which breaks with custom color models. Though it shouldn't be too difficult to abstract away the needed operations, so the other backends can keep their preferred buffer format.

For the blend mode we can certainly just move the parsing into a specialised method (just a large switch would be fine) and remove the interface from the enum.

For the filter pipeline feel free to prototype something that would fit the JavaFX implementation. I can then say more about how easy it is to adapt the awt version to the new interface. It needs a rewrite anyway :D

I think trying to get a JavaFX backend working would be a good start. I am very much looking forward to it.

[SonarSonic]

Good to know re: Texture Paints, I think I was possibly seeing the results of those anti-aliasing artifacts in an early render with some of the complex sample svgs.

Thanks for the info about the AWT color models, I use it much less in my workflow so good to know that the specific Color Models are optimised also.

In that case there are a few approaches I can think of, the first would just be to copy the image data on the fly and then dispose it once its rendered, the 2nd would be to have a growable image buffer used by Skija and the JavaFX Output implementations which grow up to the size of the images used which are disposed of with the Output. Perhaps we could even pre-allocate a buffer big enough for all the images, though I guess in some instances the image buffer might need to be larger than the draw buffer, but that might still be a good initial size. I think any other approach would be a much bigger change to the API, as in not storing any render data in AWT format. It's probably easier to think of the Skiaj and JavaFX implementations as wrappers, with a few optimisations e.g. (filter pipeline).

I'm thinking I'll probably start off like this

• Baseline wrapper with no modifications (with not everything supported)
• Make specialist paint types accessible (it should render the same at that point)
• Make blend mode configurable
• Image optimisations
• Filter Pipeline