html icon indicating copy to clipboard operation
html copied to clipboard
verified publisher icon whatwg

Add a layer and filter interface in the 2D canvas

Open graveljp opened this issue 2 years ago • 18 comments

This adds new beginLayer and endLayer functions to open and close layers in the canvas. While layers are active, draw calls operate on a separate texture that gets composited to the parent output bitmap when the layer is closed. An optional filter can be specified in beginLayer, allowing effects to be applied to the layer's texture when it's composited its parent.

Fixes #8476

  • [X] At least two implementers are interested (and none opposed):
    • Chromium
    • Gecko
    • WebKit
  • [X] Tests are written and can be reviewed and commented upon at:
    • https://github.com/web-platform-tests/wpt/tree/master/html/canvas/element/layers
    • https://github.com/web-platform-tests/wpt/tree/master/html/canvas/offscreen/layers
  • [X] Implementation bugs are filed:
    • Chromium: crbug.com/1396372
    • Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1866551
    • WebKit: https://bugs.webkit.org/show_bug.cgi?id=259494
  • [ ] MDN issue is filed: …

(See WHATWG Working Mode: Changes for more details.)

/acknowledgements.html ( diff ) /canvas.html ( diff ) /index.html ( diff ) /infrastructure.html ( diff )

graveljp avatar Jul 19 '23 19:07 graveljp

As discussed with WebKit, I have removed the XML-based CanvasFilter proposal from this PR. We will start with layers supporting only CSS filters at first. The XML filters proposal will be moved to a different PR.

graveljp avatar Jun 03 '24 13:06 graveljp

We (the WebKit team) opposed the Canvas filter API at the beginning because we thought it is an expensive operation to be applied to every single drawing command. And we thought the real use case will be applying the filter on a more complex drawing. So we supported the layer API instead and we liked passing the filter as an argument to the beginLayer() through BeginLayerOptions.

But after rethinking we found out, the filter and layer APIs can interact nicely to support both scenarios:

  1. Applying the filter on a single drawing command
  2. Applying the filter on a complex drawing.

So the canvas filter was implemented in https://commits.webkit.org/278000@main. It is still off by default but we are working on getting on by default.

But when trying to implement the layer API, the current design of this document seems to have three flaws/inconsistencies:

  1. BeginLayerOptions introduced something called the layer filter which is different from the global filter. No other state member have this pattern.
  2. The globalAlpha, globalCompositeOperation and shadow are applied only when endLayer() is called. But the global filter is not applied. Instead the layer filter is applied when endLayer() is called..
  3. The global filter is not part of layer rendering state. So unlike globalAlpha, it is not saved, reset at beginLayer() and it is not restored at the endLayer().

So I have the following proposal:

  1. Like globalAlpha, globalCompositeOperation and shadow, make the global filter a member of the layer rendering state.
  2. Do not pass anything to beginLayer()
  3. Save the global filter and reset it to "none" when beginLayer() is called and restore it back to its value when endLayer() is called.

So this is how layer API and filter API interact with this proposal:

ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);	// No filter is applied
    ctx.fillRect(...);	// No filter is applied
ctx.endLayer(); // blur filter is applied when compositing the layer to the destination context

ctx.filter = 'drop-shadow(10px 10px 5px orange)';
ctx.beginLayer();
    ctx.fillRect(...);	// No filter is applied
    ctx.fillRect(...);	// No filter is applied
ctx.endLayer(); // drop-shadow filter is applied when compositing the layer to the destination context

And this is how nested layers and filters interact

ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);	// No filter is applied
    ctx.fillRect(...);	// No filter is applied

    ctx.filter = 'drop-shadow(10px 10px 5px orange)';
    ctx.beginLayer();
        ctx.fillRect(...);	// No filter is applied
        ctx.fillRect(...);	// No filter is applied
    ctx.endLayer(); // drop-shadow filter is applied when compositing the layer to the parent layer
ctx.endLayer(); // blur filter is applied when compositing the layer to the destination context

And this is how the filter can be applied to any drawing even inside the layer:

ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);	// No filter is applied
    ctx.fillRect(...);	// No filter is applied

    ctx.filter = 'drop-shadow(10px 10px 5px orange)';
    ctx.fillRect(...);  // drop-shadow filter is applied to draw call.
    ctx.fillRect(...);  // drop-shadow filter is applied to draw call.

    ctx.filter = 'none';
    ctx.fillRect(...);	// No filter is applied
ctx.endLayer(); // blur filter is applied when compositing the layer to the destination context

And this is how filter and globalAlpha interact with canvas layers:

ctx.globalAlpha = 0.5;
ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);	// No filter or globalAlpha is applied.
    ctx.fillRect(...);	// No filter or globalAlpha is applied.
ctx.endLayer(); // globalAlpha = 0.5 and blur filter are applied when compositing the layer to the destination context

I implemented a draft of this proposal here https://github.com/WebKit/WebKit/pull/28324.

shallawa avatar Jun 03 '24 18:06 shallawa

Thanks @shallawa for the update.

One of the main motivation behind this PR was to address WebKit's concerns with the context.filter = ... API, which WebKit never approved of.

From https://github.com/whatwg/html/issues/8476#issuecomment-1306552408:

WebKit certainly has [interest] in a beginLayer()/endLayer() API. We believe the "filter each drawing call independently" behavior of the current canvas filter API was a mistake.

From https://github.com/whatwg/html/issues/5621#issuecomment-1112838052:

I think the API as it stands is a footgun, encouraging authors to write inefficient code (where every drawing effect gets filtered), and getting incorrect results (expectations of grouped effects where none occurs).

Not all implementations can simply map filters to GPU shaders, so filter operations can have significant cost. Filtering on every draw call has potentially high performance cost on those platforms.

From https://github.com/whatwg/html/issues/5621#issuecomment-1113901018:

An API that applies filters independently to each individual drawing command is bad for everyone:

  • Authors almost certainly don't want this behavior, as authors want to apply filters to images/graphics, not each individual drawing command.

  • Implementations don't want this behavior either, because the performance cost of filters is significantly higher than the performance cost of a single drawing operation. So, a modal API that causes every drawing operation to get way slower and use way more memory, would be a mistake.

Passing filters to beginLayer() was the solution we agreed on to fix this past "mistake" and provide a safe and efficient mechanism for filtering a group of draw calls.

Is WebKit changing stance on this? Will WebKit support global context.filter = ... for individual draw calls, as your third example suggests?

With your proposal, filters would no longer be scoped to the layer:

ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);  // No filter is applied
ctx.endLayer(); // blur filter is applied when compositing the layer to the destination context
ctx.fillRect(...);  // blur filter is applied

Thus, it would not be possible to use filters on layers without triggering the footgun concern raised above. This seems like a step in the wrong direction, away from WebKit's original position.

graveljp avatar Jun 04 '24 01:06 graveljp

Is WebKit changing stance on this? Will WebKit support global context.filter = ... for individual draw calls, as your third example suggests?

Yes. context.filter = ... API was landed in https://commits.webkit.org/278000@main. It is still off by default but we are working on getting it on by default.

With your proposal, filters would no longer be scoped to the layer:

ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);  // No filter is applied
ctx.endLayer(); // blur filter is applied when compositing the layer to the destination context
ctx.fillRect(...);  // blur filter is applied

Yes this is correct. And we think this is better because this is consistent with other layer rendering state members: globalAlpha, globalCompositeOperation and shadow.

Thus, it would not be possible to use filters on layers without triggering the footgun concern raised above.

Layers should solve the problem of applying the filter on individual draw calls.

This seems like a step in the wrong direction, away from WebKit's original position.

Actually we think implementing the context.filter = ... API with the canvas layers is the right direction. We think also adding the filter to the layer rendering state will make it consistent with other members like globalAlpha in this example.

ctx.globalAlpha = 0.5;
ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);	// No filter or alpha is applied.
    ctx.globalAlpha = 0.7; // Its original value will be restored when endLayer() is called.
    ctx.filter = 'grayscale(1)'; // Its original value will be restored when endLayer() is called.
    ctx.fillRect(...);	// alpha = 0.7 and grayscale filter are applied
ctx.endLayer(); // alpha = 0.5 and blur filter are applied when compositing the layer to the destination context
ctx.fillRect(...); // alpha = 0.5 and blur filter are applied

But introducing BeginLayerOptions will add ambiguity between global filter and layer filter like in this example.

ctx.globalAlpha = 0.5;
ctx.filter = 'blur(5px)'
ctx.beginLayer({filter: 'drop-shadow(10px 10px 5px orange)'});
    ctx.fillRect(...);	// no alpha is applied but blur filter is applied to draw call.
    ctx.globalAlpha = 0.7; // Its original value will be restored when endLayer() is called.
    ctx.filter = 'grayscale(1)'; // Its original value will NOT be restored when endLayer() is called.
    ctx.fillRect(...);	// alpha = 0.7 and grayscale filter are applied to draw call.
ctx.endLayer(); // alpha = 0.5 and drop-shadow filter are applied when compositing the layer to the destination context
ctx.fillRect(...); // alpha = 0.5 and grayscale filter are applied to draw call.

shallawa avatar Jun 04 '24 15:06 shallawa

context.filter = ... API was landed in https://commits.webkit.org/278000@main. It is still off by default but we are working on getting it on by default.

This is a great news for browser interoperability! The inconsistencies you noted were there because the ctx.filter state wasn't supported by Safari and the original agreement was to propose an API to replace it, not extend it. But things are different now that Safari supports it.

If ctx.filter gets full support by all browsers, it makes total sense to include it in the layer rendering state and have it apply to layers' output. This however is orthogonal to the proposal of having beginLayer accept a filter. We could have both: the two filters could apply in sequence to the layer's output, first the layer filter, then the global filter.

Having the layer accept a filter would have many advantages. It would scope the filter to the layer, thus removing the risk of filtering individual draw calls via the global filter. This would address the original concerns without introducing inconsistencies in the spec.

Then, it would allow hierarchical filter transformations, like CSS and SVG do. ctx.filter is spec'ed with the same logic as global shadows in that transforms do not apply to the filter. This makes sense because if you set a drop-shadow filter, the shadow should be the same for all draw calls, irrespective of whether a transform was used to position different draw calls in the scene. For instance:

ctx.filter = 'drop-shadow(0px 5px 0px grey)';
ctx.fillText('Has a 5px shadow ', 0, 20);
ctx.scale(3, 3);
ctx.fillText('Also has a 5px shadow', 0, 20);

Having beginLayer accept filters would make it possible to hierarchically transform filters:

ctx.fillText('Has a 5px shadow ', 0, 20);
ctx.scale(3, 3);
ctx.beginLayer({filter: 'drop-shadow(0px 5px 0px grey)'});
ctx.fillText('Has a 15px shadow', 0, 20);
ctx.endLayer();

This is how CSS and SVG work:

<div style="width: 500px; height: 60px">
  <div style="filter: drop-shadow(0px 5px 0px Grey)">
    Has a 5px shadow.
  </div>

  <div style="transform-origin: top left; transform: scale(3);">
    <div style="filter: drop-shadow(0px 5px 0px Grey)">
      Has a 15px shadow.
    </div>
  </div>
</div>

<svg viewBox="0 0 400 300" xmlns="http://www.w3.org/2000/svg">
  <filter id="filter" x="-100%" y="-100%" width="300%" height="300%">
    <feDropShadow dx="0" dy="5" stdDeviation="0" flood-color="grey" />
  </filter>

  <g filter="url(#filter)">
    <text x="0" y="20">Has 5px shadow.</text>
  </g>
  <g transform="scale(3)">
    <g filter="url(#filter)">
      <text x="0" y="30">Has 15px shadow</text>
    </g>
  </g>
</svg>

Regarding the issue of having only filters be accepted by beginLayer and not other global states, the idea of having beginLayer also accept an alpha and compositeOperation was suggested before. Maybe that's an idea we could explore further?

graveljp avatar Jun 04 '24 16:06 graveljp

This is a great news for browser interoperability! The inconsistencies you noted were there because the ctx.filter state wasn't supported by Safari and the original agreement was to propose an API to replace it, not extend it. But things are different now that Safari supports it.

If ctx.filter gets full support by all browsers, it makes total sense to include it in the layer rendering state and have it apply to layers' output.

Great. So I think we have an agreement here: ctx.filter will be supported and global filter should in the layer rendering state and have it apply to layers' output.

Then, it would allow hierarchical filter transformations, like CSS and SVG do. ctx.filter is spec'ed with the same logic as global shadows in that transforms do not apply to the filter. This makes sense because if you set a drop-shadow filter, the shadow should be the same for all draw calls, irrespective of whether a transform was used to position different draw calls in the scene.

Where is the specs that says the scaling transforms do not apply to the drop-shadow filter and the canvas shadow? I really can't find it although I found all the browsers including Safari follow this rule. The rotation and the translation are still applied.

Regarding the issue of having only filters be accepted by beginLayer and not other global states, the idea of having beginLayer also accept an alpha and compositeOperation was suggested before. Maybe that's an idea we could explore further?

Yes I agree. Maybe it is worthy a separate GitHub issue. It is up to you.

shallawa avatar Jun 05 '24 20:06 shallawa

Where is the specs that says the scaling transforms do not apply to the drop-shadow filter and the canvas shadow? I really can't find it although I found all the browsers including Safari follow this rule. The rotation and the translation are still applied.

Technically the rotation and translation don't apply to the shadow either. The shadow is casted from the translated/rotated foreground, but the background isn't getting an additionnal translation and rotation. This is defined in 4.12.5.1.21 Drawing model:

  1. Render the shape or image onto an infinite transparent black bitmap, creating image A, as described in the previous sections. For shapes, the current fill, stroke, and line styles must be honored, and the stroke must itself also be subjected to the current transformation matrix.

  2. When the current filter is set to a value other than "none" and all the externally-defined filters it references, if any, are in documents that are currently loaded, then use image A as the input to the current filter, creating image B. If the current filter is a string parseable as a filter-value-list, then draw using the current filter in the same manner as SVG.

    Otherwise, let B be an alias for A.

  3. When shadows are drawn, render the shadow from image B, using the current shadow styles, creating image C.

Thus, the only place where a transform is applied is on the stroke. The filter and shadow are applied on the transformed image.

Regarding the issue of having only filters be accepted by beginLayer and not other global states, the idea of having beginLayer also accept an alpha and compositeOperation was suggested before. Maybe that's an idea we could explore further?

Yes I agree. Maybe it is worthy a separate GitHub issue. It is up to you.

I would proceed with this PR having beginLayer accepting a filter argument and consider other render states in a future PR. The filter argument was a major motivation for implementing layers in the first place. If we launch layers without beginLayer filters, developers will have no choice but learn a bad habit of using layers with the global filter, which is not something we should encourage. If on the other hand we launch layers with beginLayer filter support, we'll be able to document it from the start by saying something like: "Global filters apply to layers, but it's preferable to filter layers using the beginLayer filter argument. This puts a scope to the filter, preventing it from applying to individual draw calls. Filtering individual draw calls with a global filter is inefficient since each draw command implicitly requires the creation of a layer. It's better to make this cost explicit by using layers in the first place."

Thoughts?

graveljp avatar Jun 10 '24 15:06 graveljp

@graveljp I'm not sure I understand. Wouldn't the changes @shallawa suggested make filter no longer truly global and the same as other state members, such as globalCompositeOperation? It seems that would still address the main motivation for adding layers. I guess I don't see why we would want an inconsistent API between them? Unless I'm missing something I'd prefer starting out with beginLayer() not accepting any arguments and then if we want to go there do it consistently for all state, not just filters.

annevk avatar Jun 10 '24 16:06 annevk

Sorry for the confusion. I meant that the filter state is a global state that applies to all draw calls at the current layer level.

Let me elaborate on my previous message. The original concern WebKit had with ctx.filter was that it applies to each draw calls individually. So, if you need to use ctx.filter to filter layers, you need to enable this mode of operation that will be inefficient for all draw calls other than layers. Take my previous example for instance:

ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);  // No filter applied.
ctx.endLayer(); // blur filter is applied.
ctx.fillRect(...);  // Implicitly creates a layer to apply the blur filter.

Here, a developer wanting to do it right and use layer-only filters would need to do either:

ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);  // No filter applied.
ctx.endLayer(); // blur filter is applied.
ctx.filter = 'none';
ctx.fillRect(...);  // No filter applied.

or:

ctx.save();
ctx.filter = 'blur(5px)';
ctx.beginLayer();
    ctx.fillRect(...);  // No filter applied.
ctx.endLayer(); // blur filter is applied.
ctx.restore();
ctx.fillRect(...);  // No filter applied.

The beginLayer filter argument would be friendlier and safer to use:

ctx.beginLayer({filter: 'blur(5px)'});
    ctx.fillRect(...);  // No filter applied.
ctx.endLayer(); // blur filter is applied.
ctx.fillRect(...);  // No filter applied.

Is this no longer a concern?

graveljp avatar Jun 10 '24 16:06 graveljp

Is this no longer a concern?

I think without implementing the layers we already have a problem. The drop-shadow can be applied by two different ways:

  1. Shadow properties
ctx.shadowColor = "green";
ctx.shadowOffsetX = 10;
ctx.shadowOffsetY = 10;
ctx.fillRect(...);
  1. Global filter
ctx.filter = 'drop-shadow(10px 10px 0 green)';
ctx.fillRect(...);

And with the beginLayer() filter argument, we are making things even worse by introducing a third way to do exactly the same thing:

  1. beginLayer() filter argument
ctx.beginLayer({filter: 'drop-shadow(10px 10px 0 green)'});
    ctx.fillRect(...);
ctx.endLayer();

The only difference between the beginLayer() and the other two methods is beginLayer() implicitly includes save/restore for the layer rendering state.

shallawa avatar Jun 11 '24 00:06 shallawa

If I may chime in, first it's great to see WebKit came back on this, thanks for your hard work.

Then, I see @graveljp's point, but having a discrepancy between filter and other render states doesn't sound good to me. The same concerns can be raised about globalCompositeOperation and shadows, and alpha... The only difference is that these were supported by all browsers long before. But still if one wanted to do it correctly, they'd have to either save()/restore() around each layers, or to reset manually to the defaults after the layer is drawn.

I already expressed this, but as a web-dev who's already in a habit of using multiple <canvas> elements to achieve what the layers are bringing, I would strongly prefer to have an API where all the layer modifying attributes – alpha, compositeOperation, filter, shadowXXX, and probably even clipPath and transform – are attributes to beginLayer() or to endLayer() (i.e. BeginLayerOption), and to have all the CanvasStates reset at the beginning of the layer definition. This would allow modular scripting where each "layer" can work with a clean state, and each consumer of the layer can apply the modifications it sees fit.
I heard the argument that resetting everything might be slow on the implementers' end, but I bet that in most cases authors will have to do it themselves anyway. So better have an user-friendly API from the get-go. I think it's agreed that layers are far less prone to be a footgun than individual filters and shadows. So better make it attractive so that users actually adopt it.

Having a mix of states that do leak inside the layers, others that do not but will affect how the whole layer and the other drawings from the owner are rendered, and yet others that will only affect how the layer is rendered, sounds very complex.

At worst, if we can't an reach agreement on passing all the render states as BeginLayerOption, I'd prefer to have no option at all. The hierarchical thing can also be achieved by nesting one more layer.

Kaiido avatar Jun 11 '24 00:06 Kaiido

The only difference between the beginLayer() and the other two methods is beginLayer() implicitly includes save/restore for the layer rendering state.

https://github.com/whatwg/html/pull/9537#issuecomment-2147998208 above mentions another difference. Transforms are not applied to filters specified via ctx.filter. They would be applied to filters applied via beginLayer.

Then, I see @graveljp's point, but having a discrepancy between filter and other render states doesn't sound good to me.

To be clear, I'm not against adding all render states in BeginLayerOption, I'm just trying to understand if we need to add all of these in this PR, or whether we can do some of that in follow ups. I think that having filter in BeginLayerOption provides more value than the other states, which it seems we want to include mostly for the sake of consistency. But I see that "eventual consistency" isn't getting much traction, so I guess we can rule that out.

I'll look at spec'ing and implementing BeginLayerOption with alpha, compositeOperation, filter and shadowXXX.

graveljp avatar Jun 11 '24 17:06 graveljp

They would be applied to filters applied via beginLayer.

That's one more surprising behavior and I don't see where it would come from in the current PR. In https://github.com/whatwg/html/pull/9537#issuecomment-2147998208 you stated that it's how CSS and SVG work, but there, it's not only the filters that are affected by the transform, but the whole coord system. For instance in your example the default font-size has also been scaled by 3. If I'm not mistaken, the equivalent for layers would be to have the current transform affect how the whole layer is rendered, not just the filter.

I'll look at spec'ing and implementing BeginLayerOption with alpha, compositeOperation, filter and shadowXXX.

To clarify my personal opinion further, I believe we should have either BeginLayerOption, either have the context's current rendering states affect the layer's rendering, but not both. Having both will inevitably lead to confusion as to which states will apply, in which order, additive or replacing, etc.

Kaiido avatar Jun 11 '24 22:06 Kaiido

That's one more surprising behavior and I don't see where it would come from in the current PR.

You are right that the current PR doesn't fully correctly this.

To avoid resampling layer's output, which would be slow and lower image quality, the current PR keeps the transform global and doesn't reset it when entering layers. Thus, draw calls write pixels at the position they will have in the final raster. Because there is no concepts of layer-local transforms and because ctx.filter doesn't observe the current transform, it follows that ctx.filter never observes the transform irrespective of whether it's nested in a layer. What's missing is for the spec to mention that the transform should apply to the filter in BeginLayerOption.

I think that in the above, the surprising behaviour is that context filters inside a layer are not affected by the transform in the parent layer. When scaling a whole layer, the content of the layer in all it's details should scale the same way and that should include ctx.filter. Maybe we need to add a concept of parent transform, which would be applied to ctx.filter and ctx.shadowXXX. If we did this, then indeed BeginLayerOptions would be equivalent to the context state plus a save/restore pair. BeginLayerOptions would then be just a syntactic sugar allowing a safer and nicer usage pattern.

You have mentioned the idea of resetting the transform inside layer. It's an interesting idea, but I don't know how we could spec that without requiring layer resampling. It looks like we would need to keep a global transform regardless, to draw pixels at their intended final position, and keep an "artificial" layer-local transform for the only purpose of having getTransform return a layer-local matrix. This adds complexity and disaligns the API with how the canvas really works, for little added value.

graveljp avatar Jun 12 '24 02:06 graveljp

Making the transform layer-local (resetting it when we enter a layer) might have another advantage. If the transform is always global, calling setTransform from within a layer is really strange.

If a non-invertible matrix is used, nothing can get drawn to the canvas:

ctx.scale(0, 0);
ctx.fillRect(0, 0, 10, 10);  // Draws nothing.

This should apply to layers too:

ctx.scale(0, 0);
ctx.beginLayer();
  ctx.fillRect(0, 0, 10, 10);
ctx.endLayer()  // Draws nothing.

But what happens if setTransform is called from within the layer? If the transform is global, we have:

ctx.scale(0, 0);  // Non-invertible matrix.
ctx.beginLayer();  // The whole layer is non-rasterizable.
  ctx.fillRect(0, 0, 10, 10);  // Can't be drawn, the transform is still non-invertible.
  ctx.setTransform(1, 0, 0, 1, 0, 0);  // Restores the matrix to identity.
  ctx.fillRect(0, 0, 10, 10);  // Unclear what happens here.
ctx.endLayer();

If the transform is layer-local, we have:

ctx.scale(0, 0);  // Non-invertible matrix.
ctx.beginLayer();  // The whole layer is non-rasterizable.
  ctx.fillRect(0, 0, 10, 10);  // Can be drawn in the layer (but implementations could optimize away if wanted).
  ctx.setTransform(1, 0, 0, 1, 0, 0);  // No-op, the layer transform is already identity.
  ctx.fillRect(0, 0, 10, 10);  // Same: can be drawn.
ctx.endLayer();  // Draws nothing to the top level output bitmap.

I think we could spec that by maintaining a parent transform (already needed for ctx.filter), a layer transform, and then having all draw calls use parent transform x layer transform to transform strokes and images.

Would that work?

graveljp avatar Jun 12 '24 19:06 graveljp

Hi all,

I uploaded a few updates to this PR following to the above discussions. The changes since the last update include:

  • ctx.filter and the current transformation matrix are now part of the the layer rendering states. This means that they are both reset to their default value when entering a layer.
  • As before, filters and shadows ignore the layer's current transform, but they are now impacted by the transforms of parent layers. This means that if you scale a layer containing a shadowed shape, the shape and the shadow will scale together.
  • I have removed the BeginLayerOptions dictionary, meaning that beginLayer() no longer has a parameter. Filters can now only be specified via ctx.filter.

The current transformation matrix change was tricky to do. On the one hand, shadows and filters are no longer global, so we need to keep track of the coordinate spaces in which they are drawn. On the other hand, we have concepts like the current default path which stores coordinates pre-transformed with with the global matrix. To make everything work, the spec now defines three different types of matrices:

  • The current transformation matrix: the matrix local to the current layer. This is the matrix manipulated by the transforms API.
  • The parents transformation matrix: the multiplication of the current transformation matrix of all ancestors of the current layer. This is the matrix applied to the shadows and filters.
  • The total transformation matrix: the multiplication of all matrices, equivalent to multiplying parents and the current transforms. This is the matrix applied to shapes, images and paths.

To apply the shadow on pre-transformed shapes like the ones using the current default path, I added an algorithm for applying a filter or shadow "in layer coordinate space".

Please review these latest changes and let me know if there's any issue.

graveljp avatar Sep 23 '24 20:09 graveljp

I still haven't had enough time for a thorough review, but regarding the new total transformation matrix property, instead of being an actual object that is set, it's live and always the result of the multiplication of both other matrices, right? Seems a bit unusual to my non-editor's eyes, but I'll let @domenic confirm it can work like this.

Otherwise at a glance it looks good (modulo some typos), but I'll try to take the time to think of all the edge cases.

Kaiido avatar Sep 26 '24 00:09 Kaiido

regarding the new total transformation matrix property, instead of being an actual object that is set, it's live and always the result of the multiplication of both other matrices, right?

That's the idea. In the PR, I wrote:

The <span>total transformation matrix</span> [...] is always equals to the results of
multiplying the <span>parents transformation matrix</span> with the <span>current transformation
matrix</span>.

Otherwise at a glance it looks good (modulo some typos), but I'll try to take the time to think of all the edge cases.

Thanks!

graveljp avatar Sep 26 '24 18:09 graveljp

This needs rebasing, but it's also not entirely clear to me this has support from WebKit. After double checking colleagues we see value in the CSS-based mechanism to specify filters, but not the XML-derived object model. We also gave this feedback last year in the corresponding issue.

annevk avatar Nov 12 '24 10:11 annevk

we see value in the CSS-based mechanism to specify filters, but not the XML-derived object model

This has been addressed already. The XML filter support was dropped in https://github.com/whatwg/html/pull/9537#issuecomment-2145148142, in favor of supporting only CSS filters. After receiving more feedback from WebKit, I then dropped the layer's filter parameter altogether, in favor of using the context filter. See https://github.com/whatwg/html/pull/9537#issuecomment-2369295644.

graveljp avatar Nov 12 '24 14:11 graveljp

My apologies, I was led to believe that feedback was still unaddressed. If you are willing to resolve the branch conflicts I can take care of review by Nov 22 (aiming for early next week). Hopefully @Kaiido can also make some time as they often have useful insights.

annevk avatar Nov 12 '24 15:11 annevk

I rebased the PR and added a few minor fixes.

graveljp avatar Nov 15 '24 00:11 graveljp