Apply shader to Sprite dynamically or permanently

[g-apparence]

What could be improved

Currently applying a shader directly to a Sprite could be easier. I see two different way

• apply a shader to a sprite permanently
• rendering a sprite using the sprite image

Why should this be improved

Shaders can help customizing the game experience.

• taking a hit
• changing colors of enemies
• creating a shadow
• creating effects
• ...

Any risks?

I don't see any. Just shaders stays non trivial.

More information

I already made a try using extension on a project. Heres what I made.

applyShader

• FragmentProgram is in parameters as you want to load them before the game launch
• This will replace permanently the image of the sprite
• Transforming a canvas to an image is not that fast (this is an async function). So we have to call this in the onLoad function of the sprite or animation group.

extension on Sprite {
  /// Apply permanently a shader to an image
  /// useful if you don't want to use the orginal image anymore
  Future<void> applyShader(FragmentProgram fragmentProgram) async {
    var widgetFrameSampler2D = ImageShader(
      image,
      TileMode.repeated,
      TileMode.repeated,
      Matrix4.identity().storage,
    );
    final shader = fragmentProgram.shader(
      floatUniforms: Float32List.fromList(
        <double>[
          image.width.toDouble(),
          image.height.toDouble(),
          1.0,
        ],
      ),
      samplerUniforms: [widgetFrameSampler2D],
    );
    var painter = Paint()..shader = shader;
    var pictureRecorder = PictureRecorder();
    var canvas = Canvas(pictureRecorder);
    canvas.drawRect(
      Rect.fromLTWH(0, 0, image.width.toDouble(), image.height.toDouble()),
      painter,
    );
    var picture = pictureRecorder.endRecording();
    var shaderedImage = await picture.toImage(image.width, image.height);

    image = shaderedImage;
  }
}

renderWithShader This method allows you to draw your shader using the image of the sprite as the input.

• call this method before or aftering the render method
• or replace the call of render
• cons : you have to take correctly the pixel according to the sequence if you are using an animation group

extension on Sprite {
  // apply a shader to a single render cycle
  void renderWithShader(
    FragmentProgram fragmentProgram,
    Canvas canvas, {
    Float32List? floatUniforms,
    Vector2? position,
    Vector2? size,
    Anchor anchor = Anchor.topLeft,
  }) {
    var widgetFrameSampler2D = ImageShader(
      image,
      TileMode.repeated,
      TileMode.repeated,
      Matrix4.identity().storage,
    );
    final shader = fragmentProgram.shader(
      floatUniforms: floatUniforms,
      samplerUniforms: [widgetFrameSampler2D],
    );
    var painter = Paint()..shader = shader;

    position ??= Vector2.zero();
    size ??= srcSize;
    position.setValues(
      position.x - (anchor.x * size.x),
      position.y - (anchor.y * size.y),
    );
    final drawRect = position.toPositionedRect(size);

    canvas.drawRect(drawRect, painter);
  }
}

With SpriteAnimationGroupComponent On SpriteAnimationGroupComponent this allows to apply shader to all sprites or just the current one

extension on SpriteAnimationGroupComponent {
  /// Apply permanently a shader to all sprites of the group
  Future<void> applyShader(
    SpriteAnimation animation,
    FragmentProgram fragmentProgram,
  ) async {
    for (int i = 0; i < animation.frames.length; i++) {
      await animation.frames[i].sprite.applyShader(fragmentProgram);
    }
  }

  // render a shader to a single render cycle
  void renderWithShader(
    FragmentProgram fragmentProgram,
    Canvas canvas, {
    Float32List? floatUniforms,
  }) {
    animation?.getSprite().renderWithShader(
          fragmentProgram,
          canvas,
          // position: position,
          size: size,
          floatUniforms: floatUniforms,
        );
  }
}

Example of calling renderWithShader in a Sprite This exemple shows how I ended calling it inside the render function of a SpriteAnimationGroupComponent.

@override
  void render(Canvas canvas) async {
    super.render(canvas); // <- we want to render the initial sprite

    var sprite = animation!.getSprite();
    var image = sprite.image;
    // this is all the params we needs to correctly pick the color of the image and manipulate it 
    var params = Float32List.fromList(
      <double>[
        image.width.toDouble(), // input image size
        image.height.toDouble(), // input image size
        size.x, // output drawing size
        size.y, // output drawing size
        animation!.currentIndex.toDouble(),  // sprite index drawing size
        sprite.srcSize.x, // textureSize
        sprite.srcSize.y, // textureSize
      ],
    );

    renderWithShader(
      _shadowShader,
      canvas,
      floatUniforms: params,
    );
  }

Hope this is understandable. Maybe there's already another way of doing this and I missed it. Good job working on Flame, that's an amazing package 👍. Let me know if that may interests you to include it in Flame

Gautier - 🤘

[spydon]

Sprite and SpriteComponent and the rest already has a paint field which it uses for rendering, couldn't you just set the shader there?

[ufrshubham]

The idea of having pre-shaded image is quite interesting.

For the other part, instead of having a separate render method, flame can just set the 0th texture as the current image of the component if a custom shader is set.

[spydon]

The idea of having pre-shaded image is quite interesting.

For the other part, instead of having a separate render method, flame can just set the 0th texture as the current image of the component if a custom shader is set.

Aaah, now I understand what this was about, missed that it only did the apply once.

[g-apparence]

Yes there is two features. I could have open two tickets but...

[st-pasha]

For the applyShader function, why do we want to apply the shader to Sprite, instead of to the underlying Image? Seems like the latter would be useful in more contexts.

[g-apparence]

@st-pasha That's exactly the idea. Apply shader here is just replacing the current image in the sprite. Also note that the canvas to image function is async so better doing while loading.

I've made a first version, before going further I wrote this issue. I suppose there are some improvements to do (like for animation groups and other types of sprites).