Support mirrored video for createCapture

[ziyuan-linn]

Increasing Access

For many creative coding projects that involve the webcam, we want to display the horizontally flipped webcam footage rather than the original footage. The flipped footage resembles a mirror and therefore provides a more natural interaction. This feature would give people with little or no prior experience in creative coding an intuitive way to mirror the webcam footage.

Most appropriate sub-area of p5.js?

• [ ] Accessibility
• [ ] Color
• [ ] Core/Environment/Rendering
• [ ] Data
• [X] DOM
• [ ] Events
• [ ] Image
• [ ] IO
• [ ] Math
• [ ] Typography
• [ ] Utilities
• [ ] WebGL
• [ ] Build Process
• [ ] Unit Testing
• [ ] Internalization
• [ ] Friendly Errors
• [ ] Other (specify if possible)

Feature request details

This feature would give users an option to mirror the webcam video. Some potential interfaces :

function setup() {
  capture = createCapture(VIDEO);
  mirroredcCpture = mirrorCapture(capture);
}

or alternatively

function setup() {
  mirroedCapture = createCapture(VIDEO, { flipped: true });
}

Unlike using functions like scale(-1, 1) which doesn't change the original video element, this feature would return a new video element. This would be easier to use than scale, which often results in things getting drawn to unexpected places and text being inverted. This feature would especially be helpful to the ml5 community since we could run prediction directly on the mirrored video element rather than trying to manually mirror both the video and the outputted data.

Here is a link to the discussion about this feature in the ml5 repo.

@shiffman Please feel free to add anything I missed :}

[welcome[bot]]

Welcome! 👋 Thanks for opening your first issue here! And to ensure the community is able to respond to your issue, please make sure to fill out the inputs in the issue forms. Thank you!

[Qianqianye]

Thanks @ziyuan-linn for brining up this issue. I'm inviting the p5.js DOM stewards to this discussion to share you thoughts or questions about feature request @SarveshLimaye, @SoundaryaKoutharapu, @ramya202000, @BamaCharanChhandogi, @Obi-Engine10, @MarceloGoncalves, @hiddenenigma. Thank you!

[limzykenneth]

Just want to bring up from the discussion in ml5 here as well, the current hacky but efficient way to deal with this is through drawing the capture feed onto a p5.Graphics and flipping it with scale(1, -1). For internal implementation we need something at least as performant and not operating on the pixels array. I'm not sure if there are any browser API already available that can help with this but it's worth looking into.

[davepagurek]

I believe createCapture() returns a p5.MediaElement. For these, we already internally draw them to a canvas in _ensureCanvas and then use that when rendering them: https://github.com/processing/p5.js/blob/95b82ea1d5a6264f8bb9acbc71b6e49104ac9522/src/dom/dom.js#L3222

So potentially we could do the flipping in there when we draw it, which wouldn't be much additional cost?

[shiffman]

I believe createCapture() returns a p5.MediaElement. For these, we already internally draw them to a canvas in _ensureCanvas and then use that when rendering them So potentially we could do the flipping in there when we draw it, which wouldn't be much additional cost?

I think this makes a lot of sense @davepagurek! We'd love to help with this from the ml5.js side. We could potentially:

1. Implement a proof of concept in ml5.js itself and then if it works well, we could bring it over to p5.js natively?
2. Open a branch in p5.js directly and work on it there first?

Do you all have a preference?

[davepagurek]

@shiffman I imagine it might be a little less code to do it directly in p5 as you would be able to augment the existing class, but either works if it'll be more familiar/easier doing it externally first! I'm happy to give pointers if you run into any trouble either way 🙂

[Vishal2002]

I want to know more about this feature from @ziyuan-linn, can you elaborate more on this if possible ?

[Vishal2002]

Is it still open to contribute if yes then I have done what said.

[shiffman]

Wow, super excited to see this get merged, thank you @Vishal2002! This will really help with the ml5.js examples!

[Vishal2002]

Wow, super excited to see this get merged, thank you @Vishal2002! This will really help with the ml5.js examples!

Thank you @shiffman It's my pleasure to help the community.

[limzykenneth]

This is released now with 1.9.1

[MarcoHess]

I have a question on how this is now supposed to work with the machine vision. I was trying some experiments using a webcam and poseNet but if I follow the example from the documentation with const flippedImage = ml5.flipImage(input); in the draw() function, yes I can flip the image but the poseNet still uses the original video stream to perform the pose detection and as such what I now draw on the canvas is the non mirrored pose detection on the mirrored image.

What would I need to do to input the mirrored video stream (as opposed to just a single image) into the poseNet pose detection?

function setup() {
  createCanvas(canvas_width, canvas_height);
  video = createCapture(VIDEO);
  video.size(width, height);
  // How do I mirror the video here to supply to poseNet?
  poseNet = ml5.poseNet(video, modelReady);
  poseNet.on("pose", function (results) {
    poses = results;
  });
  video.hide();
}

[shiffman]

Hi @MarcoHess, this is being addressed in the next version of ml5.js, due to be released soon, follow along here: https://github.com/ml5js/ml5-next-gen/pull/127