jump method is getting stuck

[MCPepsh]

I want to process a video frame by frame, but the jump(float where) method is either just taking ages or stuck in some kind of infinite loop or something like that. I used the Frames.pde example and just tweaked it a little for my needs. Another problem is that mov.frameRate is -1 most of the time. I think both problems are caused by the fact that the movie just started playing very short before, but I might be wrong. Am I doing something wrong or is this a bug?

I just changed very little:

/**
 * Frames 
 * by Andres Colubri. 
 * 
 * Moves through the video one frame at the time by using the
 * arrow keys. It estimates the frame counts using the framerate
 * of the movie file, so it might not be exact in some cases.
 */

import processing.video.*;

Movie mov;
int newFrame = 0;
float framerate = 30;

void setup() {
  size(1920, 1080);
  background(0);
  
  // Load and set the video to play. Setting the video 
  // in play mode is needed so at least one frame is read
  // and we can get duration, size and other information from
  // the video stream. 
  mov = new Movie(this, "video.mp4");
  
  // Pausing the video at the first frame. 
  setFrame(0);
}

void movieEvent(Movie m) {
  m.read();
}

void draw() {
  background(255);
  image(mov, 0, 0, width, height);
  saveFrame("frames/frame" + nf(getFrame(), 3) + ".png");
  fill(127);
  text(getFrame() + " / " + (getLength() - 1), 10, 30);
  newFrame++;
  setFrame(newFrame);
}

int getFrame() {    
  return ceil(mov.time() * 30) - 1;
}

void setFrame(int n) {
  mov.play();

  // The duration of a single frame:
  float frameDuration = 1.0 / framerate;

  // We move to the middle of the frame by adding 0.5:
  float where = (n + 0.5) * frameDuration;

  // Taking into account border effects:
  float diff = mov.duration() - where;
  if (diff < 0) {
    where += diff - 0.25 * frameDuration;
  }
  
  // The Program runs until here
  mov.jump(where);
  // But never executes the following code
  mov.pause();
}  

int getLength() {
  return int(mov.duration() * framerate);
}

@shiffman

[codeanticode]

@MCPepsh Try the latest beta, it fixes errors in the implementation of the time() and duration() functions that may be related to this issue.

[neilcsmith-net]

@codeanticode there are multiple issues with the current implementation of jump(), and the example above, that were discussed with Gohai a while back.

Mainly, I highly recommend moving all GStreamer calls off the rendering thread and on to the GStreamer event thread using Gst.invokeLater() or the executor. Calling getState() off of the event thread, and definitely on the renderer thread, is problematic. You really need a way inside draw() to wait for a frame to be received to make the above code work.

The AppSink.NEW_PREROLL implementation needs fleshing out - then the calls to play() and pause() in that example can be removed.

[codeanticode]

@neilcsmith-net thanks for the comments. Could you point to some code demonstrating the correct use of Gst.Invokelater(). PRs are also appreciated :-)

[neilcsmith-net]

Feel free to adapt anything from https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java It's under the same license.

async() is just a shorthand wrapper around Gst,invokeLater() (or the executor, which amounts to the same thing).

In particular, equivalent of jump() is at https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java#L171 and in handleSeek() - same seek code used in a few places.

[neilcsmith-net]

And also https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/PImageSink.java which abstracts out the AppSink callbacks so they can be used in capture and video player.

[codeanticode]

@neilcsmith-net could you expand on "The AppSink.NEW_PREROLL implementation needs fleshing out - then the calls to play() and pause() in that example can be removed." Thank you!

[neilcsmith-net]

@codeanticode seeking (jump) while in paused state should work if you update and link in AppSink.NEWROLL to do basically the same as the new sample callback. That way, you don't have to do the quick play and pause again, which may cause problems and leave the seek in the wrong place anyway. Possibly better not to use the accurate seek flag here, or at least make it optional, too.

[codeanticode]

Implemented with https://github.com/processing/processing-video/pull/122/commits/a95fcb7aa1aa9d5101f0a6921765b80c50cd981d

[codeanticode]

Beta 4 puts the seek operations in jump() in the GStreamer event thread, but not the play/stop/pause calls. All examples seem to work fine on Mac, but Scratch, which makes heavy use of jump and pause, sometimes (not always) hangs on Windows and Linux. Reopening, will revise in preparation for the v2.0 stable release.

[hamoid]

I tried in Ubuntu 19.04. The examples scratch, reverse and frames don't seem to work properly.

scratch gets stuck after a second or two.

reverse gives me a black screen.

frames is initially black. Nothing seems to happen by pressing RIGHT a few times. If I hold it down, the video eventually shows up. But the frame counter is a bit wonky. It often shows -1. Sometimes seeking is very slow, so I see the counter change but not the frames. This is not due to the frames being repeated as they are, but I seek 10 or 20 frames forward and the image doesn't change.

For this third example, what about having different background and text color? (both are 0). Also, what about reading mov.frameRate and mov.duration() once inside setup? In getFrame() the frameRate seems to be hard coded to 30.

In the console I see Processing video library using GStreamer 1.15.90

[neilcsmith-net]

Beta 4 puts the seek operations in jump() in the GStreamer event thread, but not the play/stop/pause calls. All examples seem to work fine on Mac, but Scratch, which makes heavy use of jump and pause,

If you're going to do this, you'll need to move play/stop/pause into the GStreamer event thread, or in that example you'll end up with a thread race where you're probably mostly doing play/pause/jump

Maybe also take the play/pause/stop implementations from https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java That class is well battle tested for this particular use!

You need to fully implement the New Preroll listener to extract frames, which will mean no need to call play then jump then pause in the example at all - just jump in paused state. Those constant state changes are a possible issue in themselves.

Be a little wary of calling getState(), at least on the Processing thread - maybe use the version with a timeout, and I'm not sure you need to in play / pause at all? Mind you, I am where the rate isn't 1, although it is in the GStreamer thread.

Consider a short time out getState() in seek, making sure it's in the GStreamer event thread (like at https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java#L285 ) - it's a little protection against swamping GStreamer with too many seek events while it's in the middle of a seek.

In the console I see Processing video library using GStreamer 1.15.90

Is that accurate or a bug in our version reporting? It's a non-stable GStreamer release.

[codeanticode]

Cool thanks, I will look into Praxis' thread handling and see how I can adopt it into the video library.

As for the version reporting, the library is using Gst.getVersion()

[codeanticode]

@hamoid do you still get playback issues using the latest version of the video library (2.2.x) on Linux?

[hamoid]

Hi, I do still get issues. I just downloaded version 2.2.2 and placed it into the libraries folder. After restarting Processing 4.1.1 the examples are shown under contributed libraries (not as a built in library as before).

• Scratch works for a while, then no longer updates.
• Reverse shows a still image.
• Frames works, although if I leave the arrow keys pressed for long enough it stops updating the counter and the video. It comes back when I release those keys. At the ends the frame count can display frame number "-1".

This is with an i9 cpu, mobile 3080 gpu, up-to-date Manjaro (Arch Linux variant).