Packet timestamp mismatch on a calculator receiving from stream "free_memory"

[spennihana]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

MacOS

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

No response

Programming Language and version

Javascript

MediaPipe version

0.10.3

Bazel version

No response

Solution

ImageSegmenter

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

I am receiving user reports with this error, but am unable to reproduce this locally. Any help with debug steps would be much appreciated!

We are using the `segmentForVideo` API with `runningMode="VIDEO"`. We pass in a `performance.now()` for the timestamp.

Here's the error below: 


Adding Image to stream image_in was
not ok: INVALID ARGUMENT: Graph has errors:
Packet timestamp mismatch on a calculator receiving from stream
"free memory". Current minimum expected timestamp is 43987475001 but
received 43987475000. Are you using a custom InputStreamHandler? Note that
some InputStreamHandlers allow timestamps that are not strictly
monotonically increasing. See for example the ImmediateInputStreamHandler
class comment.
[type.googleapis.com/mediapipe.StatusList='\n\xed\x02\x08\x03\x12\xe8\x02Pac
ket timestamp mismatch on a calculator receiving from stream
\"free memory\". Current minimum expected timestamp is 43987475001 but
received 43987475000. Are you using a custom InputStreamHandler? Note that
some InputStreamHandlers allow timestamps that are not strictly
monotonically increasing. See for example the ImmediateInputStreamHandler
class comment.']
=== Source Location Trace:
===
third_party/mediapipe/framework/input stream manager.cc:139

I'm noticing that each error is always 1 off. Here are several more errors:

 Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 20937247001 but received 20937247000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 20937247001 but received 20937247000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 07, 2023 6:28:45
 Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 5084001 but received 5084000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 5084001 but received 5084000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 07, 2023 2:51:12
 Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 60499001 but received 60499000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 60499001 but received 60499000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 07, 2023 2:15:30
Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 2672365001 but received 2672365000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 2672365001 but received 2672365000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 07, 2023 1:48:32
 Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 65295527001 but received 65295527000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 65295527001 but received 65295527000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 07, 2023 1:18:48
 Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 1015559001 but received 1015559000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 1015559001 but received 1015559000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 06, 2023 23:22:19
 Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 2979851001 but received 2979851000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 2979851001 but received 2979851000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 06, 2023 22:58:36
Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 30420462001 but received 30420462000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 30420462001 but received 30420462000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 06, 2023 22:01:17
 Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 43987475001 but received 43987475000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok, CalculatorGraph::Run() failed in Run: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 43987475001 but received 43987475000. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed
September 06, 2023 18:29:25
 Graph has errors: Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 95374931001 but received 95374931000. Are you using a custom InputStreamHandler? Note that some InputStre

### Describe the expected behaviour

```shell
Help in debugging and/or work arounds would be great (e.g. would the `segment` API with `runningMode="IMAGE"` have the same problem potentially with the `FREE_MEMORY_STREAM`?). 

Any help or hints would be great thanks.

Standalone code/steps you may have used to try to get what you need

I don't have a repro yet unfortunately. Just these logs

Other info / Complete Logs

No response

[kuaashish]

Similar to issue #4654

[kuaashish]

Hi @spennihana,

Could you kindly confirm whether you are still seeking a resolution for this issue? According to the comment provided here, it appears that the issue has been resolved on your end.

If there are no longer any issues from your perspective, may we proceed to close the matter and mark it as resolved?

Thank you!

[alankent]

I am seeing the same problem. The other ticket said they resolved it by using IMAGE mode instead of VIDEO mode, but when I do that it tells me I must use VIDEO mode (I am sending it a webcam stream). I can get a frame or two at times. But I am getting timestamps with values such as 0 and 1.

If I had to guess, the video object .currentTime method is being used, but it is in seconds (not ms). The values are different, (e.g. 1.53 and 1.59) but after truncating to an int the difference is less than 1. (This is based on a call seeming to work if the previous time was 0.93 -> 1.32, but then the next fails with 1.32 -> 1.75.) This is just conjecture however. I have tried only calling once every second and it still fails, so this might be a red herring.

Unhandled Runtime Error
Error: Encountered multiple errors: Adding proto packet of type mediapipe.NormalizedRect to stream norm_rect was not ok: $Graph has errors: 
Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 1 but received 0. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment., WaitUntilIdle failed: $CalculatorGraph::Run() failed in Run: 
Packet timestamp mismatch on a calculator receiving from stream "free_memory". Current minimum expected timestamp is 1 but received 0. Are you using a custom InputStreamHandler? Note that some InputStreamHandlers allow timestamps that are not strictly monotonically increasing. See for example the ImmediateInputStreamHandler class comment.; WaitUntilIdle failed

Note printing out the timestamp gives big numbers, not 0 and 1. So confusing, but going to try some more ideas later and try to find some other sample code. (I really want to try the holistic detector - I want arms, fingers, and facial expressions. https://developers.google.com/mediapipe/solutions/vision/holistic_landmarker says "coming soon".)

[galharth]

Any update?

[kolorfilm]

Have the same issue for a while now. Can confirm it happens on Android using mediapipe in the browser (JS web application). Seeing this on our Sentry dashboard a few times for a few users. Any update?