Not able to grab video with macOS 12.6.6

[aikkala]

Thanks for the awesome library!

I've been using the library on Windows successfully, but when running it on macOS (12.6.6) there seem to be some problems with grabbing the video. If I run the client_stream_pv.py script, I get the following error message:

Pose at time 10668188876
[[-0.9422886   0.07307079 -0.3267378   0.        ]
 [ 0.00987262  0.9815364   0.19103575  0.        ]
 [ 0.3346641   0.17678398 -0.92560947  0.        ]
 [-0.04399353  0.08268791  0.14916018  1.        ]]
Focal length: [1459.3796 1460.1498]
Principal point: [941.4674 512.4374]
Traceback (most recent call last):
  File "/Users/aikkala/Workspace/hl2ss/viewer/client_stream_pv.py", line 89, in <module>
    cv2.imshow('Video', data.payload.image)
cv2.error: OpenCV(4.8.0) /Users/xperience/GHA-OpenCV-Python/_work/opencv-python/opencv-python/opencv/modules/highgui/src/window.cpp:971: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'imshow'

Has anyone encountered this sort of an issue before? Any suggestions how to approach this?

I think there's a chance this isn't related to your library, but is rather some other macOS issue. If I try to use the "Live preview" feature in Windows Device Portal, the video doesn't render either, and I get an error message saying

Live preview stalled due to low bandwidth or loss of connectivity. If this problem persists, consider using a lower quality stream.

This happens even with the lowest video resolution. I can, however, record a video and take a photo without problems.

[aikkala]

And another, perhaps more informative, example. When I run sample_video.py script, I get the following error message

Invalid NAL unit 4, skipping.
PPS id out of range: 1
Invalid NAL unit 4, skipping.
PPS id out of range: 1
Error parsing NAL unit #2.
Process _source-2:
Traceback (most recent call last):
  File "/Users/aikkala/miniconda3/envs/hl2ss/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/aikkala/Workspace/hl2ss/viewer/hl2ss_mp.py", line 95, in run
    self._source.open()
  File "/Users/aikkala/Workspace/hl2ss/viewer/hl2ss.py", line 1276, in open
    self.get_next_packet()
  File "/Users/aikkala/Workspace/hl2ss/viewer/hl2ss.py", line 1280, in get_next_packet
    data.payload = self._codec.decode(data.payload)
  File "/Users/aikkala/Workspace/hl2ss/viewer/hl2ss.py", line 814, in decode
    for frame in self._codec.decode(packet):
  File "av/codec/context.pyx", line 509, in av.codec.context.CodecContext.decode
  File "av/codec/context.pyx", line 524, in av.codec.context.CodecContext.decode
  File "av/codec/context.pyx", line 427, in av.codec.context.CodecContext._send_packet_and_recv
  File "av/error.pyx", line 336, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input; last error log: [hevc] PPS id out of range: 1

[jdibenes]

Hello, These errors occur when the server cannot transfer the compressed frames fast enough (low bandwidth/poor connectivity for example). You can try adding a check before the call to cv2.imshow in client_stream_pv.py:

if (data.payload.image is not None and data.payload.image.size > 0):
    cv2.imshow('Video', data.payload.image)

and see if you get any video. It might be a macOS issue as you mentioned but I don't have access to a mac to test. Also, we've received reports of connectivity issues being resolved after updating the HoloLens OS so maybe that could work.

[aikkala]

Hi,

Thanks for the suggestions. It seems like data.payload.image is always None, and hence nothing is shown. I also checked that I have the most recent HoloLens update 22621.1125. I'll continue to look into this, and report here if I make any progress.