PyAV Support data streams with NULL codec.

Spurred from #353, and others, it would be great if PyAV would support data streams. Those streams appear to be ones with no codec.

Since PyAV is built generally assuming that there is a codec per stream, many things break or even segfault as we start adding the support. It shouldn't be "hard", but it is going to be a big pass over all of the container code to add a ton of if codec checks.

Sep 06 '18 12:09 mikeboers

One problem I encountered while trying to add support for subtitle encoding is that our encode() methods explicitly expect a subclass of Frame, and SubtitleSet doesn't inherit Frame. This is a shame, because it makes encode and decode asymetrical.

Oct 15 '18 11:10 jlaine

What kind of "support" are we talking about? Can we do much more than muxing / demuxing packets?

Nov 07 '18 15:11 jlaine

Thats it. Just packets.

Personal use case: I want to have a data-channel in my security camera footage with per-keyframe real-world-clock time data because the timestamps are a bit wonky.

Nov 07 '18 17:11 mikeboers

Or, even more hilariously... Python defined codecs.

Nov 07 '18 17:11 mikeboers

OK, how do we go about testing this? I mean : testing in a way that ensures more than just PyAV round-tripping?

Nov 07 '18 18:11 jlaine

Maybe use the ffmpeg cli to embed/extra data streams and assert they made it.

Maybe like https://superuser.com/questions/1243257/save-data-stream

Nov 07 '18 21:11 mikeboers

I could seriously use this. I am currently working on hacking around and getting KLV data stream functionality into PyAv and it is very difficult without it.

Nov 27 '18 22:11 samuel-davis

If either of you could tell me a workaround to easily get the video and data packet for the same time it would be extremely helpful.

Ie: Image Data from Frame 1 with KLV Data associated with Frame 1.

I managed to get the KLV data out of the data frame by doing this

    if packet.stream.type == 'data':
        bytes = packet.to_bytes()

        key, value = decode(bytes, 16)
        print('KLV Key = {}'.format(key))
        print('KLV Value = {}'.format(value))

Although I think I just smacked myself in the head and I may have it now.

Nov 27 '18 22:11 samuel-davis

Id love something like what is in the scratchpad directory for decode.

Where after obtaining a frame I can then check the type of the packet and do something dependant on what it is.

Nov 27 '18 23:11 samuel-davis

@samuel-davis Looks like you have your work for a first PR cut out then!

Nov 28 '18 07:11 jlaine

Hi! I came across the Null codec problem while attempting to probe gopro camera videos. The video has an audio and a video stream, as well as 3x data streams, which I can not access. I suspect they may be holding accelerometer data etc. I would love to see the ability to read such streams. I am considering storing data in such streams as well. Is there an alternate way of reading this data?

Oct 02 '19 21:10 jakams

We've not made progress on this.

I think you can use the ffmpeg CLI to save data streams, but I haven't figured out how to use it to embed data streams.

I would welcome a small example if your footage with many data streams to see what it looks like.

Oct 03 '19 14:10 mikeboers

Thanks. You are right, I found some relevant work regarding gopro data at https://github.com/stilldavid/gopro-utils and managed to get to the stream using a command from the link:

ffmpeg -y -i GOPR0001.MP4 -codec copy -map 0:3 -f rawvideo out-0001.bin

I would have preferred a direct stream, but this will do for me. Though I don't know about embedding either... I am no expert when it comes to ffmpeg :/

Data streams can be quite interesting, as they can store GPS data, gyro, accelerometer and other sensor data. I see potential here for the future.

At the moment I don't have any small video examples. I will look into procuring one and get back to you.

Oct 03 '19 17:10 jakams

As promised, a small 2 second video example:

gopro7_sample.zip

Oct 11 '19 14:10 jakams

@jakams Thanks for the sample.

Playing around with the current HEAD, I can demux the data packets and get their content. I don't know how we got to this point, but you can get the raw data:

>>> packet.to_bytes()
b'GPRO@\x02\x00\x00HD7.01.01.70.00LAJ8080132102288\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xc0(\xa7\xd2\xd3!\x9a\x04\x806k\x8f\x03\x02\x068C3281325203581\x00HERO7 Black\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0c\x000\xce\x84A\x10\x00\xc0(\xa7\xd2\xd3!\x9a\x04\x806k\x8f\x03\x02\x068\x97\xa2\xe3\x07M\x00\x01\x00\x00\x00\xb5A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00HD7.01.01.70.00LAJ8080132102288\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xc0(\xa7\xd2\xd3!\x9a\x04\x806k\x8f\x03\x02\x068C3281325203581\x00HERO7 Black\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xc0(\xa7\xd2\xd3!\x9a\x04\x806k\x8f\x03\x02\x068\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0c\x00UY\x00\x00\x00\x00NYAUTO\x00\x00\x00\x00HIGH\x00\x00GOPRO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x0c\x00\x00d\x00\x00\x000.0\x00\x00\x002_1SEC\x00\x00LNNYHS EIS\x00\x00NOFF\x00\x00\x00\x00\x00AUTO\x00\x00\x00\x00ASK\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x90_\x01\x00\x80\x07\x00\x008\x04\x00\x00\x01\x00\x00\x00\x80\x07\x00\x008\x04\x00\x00\x01\x00\x00\x00\x80\xbb\x00\x00d\x00\x00\x00\x18\x00\x00\x00\x02\x00\x00\x00\x80\xbb\x00\x009\xb8\x9f]\xb7C\x01\x00\xd4\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

Oct 15 '19 15:10 mikeboers

Im curious where this issue stands -

I have 2 kinds of data tracks I need to read and write - firstly - time code data tracks from Air ready client masters (these are masters in Pro Res 422 HQ, up to 16 discrete tracks of mono audio, and time code track).

Second, optional, is a QuickTime boxed timed metadata format track for my own timed metadata format that is proprietary.

it appears I cant make data streams from a template due to a no codec error (I'll file a separate bug) - im just generally curious if I can make my own packet and write to a custom data track and ensure its read as a complaint time code track.

Thanks!

Here is a sample output of a test file (this is not the exact master, but the track layout is the same - and yes im aware the discrete audio tracks have the same layouts/channels, this appears to be how HBO saves their masters - dont ask)


Video Index:  12
Video ID:  13
Video Profile:  LT
Video Format: yuv422p10le
Video Resolution: 854 x 480
Video Framerate: 24000/1001
Video Metadata: {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'und', 'handler_name': 'Core Media Video', 'encoder': 'Apple ProRes 422 LT', 'timecode': '01:12:20:06'}
---
Data Index: 13
Data ID: 14
Data Profile: None
Data Format: 
Data Metadata: {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'und', 'handler_name': 'Core Media Time Code', 'timecode': '01:12:20:06'}
---
Audio Index:  0
Audio ID:  1
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  1
Audio ID:  2
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  2
Audio ID:  3
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  3
Audio ID:  4
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  4
Audio ID:  5
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  5
Audio ID:  6
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  6
Audio ID:  7
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  7
Audio ID:  8
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  8
Audio ID:  9
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  9
Audio ID:  10
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  10
Audio ID:  11
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000
---
Audio Index:  11
Audio ID:  12
Audio Profile:  None
Audio Format:  s16
Audio Layout:  mono
Audio Metadata:  {'creation_time': '2020-09-16T21:53:24.000000Z', 'language': 'eng', 'handler_name': 'Core Media Audio'}
Audio Channel:  front center
Audio Sample Rate:  48000

Sep 20 '20 00:09 vade

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Mar 28 '22 02:03 github-actions[bot]

I hadn't notice wehad samples for this, let's keep it alive.

Mar 28 '22 05:03 jlaine

I, too, am looking to write out a data stream. Even in the broader AVcodec ecosystem, it seems surprisingly challenging to write streams other than video, audio, and subtitles.

Jun 03 '22 00:06 xkortex

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Nov 21 '22 03:11 github-actions[bot]