Preserving relationship of ancillary data with video frames

[MathieuDuponchelle]

trafficstars

Hey, I'm looking at making use of the ancillary data support, but I am a bit perplexed by the fact that ancillary data is an entirely different payload type, to be sent over a separate endpoint.

Many ancillary mapping mechanisms, such as SMPTE 2010 (Mapping of ANSI/SCTE 104 Messages), require a strict relationship between a given video frame and its ancillary data (eg a single SCTE 104 message should be associated to one frame with that scheme).

As far as I understand, requiring ancillary data to be sent and received through a separate endpoint severs that relationship, am I missing something? How do you recommend a receiver re-associates each video frame with its respective ancillary data?

[MathieuDuponchelle]

ping :)

[mhhen]

I apologize for not getting back to you earlier. As you stated, the payload types for video, audio and ancillary data are different. However, each type can be sent and received through the same endpoint. The origination_ptp_timestamp value can be set by the transmitter and then used by the receiver to re-associate video and ancillary data payloads. Please let us know if you have additional questions.

[MathieuDuponchelle]

Thanks for replying :) I do have further questions:

• How would a receiver know that a given ancillary stream relates to a given video stream? Can't any number of video streams be transmitted through the same endpoint?

• If re-associating using the origination_ptp_timestamp, does the receiver need to introduce latency? If so, how much?

[mhhen]

There is a stream_identifer that is associated with each payload and is set by the transmitter. This value can then be used by the receiver to re-associate the video and ancillary payloads.

There is no recommended latency requirement for a receiver in order to re-associate payloads. Since an ancillary data payload is typically small, it could be placed in the transmit queue just prior to its associated video frame so it is available on the receive when the video payload arrives.

[MathieuDuponchelle]

There is a stream_identifer that is associated with each payload and is set by the transmitter. This value can then be used by the receiver to re-associate the video and ancillary payloads.

Right, but that would require information to be transmitted out-of-band wouldn't it? Shouldn't the mapping information be made available through the API, so that any receiver can talk to any (third party) transmitter?

There is no recommended latency requirement for a receiver in order to re-associate payloads. Since an ancillary data payload is typically small, it could be placed in the transmit queue just prior to its associated video frame so it is available on the receive when the video payload arrives.

Again, wouldn't this mean the receiver needs to rely on a particular, unspecified behavior from the transmitter?

[mhhen]

The goal of the AVM API is to provide interoperability between transmitters and receivers. We are open to suggestions, especially concerning interoperability. Using the existing API, another suggestion would be to provide related video, audio and metadata on a single endpoint. Overall latency is very low, so ordering of related payloads should not be a concern. It was intended as a possible implementation suggestion. Please continue to provide feedback, suggestions and possible solutions in this area.

[MathieuDuponchelle]

I see OK. Well my suggestion would be to transmit ancillary data tightly coupled with the video data it is associated with, and to preserve that coupling for the receiver side.

[MathieuDuponchelle]

@mhhen to be clear, is order of delivery of payload data guaranteed?

[mhhen]

Correct. The order of delivery for payloads matches the order the calls to send payloads were made for an endpoint. The payloads are placed into a queue and are then processed here by a transmit thread.

A proposal concerning the coupling of the video and ancillary data API would be great! Or even better, submit in the form of a pull-request.

[MathieuDuponchelle]

Correct. The order of delivery for payloads matches the order the calls to send payloads were made for an endpoint. The payloads are placed into a queue and are then processed here by a transmit thread.

Right, that's transmission side, but is the order guaranteed on receiver side?

[MathieuDuponchelle]

A proposal concerning the coupling of the video and ancillary data API would be great! Or even better, submit in the form of a pull-request.

That's a bit above my paygrade I'm afraid, I've only been tasked with writing GStreamer wrappers around this API, not to extend / amend it :)

[mhhen]

Payloads that fit into a single adapter level packet can be received out-of-order. However, in actual practice, video payloads (require hundreds of adapter level packets) are mixed with with ancillary data payloads, they would be received in the order sent.

Please continue to pass on suggestions and improvements as your work progresses.

[MathieuDuponchelle]

OK, I suppose this guarantee will have to do. Note that another one of my suggestions still stands @ https://github.com/aws/aws-cdi-sdk/issues/95 , the ability to instantiate any number of adapters would make things immensely easier for me.

Also for ancillary data figuring out how to size the adapter is going to be another question I'd rather not have to solve :)