CEA-608 closed captions

[mangui]

It would be great to have support for embedded closed captions, specifically CEA-608.

For reference, here are details on JWPlayer's version: http://support.jwplayer.com/customer/portal/articles/1430278-cea-608-captions

[triwav]

Was just about to request this. You beat me to this. Either this or Apple WebVTT way would be great.

[hmatanock]

Vote up for this.

[mangui]

sample streams welcomed

[stjernstrom]

Hi

What kind of samples do you need. You can find one example of HLS captioning here:

http://www.cpcweb.com/webcasts/webcast_samples.htm#HLS http://www.cpcweb.com/webcasts/hls/cpcdemo.m3u8

[jlacivita]

Okay, i've done a ton of research on this.

When you're parsing an AVC PEC packet, there's an additional packet type for SCTE 128 signals, with a value of 6. These packages are known as "Supplemental Enhancement Information" or SEI.

These can be many things, including 608/708 caption data. To confirm it's 608/708 captions:

• first ignore the first 3 bytes, which is a static header
• next, read a one byte int, followed by a two byte int and a 4 byte int. If these are 0xB5, 0x0031, and 0x47413934 respectively, then it's either 608/708 captions data, or "bar" data (which i'm guessing is for letterboxed content)

to distinguish between CC_DATA and "bar" data, grab the next 1 byte. 3 is CCs, 6 is "bar"

If it's "3" then you need to determine how much CC data is in the SEI, to do this, you grab the next 8 bits, but zero out the first 3 bits, no matter what they are, and convert to an integer (i.e. you're just using 000 and the last 5 bits of the byte) - This value tells you how many CC s there are, which you can use to calculate the number of bytes needed: 2 + count * 3, since each cc item is 3 bytes, and there's two bytes of headers

From here, i'm pretty sure we just need to convert the cc_data bytes (not the rest of the bytes we had to parse before that) into Base64, and dispatch them via an onCaptionInfo event, with a payload of:

{ type: "708", data: base64StringOfBytes }

At this point, i believe we can simply leverage a 3rd party OSMF live CC plug-in.

I figured this out from: http://www.scte.org/documents/pdf/standards/ANSI_SCTE%20128%202010-a.pdf

Then you can test it out using Adobe's CC lib by downloading a free trial of Adobe Media Server, and looking for OSMFCCLib.swc. It works with HDS, and it parses the 608/708 stuff for you, as long as the onCaptionInfo events are fired from the stream.

I've got a modified version of TSDemuxer that does all of the parsing, but I can't figure out how to add / listen to the FLVTags, or if i'm creating them correctly. It's just a new frame.type section in _parseAVCPES, and a Base64 method from Flex.

What's the best way to review this with you?

[jlacivita]

I've got an experimental version that inserts onCaptionInfo calls into HLSNetStream.

They're being dispatched fine, and Adobe's OSMFCCLib kind of parses, them, but i'm seeing the text get a little garbled. I'm wondering if maybe they're being dispatched in the wrong order, because you can recognize the letters, but they're mixed up a bit.

https://github.com/jlacivita/flashls/tree/master

[jlacivita]

I've been testing with the m3u8 above: http://www.cpcweb.com/webcasts/hls/cpcdemo.m3u8

[mangui]

nice !

have you been able to sort out your garbled text issue ?

[jlacivita]

Not yet...

[jlacivita]

I'm taking another stab at this. It's going to require a base64 library. is there one you prefer to use?

[mangui]

hi @jlacivita great ! bloody crypto is already used in chromeless player and available in the repo

[jlacivita]

Okay, i'm pretty sure i'm inserting the 608 captions into the stream correctly. However, I suspect there's a timing issue with what order they come down in.

There's a few things I'm blocked on:

1. VideoFrames don't have any way to calculate their timecode... i need to add some offset to PES.pts so that all of the CCs from one PES packet aren't at the same timecode
2. I'm not sure if the order of the CCs in the encoded bytes is the correct order to put them into the HLSNetStream
3. I'm not sure how your FLVTags work, i'm sort of stealing the logic from onID3Data.

I am still seeing the garbled Captions, but they're appearing at about the right time, and the garbled letters do resemble the expected captions, they just in the wrong order or something.

The strange thing is that occasionally, if my CPU is stressed, the captions show up properly... as if missing frames somehow gets the onCaptionInfo messages sorted properly.

Any thoughts?

[mangui]

Hi @jlacivita , seeing the code would help. any pointer ?

[jlacivita]

http://demo.theplatform.com/pdk/flashls/TSDemuxer.as.zip

Forked about a week ago.

[mangui]

you should merge this commit : https://github.com/mangui/flashls/commit/a0d6aa676ae2a4018a50fb9cda4534c499638bb0, if you don't you will be subject to potential issues. also I am not clear about your PTS computation around videoframes. i don't think you need to offset them. regarding garbling, is it garbled only after netstream callbacking onCaptionInfo() ? or also while parsing ?

[jlacivita]

Hi I merged the commit, with no change. I also removed the timestamp computation, and just use pes.pts for all of the FLVTags.

It's hard to say if they are garbled during parsing, as i'm not parsing them completely, just enough to insert them into the stream. I suppose that is my next step...

[jlacivita]

Okay, new information.

The CC's don't start getting garbled until the entire M3U8 is buffered, so something is going wrong once buffering is done.

If i throttle my connection to prevent the buffering from completing before the subtitles, then they look perfect every time.

Any idea what code could be running at the end of buffering to break the onCaptionInfo messages?

[jlacivita]

:( that's not actually the case.

I do know that every once in a while, the captions start rendering properly without changing the code.

It's still a mystery as to why.

[bwalls]

Would love to see this working!

[dhassoun]

I second that in a big way ;)

On Fri, May 29, 2015 at 9:47 AM, bwalls [email protected] wrote:

Would love to see this working!

— Reply to this email directly or view it on GitHub https://github.com/mangui/flashls/issues/11#issuecomment-106853496.

*[ David Hassoun | Principal ] *[ Adobe Certified Master Instructor ] [ p. 303.872.0442 | c. 303.359.7466 | f. 303.433.4002 ] [ RealEyes Media, LLC | www.realeyes.com ] [ 940 Logan | Denver CO, 80203 ]

[triwav]

Not trying to get this off track but just curious what you guys are using get 608 captions into your feeds. I'm not aware of any open source solutions. We currently use FFmpeg and I know for sure it doesn't have this capability.

[bwalls]

I work for NASA. We do frequent live shows that are captioned in real time through external providers, and inserted into the video stream by something like an Evertz or EEG caption inserter. The content either goes to NASA TV (we have three streams that go our 24/7 - see http://www.nasa.gov/multimedia/nasatv/index.html, and to Ustream) or in some cases to other internal or external websites. The NASA TV content that's not live is played out from a broadcast-style playout server and sent to satellite, with embedded captioning support. The NASA TV streams treat it all as live content that we either grab just before it goes to satellite, or pulled off satellite and encoded. We are required to provide captioning by Section 508 of the Rehabilitation Act, and prefer to use the same method for Broadcast and Web delivery. Live events not going to NASA TV is where it would be great to have a free player that supports HLS or HLS in Flash and closed captions.

The live encoding typically uses hardware encoders with SDI input to maintain the metadata. So Elemental or Inlet encoders for NASA TV, TriCasters and various other gear in the field. I don't know of any software encoders that currently support the 608/708 metadata for live capture.

For produced content, Adobe Premier allows editing and embedding of captions. Other non-linear editors have solutions with plug-ins. If we're only delivering to the web, then WebVTT or SRT or some other sidecar is fine, but we often need to play that same content over internal cable systems, so again embedded is nice. And it's good to have the captions embedded when folks download the files. The downloader may not need or care about captions, but someone they send it to may.

I don't know of any open source solutions, either, at least nothing easy or reliable.

[jlacivita]

@mangui I'm not able to figure this one out on my own, but this is a very important feature for an HLS client. Live content is almost exclusively encoded with CEA 608/708, and it's often preserved when the content moves over to VOD. Since CEA 608/708 is part of the TS fragments, there's no easy way for code outside of flashls to decode and sync them. As a replacement for the

Any chance we could get this added to flashls in an upcoming release?

[kfeinUI]

Vote up. Would be a great help. Sounds like @jlacivita is close. Just needs a little support to get it performing smoothly.

[gyachuk]

I'd also love to see this integrated. I've merged jlacivita's code into my clone of your dev branch, and added an HLSEvent for CLOSED_CAPTION_INFO. I've checked that I'm getting the same Base64 encodings as I get from @jlacivita's branch.

@mangui, do you want a pull request, or do you prefer to take this from @jlacivita?

@jlacivita, what do we need to do with the encoded data block in order to see the actual Closed Captions? It looks like your code just currently logs the encoded string to the browser console.

Let me know if I can help.

[kfeinUI]

@ gyachuk It's a matter of integrating the captioning library from Adobe into your player. Quite easy as it's just a wiring job. It comes packaged with the Adobe Media Server. As long as you wire it up and fire off those events, it will do the job from there. On Jul 4, 2015 4:43 PM, "gyachuk" [email protected] wrote:

I'd also love to see this integrated. I've merged jlacivita's code into my clone of your dev branch, and added an HLSEvent for CLOSED_CAPTION_INFO. I've checked that I'm getting the same Base64 encodings as I get from @jlacivita https://github.com/jlacivita's branch.

@mangui https://github.com/mangui, do you want a pull request, or do you prefer to take this from @jlacivita https://github.com/jlacivita?

@jlacivita https://github.com/jlacivita, what do we need to do with the encoded data block in order to see the actual Closed Captions? It looks like your code just currently logs the encoded string to the browser console.

Let me know if I can help.

— Reply to this email directly or view it on GitHub https://github.com/mangui/flashls/issues/11#issuecomment-118553405.

[mangui]

@gyachuk plz submit a PR, it will be easier to review, tks Cheers, Mangui

[gyachuk]

Thanks @kfeinUI. I see that there is already an OSMF.swc in flashls/lib/osmf. Is that the one to use? I've also downloaded Adobe Media Server and have found OSMFCCLib.swc down in the Linux samples folder. Perhaps that's the right one to use? Haven't found much useful documentation on how to use it (in the context of flashls). Any pointers would be greatly appreciated. Thanks.

[gyachuk]

Also, I realize that this might not be exactly the right forum for this, since I'd most likely be using it in the host application (mediaelements).

[jlacivita]

Since this page is the place for all things related to flashls + CEA-608, here's a link to my pull request, focussed entirely on this issue:

https://github.com/mangui/flashls/pull/334

I've wired up onCaptionInfo events, and the OSMFCCLib renders them, but it seems that some of the characters are missing and i'm stuck as to why. I need help to get this working before we can actually do the pull.