HLS-FMP4 DTS:X Profile 2 Streams are missing first 2 seconds during Tunnel mode (Direct) Playback due to timestampAdjuster in FragmentedMp4Extractor.java

[cedricxperi]

Version

Media3 main branch

More version details

Devices that reproduce the issue

All Android TV with DTSX Decoder. E.g. Xiaomi Q2 55"

Devices that do not reproduce the issue

None.

Reproducible in the demo app?

Yes

Reproduction steps

• Add the following URL to media.exolist.json: https://secured.streamdts.com/dtsx/dtsx/hls/fmp4/callout/index.m3u8
• Enable Tunnel mode
• Playback the stream and observe that the first 2 seconds are skipped.
• Disable Tunnel mode
• Playback the stream and observe that playback is now ok.
• Playback is also ok with the follow DASH stream in tunnel mode. And I have verified that for DASH, there is no timestamp adjustments that are going on: https://secured.streamdts.com/dtsx/dtsx/dash/callout/index.mpd

NOTES: HLS-FMP4 DTS:X Profile 2 Streams are missing first 2 seconds of audio during Tunnel mode (Direct) Playback due to timestampAdjuster in FragmentedMp4Extractor.java.

• This only happens for Tunnel mode playback with DTS:X audio playing back via Direct Passthrough.
• This does not happen when playing back through MediaCodec DTSX Decoder.
• When the following lines (around line 1538) in FragmentedMp4Extractor.java are commented, the playback seems ok. if (timestampAdjuster != null) { sampleTimeUs = timestampAdjuster.adjustSampleTimestamp(sampleTimeUs); }
• FYI, DTS:X Profile 2 audio codec has a sync and non-sync frame concept. So once the first sync frame is lost, the whole section of audio cannot be decoded properly until the next sync frame can be located. In this stream, the sync interval is 2 seconds. It correlates with the 2 seconds lost in the beginning.
• What does timestampAdjuster do exactly? When are these adjustments to the presentation time stamps required?
• Is the presentation time stamp adjustments necessary when playing back via Direct Passthrough?
• How can we resolve this issue more gracefully?

Thanks!

Expected result

Playback does not skip the first 2 seconds.

Actual result

Playback skipped the first 2 seconds.

Media

HLS-fMP4 stream. First 2 seconds lost during playback in tunnel mode: DTS:X 5.1.4 - HLS fMP4: https://secured.streamdts.com/dtsx/dtsx/hls/fmp4/callout/index.m3u8

DASH stream with no playback issues: DTS:X 5.1.4 - DASH: https://secured.streamdts.com/dtsx/dtsx/dash/callout/index.mpd

The PTS values are the same in both streams.

Bug Report

• [ ] You will email the zip file produced by adb bugreport to [email protected] after filing this issue.

[cedricxperi]

@tianyif If you have the bandwidth, kindly have a look at this issue above and let me know if it rings any bell. Thanks!

[cedricxperi]

@tianyif Just checking if you have any updates for me on this issue. Thanks.

[cedricxperi]

@tianyif We just received notice that this issue is affecting some Android TVs running Disney+ apps. Kindly look at this issue. It is highly likely that any audio codec with a Sync Frame concept delivered with HLS-fMP4 will be affected. Thanks.

[cedricxperi]

@tianyif Do you have any updates on this issue? Please see my earlier comments. Thanks.

[tianyif]

Hi @cedricxperi,

Sorry for the really late response! I'm taking look on this issue, however I can only try to reproduce this issue on our testing TV tomorrow as remote doesn't work until I get the battery refilled.

However, I'm wondering if the our library supports the DTS stream with the fMP4. We have DtsReader which is responsible for parsing the DTS stream but it is currently only used in the TS extractor. So I guess not being able to parse correctly can lead to the sync issue?

[cedricxperi]

@tianyif No worries. To my knowledge, DTS:X parsing is supported in fMP4 (sample description box etc). DASH is using that and has no issues. Thanks.

[tianyif]

@cedricxperi，

Thanks for the clarification! Unfortunately I was not able to reproduce with our TV as ours doesn't support DTS:X Profile 2. As you said you could reproduce this issue with demo app, could you please add a log for the avSyncPresentationTimeUs in this method and let me see how the log is produced for both tunneling mode on and off?

I haven't seen any potential reason that samples queued can be different between tunneling mode on and off actually, so I'm not sure if we can see difference at the mentioned place. I assume there are some gaps between the AV sync in ExoPlayer and the Android platform at this moment. But let's see.

[cedricxperi]

avSyncPresentationTimeUs_No_tunnelling.txt avSyncPresentationTimeUs_with_tunnelling.txt @tianyif Attached are the log files. However, I realised that this issue is only seen on the Xiaomi TV when I turn off "USB debug". In order to capture the log files, I have to turn on "USB Debug". When I do that, the first few seconds of the HLS-fMP4 file is NOT lost. However, the playback stutters probably due to the Xiaomi TV's CPU utilisation issue when USB debug is enabled. Even AAC has stuttered playback once USB debug is "on". So you will see that the time stamps in both files are the same because the first few seconds are not lost when "USB Debug" is enabled. I am not sure how helpful the log files are. Thanks.

[cedricxperi]

@tianyif FYI, the log files are obtained via the following changes: https://github.com/cedricxperi/media/commit/5a8f3ee70643a601d4d8a02158ad4304a441137e May I know where you are based? So that I can recommend some TVs in your region that support DTS:X Tunnel mode playback. Thanks.

[tianyif]

Thanks @cedricxperi! I received the logs and also thanks for providing new observation. Pretty surprised to see that USB debug introduces some difference. Would you be able to try wireless debugging?

I'm UK based and please let us know the TVs that supports DTS:X Tunnel mode, but to be honest I'm not sure if we can really get one of them. I'll discuss this issue with the team next week.

[cedricxperi]

avSyncPresentationTimeUs_direct_passthrough_playback_tunnelling_enabled.txt avSyncPresentationTimeUs_media_codec_playback_tunneling_disabled_dts_removed_from_all_surround_encodings.txt @tianyif Managed to capture the logs by not debugging the app directly from Android Studio. Attached are the 2 logs. One is through direct pass through playback and the other is through media codec playback. To force media codec playback I commented out all the DTS entries in AudioCapabilities.java, ALL_SURROUND_ENCODINGS_AND_MAX_CHANNELS.

From the log, you will see that the time stamps are different for tunnel mode (video) with direct passthrough playback for DTS audio vs Non-tunnel mode (video)+ mediacodec playback for DTS audio. Thanks.

P/S: I tried wireless debugging but the results were the same as USB debugging. Will get back to you on TV models.

[tianyif]

Thanks @cedricxperi! Received the two new logs! Just want to clarify - do the names of the above two logs need to be swapped? I see the timestamps of the avSyncPresentationTimeUs_direct_passthrough_playback_tunnelling_enabled are the same as the ones sent in the previous comment, where there were no 2 secs dropped, but theavSyncPresentationTimeUs_media_codec_playback_tunneling_disabled_dts_removed_from_all_surround_encodings.txt is different.

[cedricxperi]

@tianyif I think you can ignore the log files in the previous comment because those playback actually stutters and it looked to me like the Xiaomi TV was struggling to playback with USB debug turned on. The new two logs file names are correct. Thanks.

[cedricxperi]

@tianyif Do you have any updates for me on this issue? Anything unusual with the latest log files that I sent earlier? Thanks.