ll-hls reliably stops working after ~10 minutes of smooth playback

[naanlizard]

Describe the bug LL-HLS works and generally is very good, but after 10 minutes or so, playback reliably stops and the player (ovenplayer) starts buffering endlessly until you refresh.

To Reproduce

Dockerfile:

FROM airensoft/ovenmediaengine:0.14.5

RUN apt-get update && apt-get install -y curl

ENV API_ACCESS_TOKEN=x

WORKDIR /opt/ovenmediaengine/bin

COPY ./Server.xml /opt/ovenmediaengine/bin/origin_conf/Server.xml
COPY EdgeConfig.xml ./edge_conf/Server.xml

COPY entrypoint.sh .

HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 CMD curl --fail -H "Authorization: Basic $(printf ${API_ACCESS_TOKEN} | base64)"  'http://localhost:48081/v1/vhosts' 

CMD ["bash", "entrypoint.sh"]

Server.xml

<?xml version="1.0" encoding="UTF-8"?>

<Server version="8">
	<Name>OvenMediaEngine</Name>
	<!-- Host type (origin/edge) -->
	<Type>origin</Type>
	<!-- Specify IP address to bind (* means all IPs) -->
	<IP>*</IP>

	<!-- 
	To get the public IP address(mapped address of stun) of the local server. 
	This is useful when OME cannot obtain a public IP from an interface, such as AWS or docker environment. 
	If this is successful, you can use ${PublicIP} in your settings.
	-->
	<StunServer>stun.l.google.com:19302</StunServer>

	<!-- Settings for the ports to bind -->
	<Bind>

		<Managers>
			<API>
				<Port>48081</Port>
				<WorkerCount>1</WorkerCount>
			</API>
		</Managers>


		<Providers>
			<!-- Pull providers
			<RTSPC>
				<WorkerCount>1</WorkerCount>
			</RTSPC>
			<OVT>
				<WorkerCount>1</WorkerCount>
			</OVT>
			-->
			
			<!-- Push providers -->
			<RTMP>
				<Port>${env:OME_RTMP_PROV_PORT:1951}</Port>
				<WorkerCount>1</WorkerCount>
			</RTMP>
			<SRT>
				<Port>9999</Port>
				<WorkerCount>1</WorkerCount>
			</SRT>
			<WebRTC>
				<Signalling>
					<Port>${env:OME_SIGNALLING_PORT:3333}</Port>
					<WorkerCount>1</WorkerCount>
				</Signalling>

				<IceCandidates>

					<TcpRelay>*:3478</TcpRelay>
					<!-- TcpForce is an option to force the use of TCP rather than UDP in WebRTC streaming. (You can omit ?transport=tcp accordingly.) If <TcpRelay> is not set, playback may fail. -->
					<TcpForce>false</TcpForce>

					<IceCandidate>${env:OME_ICE_CANDIDATES:*:10000-10005/udp}</IceCandidate>
					<TcpRelayWorkerCount>1</TcpRelayWorkerCount>
				</IceCandidates>
			</WebRTC>
	
		</Providers>

		<Publishers>
			<WebRTC>
				<Signalling>
					<Port>${env:OME_SIGNALLING_PORT:3333}</Port>
					<!-- If you want to use TLS, specify the TLS port -->
					<TLSPort>${env:OME_SIGNALLING_PORT_SECURE:3334}</TLSPort>
					<WorkerCount>1</WorkerCount>
				</Signalling>
				<IceCandidates>
					<!-- 
						If you want to stream WebRTC over TCP, specify IP:Port for TURN server.
						This uses the TURN protocol, which delivers the stream from the built-in TURN server to the player's TURN client over TCP. 
						For detailed information, refer https://airensoft.gitbook.io/ovenmediaengine/streaming/webrtc-publishing#webrtc-over-tcp
					-->
					<TcpRelay>*:3478</TcpRelay>
					<!-- TcpForce is an option to force the use of TCP rather than UDP in WebRTC streaming. (You can omit ?transport=tcp accordingly.) If <TcpRelay> is not set, playback may fail. -->
					<TcpForce>false</TcpForce>

					<IceCandidate>${env:OME_ICE_CANDIDATES:*:10000-10005/udp}</IceCandidate>
					<TcpRelayWorkerCount>1</TcpRelayWorkerCount>
				</IceCandidates>
			</WebRTC>

			<HLS>
				<Port>799</Port>
				<!-- If you want to use TLS, specify the TLS port -->
				<TLSPort>800</TLSPort>
				<WorkerCount>1</WorkerCount>
			</HLS>

			<LLHLS>
				<Port>799</Port>
				<TLSPort>800</TLSPort>
				<WorkerCount>1</WorkerCount>
			</LLHLS>
		</Publishers>
	</Bind>

	<!-- P2P works only in WebRTC -->
	<!--
	<P2P>
		<MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
	</P2P>
	-->

	<!--
		Enable this configuration if you want to use API Server
		
		<AccessToken> is a token for authentication, and when you invoke the API, you must put "Basic base64encode(<AccessToken>)" in the "Authorization" header of HTTP request.
		For example, if you set <AccessToken> to "ome-access-token", you must set "Basic b21lLWFjY2Vzcy10b2tlbg==" in the "Authorization" header.
	-->

	<Managers>
		<Host>
			<Names>
				<Name>*.hostname.tld</Name>
				<Name>hostname.tld</Name>
				<Name>localhost</Name>
				<Name>127.0.0.1</Name>
			</Names>
			<TLS>
				<CertPath>/opt/ovenmediaengine/cert.pem</CertPath>
				<KeyPath>/opt/ovenmediaengine/privkey.pem</KeyPath>
				<ChainCertPath>/opt/ovenmediaengine/fullchain.pem</ChainCertPath>
			</TLS>
		</Host>
		<API>
			<AccessToken>${env:API_ACCESS_TOKEN:x}</AccessToken>
		</API>
	</Managers>


	<VirtualHosts>
		<!-- You can use wildcard like this to include multiple XMLs -->
		<VirtualHost>
			<Name>default</Name>
			<!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value is output to the events log, so you can use it to aggregate statistics. -->
			<Distribution>hostname.tld</Distribution>

			<!-- Settings for multi ip/domain and TLS -->
			<Host>
				<Names>
						<Name>*.hostname.tld</Name>
						<Name>hostname.tld</Name>
						<Name>localhost</Name>
						<Name>127.0.0.1</Name>
				</Names>
				<TLS>
					<CertPath>/opt/ovenmediaengine/cert.pem</CertPath>
					<KeyPath>/opt/ovenmediaengine/privkey.pem</KeyPath>
					<ChainCertPath>/opt/ovenmediaengine/fullchain.pem</ChainCertPath>
				</TLS>
			</Host>
			
			<!-- Settings for applications -->
			<Applications>
				<Application>
					<Name>live</Name>
					<!-- Application type (live/vod) -->
					<Type>live</Type>
					<OutputProfiles>
						<!-- Enable this configuration if you want to hardware acceleration using GPU -->
						<HardwareAcceleration>false</HardwareAcceleration>
						<OutputProfile>
							<Name>bypass_stream</Name>
							<OutputStreamName>${OriginStreamName}</OutputStreamName>
							<Encodes>
								<Audio>
									<Bypass>true</Bypass>
								</Audio>
								<Video>
									<Bypass>true</Bypass>
								</Video>
								<Audio>
									<Codec>opus</Codec>
									<Bitrate>128000</Bitrate>
									<Samplerate>48000</Samplerate>
									<Channel>2</Channel>
								</Audio>
							</Encodes>
						</OutputProfile>

						<!-- Audio only -->
						<OutputProfile>
							<Name>audio_only</Name>
							<OutputStreamName>${OriginStreamName}_audio</OutputStreamName>
							<Encodes>
								<Audio>
									<Bypass>true</Bypass>
								</Audio>
								<Audio>
									<Codec>opus</Codec>
									<Bitrate>128000</Bitrate>
									<Samplerate>48000</Samplerate>
									<Channel>2</Channel>
								</Audio>
							</Encodes>
						</OutputProfile>

					</OutputProfiles>
					<Providers>
						<WebRTC />
						<RTMP />
						<WebRTC>
							<Timeout>30000</Timeout>
						</WebRTC>
					</Providers>
					<Publishers>
						<AppWorkerCount>1</AppWorkerCount>
						<StreamWorkerCount>8</StreamWorkerCount>

						<WebRTC>
							<Timeout>30000</Timeout>
							<Rtx>true</Rtx>
							<Ulpfec>true</Ulpfec>
							<JitterBuffer>true</JitterBuffer>
						</WebRTC>

						<RTMPPush></RTMPPush>

						<HLS>
              <SegmentDuration>1</SegmentDuration>
              <SegmentCount>3</SegmentCount>
              <CrossDomains>
                <Url>hostname.tld</Url>
              </CrossDomains>
            </HLS>

						<LLHLS>
							<ChunkDuration>0.2</ChunkDuration>
							<SegmentDuration>6</SegmentDuration>
							<SegmentCount>10</SegmentCount>
							<CrossDomains>
								<!-- <Url>hostname.tld</Url>
								<Url>*.hostname.tld</Url>
								<Url>https://staging.hostname.tld:8888</Url> -->
								<Url>*</Url>
							</CrossDomains>
						</LLHLS>
					</Publishers>
				</Application>
			</Applications>
		</VirtualHost>
	</VirtualHosts>
</Server>
					  

Expected behavior Smooth playback, no buffering

Logs A sample log extract (same as in ticket #847 )

omelog.txt

Player (please complete the following information):

• Device: Have tested and observed in various Chrome versions (including the most recent as of this post) on windows, Chrome on ios

Additional context Could be something in our player page, we will test that, but I figured I'd open a ticket here in case something is obviously wrong with our setup

[getroot]

Could you please test it with THEO Player (https://www.theoplayer.com/test-your-stream-hls-dash-hesp) to see if it's a problem with hls.js?

You can test by enabling the "Use your own stream" button and inserting your URL. Alternatively, you can select "HLS Low Latency - AirenSoft" and click the "CLICK HERE TO LOAD YOUR STREAM" button to test it on the server we provide.

For reference, the 857:53:26.940 part at the bottom of the "HLS Low Latency - AirenSoft" video shows the time this stream worked without interruption. It's been working fine without crashes for over 30 days already, but your crash issue is not reproduced. I would very much like to solve your crash issue.

[naanlizard]

The THEO player works great, no buffering or anything ever that I have seen (admittedly, only a day of testing)

OP on our site gets stuck buffering after some time, usually within 10 minutes, sometimes longer than that though.

OP on the demo site does this after letting it run for 12+ hours - https://i.imgur.com/wY49ucJ.gif

I can share the demo link via PM or email, I would just prefer not to share it publicly.

[getroot]

Please send demo link to [email protected]. We can try to apply some options to hls.js

[getroot]

I tested your URL playback in Seoul, Korea. As a result your chunk size (0.2s) is too small. This will only play well in the best network environment. This can cause the player's buffers to underflow when network jitter occurs, resulting in continued unstable playback. Besides, you're not letting the player have enough buffers because you've set Target latency and Max latency in the player's HLS Options. I'm guessing that's why your players are constantly experiencing buffering. On the other hand, THEO Player does not have Target latency and Max latency settings, so when jitter occurs, THEO Player is increasing the buffer sufficiently. I think that's probably why it plays better than hls.js. (Of course, you can set these options for THEO as well. It's just not on the demo site.)

As you know OvenPlayer uses hls.js to play HLS. The hls.js demo site below can give you many options and you can also monitor the buffering status of the player. It is recommended to test it while adjusting various options on the demo site below. https://hls-js.netlify.app/demo/

And in my experience, when servicing a global target, 0.5 seconds is the most appropriate for the partial segment size of LLHLS.

[naanlizard]

We will try adjusting those settings, thank you

[getroot]

And if you want to discuss further, please close this issue and create an issue in the OvenPlayer repo. Thank you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.