Remote with nginx-parallel-module

[jhemmmm]

I noticed that it takes a long process to get the header of each video when there are too many multi URLs, I was hoping to get the header of the video and cache them in parallel?. Is that possible when the mentioned module?. And can I add it as a --add-dynamic-module?.

[erankor]

Reading the metadata in parallel is not supported. I agree this could provide a nice performance improvement, but unfortunately, it's not a small change. Compiling as dynamic module is supported.

[jhemmmm]

Reading the metadata in parallel is not supported. I agree this could provide a nice performance improvement, but unfortunately, it's not a small change. Compiling as dynamic module is supported.

Yeah, I tried, I am getting some kind of memory error. It would be great if it could get supported. It is a nice performance improvement especially for having many playlists.

For example.

022/03/21 07:41:42 stream.go:34: bytes=0-4095 (2s)
2022/03/21 07:41:43 stream.go:34: bytes=4096-3478110 (2s)

2022/03/21 07:41:45 stream.go:34: bytes=0-4095 (2s)
2022/03/21 07:41:47 stream.go:34: bytes=4096-1980746 (2s)

2022/03/21 07:41:58 stream.go:34: bytes=0-4095 (2s)
2022/03/21 07:41:59 stream.go:34: bytes=4096-1980747 (2s)

If the metadata is not cached yet, the metadata are being fetched one by one, which could somehow slow down to get the master playlist. I mean comeone, it could do better than that.

[erankor]

I agree, but it's all a matter of priorities... :) I don't have the time to spend on this one now...

But, thinking out loud here... I think you could build it with nginx-lua - it should be possible to write some Lua code (in rewrite phase probably) that will issue either subrequests or real HTTP requests to localhost (there may be issues with subrequests, not sure if I ever tested something like this...) in parallel for the m3u8's of the different URLs. For example, if you get a request for http://domain/path/,1,2,3,4,.mp4.urlset/master.m3u8, it will count the commas, and issue requests in parallel for master-f1.m3u8 ... master-f4.m3u8. Once those requests finish, it will complete the rewrite phase, and allow the original request to continue normally, while using the already cached metadata. There is a small waste here of building the "child" m3u8's without using them, but it's negligible.

[jhemmmm]

I made something else instead. Since on my golang code, I have to fetch my API that runs in parallel to get the remote URLs and rearrange that with my loop and display the mapped JSON. So I have to do it like this.

for loop 
run in parallel
clip.path = fetchAPIAndReturnURL()
//fetch the URL with bytes 0-4095
fetch(URL)
//fetch the URL with 4096-random (since I do not know the random, I'll comment it out for the meantime)
//fetch(URL)
end loop

wait for parallel
return JSON clips

and on my nginx.conf

location /proxy/ {
  proxy_pass http://127.0.0.1:5050/;
  internal;
}

map $http_range $skip_cache{
	"~0-4095" 0;
	"~4096-*" 0;
	default 1;
}

server {
	listen 5050;

	location ~ /mp4/(.*) {
		set $hash $1;
		
		# Proxy
		proxy_cache default_cache;
		proxy_buffering on;
		
		# Proxy Cache Key
		proxy_cache_key $hash$http_range;
		proxy_set_header Range $http_range;
		proxy_http_version 1.1;
		proxy_cache_valid 200 206 5m;
		
		# Proxy Bypass
		proxy_cache_bypass $skip_cache;
		proxy_no_cache $skip_cache;

		proxy_pass http://servers/;
	}
}

Let me know if there's anything able to be improved here.

[jhemmmm]

@erankor How do I get the value of (bytes=4096-3478110) "3478110"?. It would be great if I could completely cache the metadata on disk and run it in parallel.

And since I have to return the JSON first, fetching the "master-f1.m3u8..." leads to an endless loop.

[erankor]

This position is the end of the moov atom in the MP4, assuming your MP4s have their metadata in the beginning (aka faststart), you'll have an ftyp atom followed by the moov atom, if you take the moov atom pos + size you'll get this number. Regarding the loop, you can add something on the URL to prevent it - some param on the path to mark the fact that it's a call initiated by your API server. But the solution I suggested was different... it was only on the nginx side, if you do that with nginx-lua, you have the full URL (including /master.m3u8) and you could use that to prevent the loop.

[jhemmmm]

This position is the end of the moov atom in the MP4, assuming your MP4s have their metadata in the beginning (aka faststart), you'll have an ftyp atom followed by the moov atom, if you take the moov atom pos + size you'll get this number. Regarding the loop, you can add something on the URL to prevent it - some param on the path to mark the fact that it's a call initiated by your API server. But the solution I suggested was different... it was only on the nginx side, if you do that with nginx-lua, you have the full URL (including /master.m3u8) and you could use that to prevent the loop.

Thanks, was able to identify the size but there's no better library available for golang at the moment.