I need to use njs to perform multiple subrequests and elaborate response of large files

[tmm360]

I have a source with large files (videos). When user makes a request, I need to misure the response size and proxy pass the response. After that transfer is finished, I need to notify an external service making a second request, passing this parameter.

Currently I've solved with two subrequests, but I know it's not a great solution. This is the concept, simplified:

	subrequest_output_buffer_size 500M;

	location / {
		js_content main.execRequest;
	}
	
	# Internal subrequests.
	location /filter {
		internal;
		rewrite ^/filter(?<realurl>/.*)$ $realurl break;

		js_body_filter main.requestFilter buffer_type=buffer;
		
		proxy_pass $my_source;
	}
	
	location /notify {
		internal;

		proxy_pass $my_external_service;
	}

async function execRequest(r) {
    try {
        var response = await r.subrequest(
            "/filter" + r.uri,
            {
                args: r.variables.args,
                method: r.method
            });

        // Proxy pass filtered response.
        proxySubrequestResponse(response, r); //not reported

        // Notify external service.
        await r.subrequest(
            "/notify",
            {
                body: JSON.stringify({
                    bodySize: r.variables.streamedBytes
                }),
                detached: true
            });
    } catch (e) {
        r.return(500, e);
    }
}

function requestFilter(r, data, flags) {
    // Filter body (very simplified).
    r.sendBuffer(data, flags);
    r.variables.streamedBytes = parseInt(r.variables.streamedBytes, 10) + Buffer.byteLength(data);
}

This solution has several disadvantages:

• accepts responses with max size of 500MB (correct?)
• keeps everything in memory (doesn't scale well)
• doesn't permit to handle cases when a result is only partially downloaded (I'm not interested in know the full file size, but only the consumed bandwidth)

Exists any better way for handle this kind of problems? Eventually using external modules.

[xeioex]

Hi @tmm360,

The more standard approach for such tasks is outputting all the necessary data to logs, For example using variables like $upstream_response_length, with a proxy_pass location and custom logs and later processing the logs by a separate tool.

You need to track some stats, right? If yes, having a well-prepared access log (with all the necessary info) feel free to write a tool that will send that data to the second server. It depends on the what you are trying to do, but by having logs you will be able to bunch multiple requests at ones, thus making you system able to cope with more load.

Also, there are nginx log analysers out there, some of them probably already have what you need.

[tmm360]

Hi @xeioex, thank you for your reply, but unfortunately this solution doesn't fit with my application. I'm not interested in logs, but I've to receive a prompt communication with rest API, because external service have to take actions, based on this notification. So it's not the case.

[xeioex]

@tmm360,

Alternatively, you may consider the post_action directive. It is unofficially supported by nginx.

In a post_action location you may add another proxy_pass and construct the desired request for your external service.

[tmm360]

Great! It looks like you have found what I needed :) I will try, thanks!

[xeioex]

Hi @tmm360,

Please, share whether you was able to use post_action to solve your task.

[tmm360]

Yes, absolutely, this was THE solution! Thank you :)

[xeioex]

@tmm360,

Glad to hear!

Feel free to share the simplified nginx.conf with the solution for anyone with the similar tasks.

[tmm360]

All right. My final solution:

	location / {
		js_body_filter main.requestFilter buffer_type=buffer;
		
		proxy_pass $my_source;

		post_action /notify;
	}
	
	location /notify {
		internal;

		proxy_pass $my_external_service;
	}

function requestFilter(r, data, flags) {
    // Filter body (very simplified).
    r.sendBuffer(data, flags);
    r.variables.streamedBytes = parseInt(r.variables.streamedBytes, 10) + Buffer.byteLength(data);
}

Much more concise and elegant!