nginx timeout when exporting large project via POST /api/projects/{project_id}/exports

[b-gran]

Describe the bug We experienced nginx timeout errors when using the recommended export snapshot API to export a large project.

The POST /api/projects/{project_id}/exports API succeeded for this large project prior to the 1.12.0 update.

The project has ~18000 tasks, ~25000 annotations, and each task has ~2kb of data. The size of the project's JSON export is ~110MB.

The nginx timeout errors have the following form:

2024/05/02 13:22:03 [warn] 21#21: *173314 upstream server temporarily disabled while reading response header from upstream, client: 10.128.xxx.xxx, server: , request: "POST /api/projects/14/exports HTTP/1.1", upstream: "http://127.0.0.1:8000/api/projects/14/exports", host: "<snip>"
2024/05/02 13:22:03 [error] 21#21: *173314 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.128.xxx.xxx, server: , request: "POST /api/projects/14/exports HTTP/1.1", upstream: "http://127.0.0.1:8000/api/projects/14/exports", host: "<snip>"

To Reproduce Steps to reproduce the behavior:

1. Create a large project with 18000 tasks, 25000 annotations, and ~2kb of data per task.
2. POST /api/projects/{project_id}/exports
3. Observe nginx timeout errors

Expected behavior POST /api/projects/{project_id}/exports succeeds regardless of the size of the project size.

Environment (please complete the following information):

• Label Studio version 1.12.0

[farioas]

Looks the backend went down because it either took > 90s to respond or lack of resources. Try to set env variable UWSGI_WORKER_HARAKIRI=0.

[sfrolich]

Looks the backend went down because it either took > 90s to respond or lack of resources. Try to set env variable UWSGI_WORKER_HARAKIRI=0.

We tried setting UWSGI_WORKER_HARAKIRI to 361 and 0 but the error still occurs.

[JaeWangL]

Me too @sfrolich

[farioas]

Oh crap, I missed that you are declaring a problem at the nginx level.

https://github.com/HumanSignal/label-studio/blob/develop/deploy/default.conf#L193-L196#6039

The timeout for nginx is hardcoded in default.conf, you can try to adjust it along with UWSGI_WORKER_HARAKIRI env variable.