Neethugkp

Results 7 comments of Neethugkp

[BUG] Mlflow returns error 504 after uploading large files (800MB +). Error with mlflow.log_artifact()

@harupy :- updated. Hope its readable.

[BUG] Mlflow returns error 504 after uploading large files (800MB +). Error with mlflow.log_artifact()

@harupy . Upload of filesize less than 750 MB works without any error. Larger file also gets uploaded in s3. But after the file upload is successful, code execution abruptly...

[BUG] Mlflow returns error 504 after uploading large files (800MB +). Error with mlflow.log_artifact()

Mlflow logs of smaller file upload ``` [2022-12-22 17:03:04 +0000] [74] [INFO] Starting gunicorn 20.1.0 [2022-12-22 17:03:04 +0000] [74] [DEBUG] Arbiter booted [2022-12-22 17:03:04 +0000] [74] [INFO] Listening at: http://0.0.0.0:5000...

[BUG] Mlflow returns error 504 after uploading large files (800MB +). Error with mlflow.log_artifact()

> @Neethugkp Increasing `MLFLOW_HTTP_REQUEST_TIMEOUT` (environment variable, default value: 120) might help. > > ``` > MLFLOW_HTTP_REQUEST_TIMEOUT=360 python run.py > ``` @harupy MLFLOW_HTTP_REQUEST_TIMEOUT=360 environment variable is set. There is no change...

[BUG] Mlflow returns error 504 after uploading large files (800MB +). Error with mlflow.log_artifact()

@BenWilson2 @dbczumar @harupy @WeichenXu123 We have tested the latest version 2.2.1. Large files are getting uploaded(tested up to logging 3GB files ). But at the end of execution its throws...

[BUG] Mlflow returns error 504 after uploading large files (800MB +). Error with mlflow.log_artifact()

@Earthwings The workaround that worked for us was by increasing the resource limits (cpu/memory). From version 2.2.0 onwards it works.

[BUG] Mlflow returns error 504 after uploading large files (800MB +). Error with mlflow.log_artifact()

@harupy Would like to reopen the issue as it fails to upload files with more than 3GB files. In older versions it failed to upload files more than 800 MB...