[Logging]: Resource exhaustion with no limit on log files creation

[Kanupriyagoyal]

Description We have observed that upon invoking POST /v2/logging API, it creates a new file.

1. Fuzzing with filenames on "log_file" POST body parameter, one can create multiple or infinite number of files leading to resource exhaustion.

2. Also, upon providing an existing file as input to "log_file" parameter, one can possibly overwrite an existing legitimate file inside the container.

As part of testing, we were able to create more than 100 files in less than a minute and File name validation is not in place.

It can overload your Server and consume all of its resources. It is easily possible for a single user/client to exhaust server resources.

Error Log --> Multiple logfile creation with no limit and same file name overwritten

Triton Information r23.04

Are you using the Triton container or did you build it yourself? Same Instructions as build.py

To Reproduce Steps to reproduce the behavior. Parameter log_file Affected URL/API(s) POST /v2/logging

1. send log_file with different names.

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

Expected behavior A clear and concise description of what you expected to happen.

• Validate the filenames provided for "log_file" parameter.(There should name check, if file already exists we should get message file already exist rather than overwriting the file, which will lead to data loss.)
• Restrict the number of files one can create per minute or minutes.

[Kanupriyagoyal]

@dyastremsky: . Here is one more on the log resource exhaustion.It would be great if you help here.Thank you !

[pradghos]

Can we restrict number of log file creation or not allow/restrict user to create log file ? Thanks!

[the-david-oy]

Thank you for your feature request. We have filed a ticket (DLIS-5274) to investigate this enhancement.

[rmccorm4]

CC @fpetrini15 just FYI

[kpg993]

@dyastremsky Is it planned for any immediate release version. Thanks in advance!!

[the-david-oy]

This has not yet been prioritized.

[nnshah1]

@pradghos , @Kanupriyagoyal , @kpg993

We are actively looking into this issue and are considering removing the capability to change the log file location / name via the REST / GRPC API.

Are there valid use cases where we want to allow for name changes but restrict the number?

OR would it be sufficient to restrict the log file name and location to be only settable via command line launch parameters?

CC: @rmccorm4 , @yinggeh

[Kanupriyagoyal]

@nnshah1 @rmccorm4 @yinggeh It should be fine to have single file with fixed name and location if no much impact on user functionality. On a precautionary measure, is there any way file size can grow and exhaust system resources.

[Hemaprasannakc]

Hello @nnshah1 ,

I see there is a PR(https://github.com/triton-inference-server/server/pull/7092 ) for Disabling the Dynamic Log file. Could you confirm if this can turn off log api so that system resource exhaust will not occur?

or , is there any other way this can be still reproduced?

[Hemaprasannakc]

Hello @dyastremsky / @nnshah1 ,

I also tried doing the trace file settings through REST API

curl -X POST -v localhost:8000/v2/models/test_model/trace/setting -d '{"trace_level":["OFF"],"trace_rate":"1000","trace_count":"-1","log_frequency":"0","trace_file”:”my_sample_trace"}

and got {"error":"trace file location can not be updated through network protocol"}

This PR #7092 is also effective for trace file settings changes too along with log file?

[nnshah1]

@Hemaprasannakc , that is correct (though I believe in a separate PR) - we have disabled the ability to change the trace / log location via the REST API. The locations are still configurable during server launch and care should still be taken to avoid file sizes becoming large. Using a tool such as https://github.com/logrotate/logrotate can help handle this in a generic way.

[nnshah1]

am closing this as complete - as the original issue of creating many files via the REST API has been resolved.