No stdout output for default parallel_workers=1

[rfalke]

trafficstars

I'm new to seldon. When I added some debug statements (using print() and also the logging module) to the model I found that it will work in load() but not predict(). I spend multiple hours to understand that multiprocessing is used by default and it is responsible for swallowing the output.

I have a few questions and suggestions:

1. AFAI understand seldon uses the same amount of CPU (1) when parallel_workers is 0 or 1 (the default). So why was 1 chosen as the default?
2. Is it possible to see the log messages from predict calls when the parallel pool is used? How?
   1. If it is possible to get the output: Can it be made the default? Or at least log a warning that further messages from predict-calls will not be shown.
   2. If it is not possible to get the output: Can parallel_workers default to 0? Or at least log a warning that further messages from predict-calls will not be shown.

I'm using 1.1.0

[adriangonz]

Hey @rfalke ,

This is currently a bug in the parallel worker implementation. Main reason seems to be that the logger is not initialised properly on the workers themselves. It should be a quick fix though, which will be addressed in time for the next release of MLServer.

As a temporary workaround (until the above is fixed), print() statements should show up, even when the model is executed in the parallel worker pool.

Regarding your other point, when parallel workers are set to 1 (i.e. the default), there will be a separate worker process where inference will be executed. This gives some "space" to the main process to handle open connections, adaptive batching, manage requests from the admin API (e.g. load / unload models), etc (which otherwise would get blocked due to the GIL). Therefore, it's generally recommended to always run with at least one worker.

[rfalke]

Sadly I don't see any messages from print() when parallel_workers is 1.

Thanks for clearing up on the other point.

[adriangonz]

Hey @rfalke ,

We are unable to replicate the point where print() messages don't show up. We can do so with logger.info() though, so we'll focus on solving the latter.