Make sysbox grpc timeouts configurable ...

[ctalledo]

Sysbox-runc communicates with sysbox-fs and sysbox-mgr via gRPC. That channel has a timeout limit of ~40 secs, which is normally more than sufficient (since containers typically start within 1->3 secs).

However, in some cases sysbox-mgr needs to perform operations that take a long time (many secs) causing the timeout to hit.

One such case occurs when mounting a host volume/directory into the sysbox container's /var/lib/docker dir; In this case sysbox-mgr needs to chown the contents of that host volume/directory, and that operation can take a long time if many files (hundreds of thousands) reside within that directory. In this case sysbox-runc will hit the timeout an emit an error such as:

failed to request mount source preps from sysbox-mgr: failed to invoke PrepMounts via grpc: rpc error: code = DeadlineExceeded desc = context deadline exceeded" 

Some users have asked that the gRPC timeout be configurable to avoid this error, even if it means waiting a long time for the container to start.

This issue calls for Sysbox to make this configurable, via a sysbox-mgr command line flag, and possibly with override via a container env variable.

[ctalledo]

One such case occurs when mounting a host volume/directory into the sysbox container's /var/lib/docker dir; In this case sysbox-mgr needs to chown the contents of that host volume/directory, and that operation can take a long time if many files (hundreds of thousands) reside within that directory.

BTW, the "real" solution for this is for Sysbox to use ID-mapped-mounts on that host volume/directory. But that would in turn require that overlayfs work on top of ID-mapped-mounts which is a feature that has only recently shown up in kernel 5.19 (see here).

Nonetheless, making the sysbox gRPC timeout configurable is still a desirable thing on its own right.