[FEATURE] Allow more logging configuration options

[tcassaert]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

The logs are currently by default in colored console output. This is fine when you're debugging something, but it can make things complicated when collecting logs and parsing them.

Describe the solution you would like

Have some CLI arguments to configure the logging. For example to output in JSON format or to disable the colored output.

Additional context

When running spark-operator controller start --help, we currently have following options:

      --zap-devel                                        Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
      --zap-encoder encoder                              Zap log encoding (one of 'json' or 'console')
      --zap-log-level level                              Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
      --zap-stacktrace-level level                       Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
      --zap-time-encoding time-encoding                  Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.

However, these don't all work.

The logging is setup here: https://github.com/kubeflow/spark-operator/blob/f75097449d7c99d8672227ffc3bed3d3387aa543/cmd/operator/controller/start.go#L308-L328.

No matter what we configure on the CLI, there will always only be a console logger (https://github.com/kubeflow/spark-operator/blob/f75097449d7c99d8672227ffc3bed3d3387aa543/cmd/operator/controller/start.go#L325)

I'm not entirely sure how to we'd want to make it better configurable. Do we want to keep using the --zap-... CLI flags, or have our own ---log-level flag?