Expose logConfiguration options for jobs running on AWS Batch

[corleyma]

By default, AWS Batch uses the awslogs log driver to ship logs for a job container to AWS CloudWatch. However, this behavior is configurable, and even the awslogs driver itself has configurability that may be of interest to Metaflow users.

Example motivating usecases:

• specifying the log group or stream prefix may be useful for organizations that have schemes dictating how logs are collected in CloudWatch
• users of the Python standard logging library may benefit from the ability to configure multi-line pattern matching to ensure tracebacks and other multi-line log messages are interpreted as single events in CloudWatch.

The ask here is to expose awslogs logging config options, preferably in two ways:

1. as properties settable per step when using the @batch decorator
2. and, as is the case for other Batch configuration, as defaults in the Metaflow config profile.

Because Metaflow currently has an implicit dependency on logs making their way to CloudWatch, allowing users to specify log drivers other than awslogs may not be feasible today. Still, this is something worth thinking about for the future.

[savingoyal]

Makes total sense. We can add this functionality in after we have published #449.

[mt-ob]

https://github.com/Netflix/metaflow/pull/1767 offers it as properties settable per step when using the @batch decorator. Should we close this?

[saikonen]

Let's leave it open for now as there is still the other use case that is missing

• Being able to configure defaults for the logging through Metaflow profile (or env variables set by the exec platform)