dagster
dagster copied to clipboard
[pipes] use PipesDefaultLogWriter for AWS EMR
Summary & Motivation
Allow receiving logs from stdout/stderr via Pipes messages.
This means simply skipping the attachment of standard PipesLogReader
.
Added a flag to PipesEMRClient
which does that. Not sure if it's the best location for this flag. Other options are:
-
PipesS3MessageReader
(actually just anyPipesMessageReader
) -
open_dagster_pipes
It looks to me like we could automate the switch to PipesDefaultLogWriter
in the remote process if this new stdio_from_messages
param was included into the Pipes payload.
How I Tested These Changes
Executed a real AWS EMR job and confirmed we are getting the logs
Changelog
[dagster-aws] added a stdio_from_messages
parameter to PipesEMRClient
. Setting it to True
will disable S3 log readers. It should be used together with a PipesDefaultLogWriter
in the EMR job.