dagster icon indicating copy to clipboard operation
dagster copied to clipboard

[pipes] use PipesDefaultLogWriter for AWS EMR

Open danielgafni opened this issue 3 months ago • 3 comments

Summary & Motivation

Allow receiving logs from stdout/stderr via Pipes messages. This means simply skipping the attachment of standard PipesLogReader. Added a flag to PipesEMRClient which does that. Not sure if it's the best location for this flag. Other options are:

  • PipesS3MessageReader (actually just any PipesMessageReader)
  • open_dagster_pipes

It looks to me like we could automate the switch to PipesDefaultLogWriter in the remote process if this new stdio_from_messages param was included into the Pipes payload.

How I Tested These Changes

Executed a real AWS EMR job and confirmed we are getting the logs

Changelog

[dagster-aws] added a stdio_from_messages parameter to PipesEMRClient. Setting it to True will disable S3 log readers. It should be used together with a PipesDefaultLogWriter in the EMR job.

danielgafni avatar Nov 08 '24 22:11 danielgafni