dagster icon indicating copy to clipboard operation
dagster copied to clipboard

Change the default implementation to calculate step stats more efficiently

Open prha opened this issue 3 months ago • 0 comments

Summary & Motivation

By default, we calculate step stats by fetching all of the events and filtering in-memory to construct the step stat summary.

This PR changes the event fetch to pass in an event type filter.

This has the biggest impact for runs that have a lot of unstructured events and that do not directly inherit from the SqlEventLogStorage (e.g. some Cloud storage implementations)

How I Tested These Changes

BK

prha avatar Nov 22 '24 18:11 prha