TimeWindowPartitionMapping with offsets between different granularity PartitionsDefinitions

[sryza]

By default, each time window partition depends on the upstream partition(s) that cover the same time window.

The start_offset and end_offset parameters on TimeWindowPartitionMapping allow shifting these windows, so that, e.g. a downstream partition depends on the upstream partition for the prior day.

However, these parameters only work if the upstream and downstream PartitionsDefinitions share the same time granularity - e.g. are both hourly or both daily.

This issue tracks expanding this support to cases where they have different granularity.

A potential approach would be for the start_offset and end_offset parameters to accept a pendulum Duration object instead of an int.

What we've heard

• siim.sillaots - https://dagster.slack.com/archives/C01U954MEER/p1686338090949019?thread_ts=1686300258.965529&cid=C01U954MEER
  • Yes, rather this. I would want to run wednesdays hourly partitions even though one of the upstreams of that is a daily partition that doesn’t have a wednesdays partition yet. Right now we “solved” the situation by setting hourly asset’s end offset -24

[sryza]

PR from a little while ago: https://github.com/dagster-io/dagster/pull/10190

[mvanaltvorst]

Hi! I've also encountered this limitation, are there any updates on this issue?

[mmutso-boku]

This would be a very useful addition

[Rahlir]

Adding my +1 on this. It would be very useful to have this.

[u-ra-ra-ra]

+1 to this.