Cutoff time for upstream dependency

[kushsharma]

Is your feature request related to a problem? Please describe. There are many cases where the upstream dependency is taking a lot longer then it should because of which all the downstream dependencies are blocked. I think there should be a way to provide a cutoff time after which the upstream dependency should be ignored.

Describe the solution you'd like Imagine a dependency graph like

flowchart TD;
    A-->B;
    X-- slow job -->B;
    B-->D;
    C-->D;

Here if X is taking too much time, we should have freedom to cutoff X from the dependency and continue the execution to not block D. We will consume X data maybe the next day?

Additional context I was thinking because these cutoff duration should be per dependency, we can define them as part of dependency config in yaml.

Just a thought, maybe we can get this feature reviewed by BI team?