Elijah ben Izzy
Elijah ben Izzy
https://github.com/DAGWorks-Inc/hamilton/pull/583 gets this started
> I think there's a place for this along with `@pipe`. Also the `data.raw -> data` fits the `@check_output` semantic. > > Although, I propose to rename it `@pipe_out` `@pipe_output`...
From @skrawcz on #749 (duplicate): **Is your feature request related to a problem? Please describe.** `@pipe` is nice, but can be a little counter-intuitive to read. i.e. the function parameter...
Thoughts: 1. Do we want to enable a per-data-quality check? Currently this is per-node. I like it this way as the API is simpler, but it could get annoying if...
Ok, I'm going to think through this API this weekend -- feel more confident about the others so i might hold off on this for a bit
One idea -- is this something that could be grouped into one? (will need a larger name) ```python @inject_subdags(config="subdag_config") def collect(subdag_results: Collection[...]) -> ...: ... ``` I think the trickier...
Ok, we've been mulling over some ideas. Here's what I've thought through. This still has some of the complexity of the framework, but its hidden in an object you manage....
What @skrawcz said, but I also want to say that I don't think you're overthinking it. Complex DAGs are just that (complex), and this is why I'm really excited about...
> > Interesting. How many nodes is your DAG overall then? It shouldn't be that slow. > > My config is a `dict[Tuple[str,str], dict[str, pd.Timestamp]]` with about 1750 elements. >...
Timeboxed this but there's some weirdness about how we enforce the config that made me not want to deal with this yet...