Ben Pfaff
Ben Pfaff
@ryzhyk This is what I arrived at after our discussion this morning. Threading Kafka will exacerbate this race but I believe it is there already.
This is fixed (in the controller, which now tells input adapters when they are allowed to submit data).
> Does the same apply for the end-of-input flag? I don't think we checkpoint EOI either. A clean way to handle checkpoint EOI might be to delete endpoints that reach...
Do we want to be able to collect a profile for the running step, so that we can see what's going on that's making the current step so long? I...
From size_of? Yeah, that can be expensive. It probably only matters for spines? And we've recently make it faster for in-memory batches, right? Just trying to think out loud here.
> > From size_of? Yeah, that can be expensive. It probably only matters for spines? And we've recently make it faster for in-memory batches, right? > > Only for estimating...
> [@ryzhyk](https://github.com/ryzhyk) names are not unique across tenants, so we might need to do (tenant, name). We might need to add that to PipelineConfig, which currently only include a pipeline...
@ryzhyk It looks to me like the pipeline manager currently supplies the pipeline UUID as `name` in `PipelineConfig`. If so, then if it supplied the pipeline's name instead, we'd get...
Assigning to @snkas because I can't do it in the pipeline process without having the pipeline name (which AFAIK the pipeline doesn't have). The pipeline manager could do this without...
Fixed with https://github.com/feldera/feldera/pull/5119