distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

[FEATURE] Show expected number of batches in logging

Open davidberenstein1957 opened this issue 10 months ago • 1 comments

Is your feature request related to a problem? Please describe. Currently, I get the message Processing batch x but this does not indicate how far we are.

Describe the solution you'd like Something like Processing batch x of total of y batches might be better

Describe alternatives you've considered N.A. Additional context N.A.

davidberenstein1957 avatar Apr 11 '24 17:04 davidberenstein1957

Yes, I agree, but it will require a bit of thought. Step are quite flexible and they can filter, add new rows, etc and there's no way to know how many rows will be added/removed by each step, making a bit hard to compute the number of batches that will be processed by certain step.

gabrielmbmb avatar Apr 15 '24 13:04 gabrielmbmb