distilabel
distilabel copied to clipboard
[FEATURE] Show expected number of batches in logging
Is your feature request related to a problem? Please describe.
Currently, I get the message Processing batch x
but this does not indicate how far we are.
Describe the solution you'd like
Something like Processing batch x of total of y batches
might be better
Describe alternatives you've considered N.A. Additional context N.A.
Yes, I agree, but it will require a bit of thought. Step are quite flexible and they can filter, add new rows, etc and there's no way to know how many rows will be added/removed by each step, making a bit hard to compute the number of batches that will be processed by certain step.