scalding icon indicating copy to clipboard operation
scalding copied to clipboard

Consistently call validateTap in all src.toIterator implement

Open gerashegalov opened this issue 8 years ago • 1 comments

All toIterator impls should consistently validate taps in order to be able to rely on the iterator completeness. Compare com.twitter.scalding.Mappable#toIterator and com.twitter.scalding.commons.source.LzoCodec#toIterator, etc.

gerashegalov avatar Feb 07 '17 01:02 gerashegalov

cc @isnotinvain the whole validation thing is racy also. I'm not sure validateTap is the full solution.

It may be that we want make it easier for the the taps to validate as they read so they don't return partial results.

johnynek avatar Feb 07 '17 01:02 johnynek