data icon indicating copy to clipboard operation
data copied to clipboard

[DataPipe] key renamer

Open tmbdev opened this issue 2 years ago • 3 comments

This PR adds a filter that allows keys to be renamed in training samples represented as dictionaries. This is particularly useful for webdataset-style data sets, but can also be used with other dictionary iterators.

tmbdev avatar May 13 '22 19:05 tmbdev

Please switch the order of inputs 'pattern' -> 'new name' looks more natural

VitalyFedyunin avatar May 19 '22 20:05 VitalyFedyunin

The usual usage is with keyword arguments using a simple key as output and a pattern as input. It also parallels assignment. I think this order is more useful. What do you think?

tmbdev avatar May 20 '22 00:05 tmbdev

In my opinion it makes sense to have two datapipes: pattern_filter_keys -> takes patterns, throws away all missmatch keys #406 and pattern_rename_keys -> takes pattern->new_name dictionary and renames keys accordingly. In this case they will follow same API patterns and would be easy to remember.

VitalyFedyunin avatar May 20 '22 01:05 VitalyFedyunin