MLUtils.jl
MLUtils.jl copied to clipboard
Port collated batch view from DataLoaders.jl
This is the first part of porting functionality from DataLoaders.jl (ref #22).
This includes porting
-
BatchViewCollated
frombatchview.jl
- Notes: the
BatchDim
machinery can probably be dropped if we can assume last dim = batch dim
- Notes: the
-
collate
from [https://github.com/lorenzoh/DataLoaders.jl/blob/master/src/collate.jl
]
batchview.jl
also includes an unimported batchsize
helper. Do we want to have a proper batchsize
?
I think collate
could be entirely replaced by batch
(one of the reasons that motivated me in doing #27)
Also, do we need something like collate? Why don't we just call getobs(data, idxs)
?
Sometimes you want a batch view that gives you a vector of observations (e.g. different-sized images), and sometimes you'll want to have them as a single array (or recursively collate observations in tuples/dicts...). For this, having something like BatchViewCollated
is essential since it allows buffered loading of observations with getobs!
.
@lorenzoh is this still necessary?
Yup 👍 This is necessary for porting the buffered batch view from DataLoaders.jl