datachain
datachain copied to clipboard
Make `.get_storage()` caching listing lazy
This is follow up for PR https://github.com/iterative/datachain/pull/294 where listing is saved (cached) inside .from_storage() method itself which means it's not lazy unlike all the other DataChain steps.
We should think how to postpone caching / saving listing when all chain steps are applied.
Ideas:
- Make some generic step that accepts function (closure) in
DatasetQuery - Create
.terminate()onDataChainclass - ...