Davis Wertheimer

Updating the dataloader to match the latest version in our public repo. Re-syncs to latest main, and incorporates many previously discussed features: - Remove dependence on the separate metadata "count...

Checking on the status of this - the memory consumption ended up being related to how the legal-file-detection was working IIRC?

Closing with #107

Hey sorry for late response, this came in just before winter holiday for us! We don't currently support JSON format, but we do support users implementing their own data formats....

Thanks @scotts , I made changes 2-4 and working on unit tests now. I'll note that `_StatefulDataset` and `_NestedStatefulDataset` largely represent legacy code, gluing things together until we decide we...

Some thoughts on point 2 @divyanshk : we could definitely separate this out into a file-interface system, plus a separate rescaling-enabled interface between nested iterable Datasets and generic indexable structures....

Hi @zhouyizhuo - I'm not entirely sure how to read your dataset info (16 training classes and 382 total images?) but this appears to be a VERY small dataset, given...

Nice, this does make more sense once models are partitioned into headless/head components!

Thanks @rualark for bringing this up! Yes the repeated `os.walk` is a pain point with respect to setup times. Unfortunately I don't think it's feasible to execute the walk only...

I thought through this a bit more and I think I have a framework for handling these behaviors with minimal demands on DCP support changes. Additionally, it also supports checkpoint...