[Feat]: Partial Re-Caching

[shawndolen]

Describe your use-case.

Right now, caching appears to be all or nothing, if you change the source dataset(s) you are forced to recache the entire concept otherwise it will error out trying to load the cache. This is annoying because if you make changes to your concepts as you're testing a training idea, you will be forced to spend time recaching all your images instead of the new ones. In addition, while caching, if there is a corrupted or invalid image, the entire process will crash and you will lose progress on your entire caching session, forcing you to retry the entire process again (after removing the bad image), this can be hours of wasted time on a large dataset.

What would you like to see as a solution?

The cache system should pay attention to which images it has already cached and attempt to only update the cache for new files. While caching, the system should keep its progress in the event of an error, it's likely these two features comingle functionally.

Have you considered alternatives? List them here.

No response