Support FAT32 max folder size limit in DiskLruCache

[vitidev]

The disk cache creates 2 files with a long name for each cached item. In my case the cache is placed on sd-card with FAT32. Many small images and cache has reached limit of "number of files per folder in FAT32" (65k entries for 8.3 names, but the long name takes a few entries). At the same time, cache size was only 30 mb.

Need the ability to set cached files to be stored in subfolders

[colinrtwhite]

The complexity of caching across multiple directories is pretty high and isn’t likely to be implemented. Seems like we could also work around this limitation by supporting short file names. Will need more investigation.

[vitidev]

The complexity of caching across multiple directories is pretty high

Mmm. Key management and journal remains the same, but change the final locations. Something like this

DiskLruCache > Entry

 val useSubDirs = true // => DiskCache.Builder().useSubDirectories()

 init {
        // The names are repetitive so re-use the same builder to avoid allocations.
        val fileBuilder = StringBuilder(key).append('.')
        val truncateTo = fileBuilder.length

        val filesDirectory=
            if (useSubDirs) directory / fileBuilder.substring(0, 2) else directory

        for (i in 0 until valueCount) {
            fileBuilder.append(i)
            cleanFiles += filesDirectory/ fileBuilder.toString()
            fileBuilder.append(".tmp")
            dirtyFiles += filesDirectory/ fileBuilder.toString()
            fileBuilder.setLength(truncateTo)
        }

        if (useSubDirs && !fileSystem.exists(filesDirectory))
            fileSystem.createDirectory(filesDirectory)
    }

It just works. And since additional configuration is used, nothing breaks for existing users.

A full cleanup will leave empty folders... but that's okay

[colinrtwhite]

I don’t think we can rely on the first two characters of the file name being unique for the directory’s name.

Also I’d check out Okio’s FileSystem and ForwardingFileSystem. DiskCache accepts a custom FileSystem and you might be able to transform the file name/location using ForwardingFileSystem.

[vitidev]

I don’t think we can rely on the first two characters of the file name being unique for the directory’s name.

What kind of uniqueness are we talking about? This is one of standard way to solve the "too many files in folder" problem. Actively used on the web.

You are using sha256 for the key in hex. With 1 character, that's only 16 folders (i.e., the cache file limit grows by 16), but this is ideal, but in reality the distribution is not even, so 2 characters is 256 folders, i.e. "with a reserve" and not so much

the file name/location using ForwardingFileSystem.

Perhaps this can help, but there is a dependence on the internal implementation of RealDiskCache + DiskLruCache, which can change and break the application