`noosphere-cli` workspace needs to support ~millions of content files across sphere graph
As of #530 , we now render and synchronize the broader Noosphere graph to the file system when using orb. As a basic implementation, this works great. However, some of the implementation may run afoul of known file system limitations related to the maximum number of files in a directory (see: https://stackoverflow.com/a/466596).
Granted, if we get into the millions of files we're going to start having other kinds of performance problems. But, in the case of file system limitations, we can probably do better than what we do today to avoid them. And, it's worth noting that in the case of .sphere/content/*, we're talking about the union of all content across all peers being rendered by orb, so it could quickly become a lot of files if, say, someone renders with a large maximum traversal depth.
We should probably begin to address this by partitioning files in the .sphere/content and .sphere/slug directories by breaking the filenames up into chunks of fixed length and building out a directory hierarchy by those chunks. In the case of .sphere/content, we already store those files by their CID, and this kind of name will naturally work well with this form of partitioning. In the case of slugs, we currently base64-encode the UTF-8 bytes, and this means the file names have varying lengths; we probably want to switch to a hash so that we can benefit from fixed length names instead.