geesefs
geesefs copied to clipboard
yandex-cloud
Hard links support
Thanks @vitalif , really cool driver, performance is great! Is there any plans to support hard links? sometimes its really required.
I don't think so... Proper support of hard links requires separation of 'inodes' and 'files' and inode reference counting and it's hard to do it in S3. The only way in which I see it can be implemented is if hard links will actually act as 'internal symlinks' and be rewritten when the "original" file is renamed. But this would require a way to quickly find all secondary links. Listing all files in S3 to find links is a very bad solution so the list of secondary links would also have to be stored in the metadata of the primary link. All of this sounds like a very cumbersome implementation so I don't think we'll do it in the near future. But if you really want it you can try to implement it on your own and submit a PR :)
P.S: It also sounds similar to deduplication, we may return to this idea when we think about deduplication
I'm trying to use S3 as remote storage for ClickHouse backups. ClickHouse backups heavily use hard links and its critical part of this solution.
@vitalif Unfortunately I dont have enough knowledge of Posix and Go to implement it. May be its possible to use another approach - keep metadata locally in some key value storage ( RocksDB, SQLite) and upload this file to S3 before unmount and download before mount. Something similar is used in this project https://www.rath.org/s3ql-docs/impl_details.html#metadata-storage
It will add some restrictions ( no simultaneous mount of same bucket to multiple hosts) but in most scenarios its not used.