WIP: new "virtual" folder type allowing to use cloud based block storage backends like aws S3

[cre4ture]

Purpose

Miro-Board with sketches

Cloud Storage Integration

I started this project due to a need of mine to integrate syncthing into a private cloud environment based on the opensource project garage. For this, it would be useful if syncthing could directly use a cloud API like S3 to store the data of the hashed blocks.

I have the impression that the strategy of storing the data in a hash block storage rather than in a file-based storage makes the implementation in general easier and avoids data duplication automatically.

A syncthing node that runs on the cloud anyway doesn't need to provide access to the files directly, and therefore, the storage system is not visible to the user and thus we can choose whatever we want there.

This PR partially solves this issue: #8113 It's only partially, because currently the virtual folders implementation doesn't support the receive encrypted feature, which is probably an issue for some use cases where no private cloud is used.

Data-Scrubbing for bit-rot prevention

Even though it is a completely different use-case, the implementation for it has some overlapping parts. In the syncthing forum I found two discussions. This is the link to the up-to-date one: Forum: bit-rot-prevention

The desired feature there is to have an automatic bit-rot detection and correction feature inside syncthing. In contrast to cloud-storage systems like garage, that have their own integrated bit-rot correction, local disk filesystems usually don't include checksums and automated correction. E.g. typical filesystems like NTFS and Ext2/3/4 do not have it. More complex filesystems like ZFS and BTRFS are usually not used on end-user devices like desktop PCs, notebooks, handys or tablets.

The big issue with providing the bit-rot correction feature is the problem of differentiating desired, good changes in the data from undesired, bit-rot caused, bad changes.

This PR included initially two different strategies (one now split to #9698) that enables both separately the detection of undesired, bad changes. Both strategies are based on providing a fuse-based filesystem mount for syncthing folder data. This way, the user is not accessing the data directly via the native filesystem but instead through the fuse-fs-syncthing mount.

Any read is directly forwarded to the storage backend. Any write or modification is handled explicitly and specially by the syncthing versioning logic. This way, all intentional, good changes are directly visible to syncthing. And any change that happens to the data in the storage backend without the interaction with syncthing can be classified as an unintended, bad change. These bad changes are to be detected by regular full-data scans that compare the actual and the expected hash of the data. After detection, the data can be corrected by downloading the data-block from one of the connected other devices.

Current State of the PR

I know that there was already a project that combines FUSE with syncthing: SyncthingFUSE. This is archived and no longer maintained. I had a look into the source code if it's possible to resurrect this code somehow. But I had to learn that the code included way too much code duplication with the original syncthing code. Updating all of this is a lot of effort. Considering that its already outdated for years, I decided it's better to start from scratch.

And I was right. The FUSE support on this PR includes not only reads as syncthingFUSE, but also creates, writes and deletes. And it's directly integrated into the official syncthing code such that minimal, to no, code duplication is achieved. Currently it downloads missing blocks for files on-demand when they are accessed. When explicitly triggering a scan via the GUI, all missing blocks for all files in the global state are pulled.

I decided to publish this PR here, even though the state is still experimental, just to get some early feedback from the maintainers. Especially feedback about the changes in the original syncthing codebase would be great. I would like to merge them in via cherry-picking as soon as possible to avoid future efforts on merging.

Testing

So far, just manual testing.

To use the feature based on the data-block-storage for cloud, one needs to create a new folder with the type "virtual". IMPORTANT: The virtual folder needs two additional parameters:

1. URL for storage backend
2. Mount path for the fuse based mount As I didn't spend time on extending the GUI more, one needs to use the path config value for the virtual folder to encode these two parameters. :virtual: and :mount_at: are used for this. E.g.:

:virtual:s3://bucket-syncthing-uli-virtual-folder-test1/Rfuse_share2:mount_at:/home/uli/nosync-data/Rfuse_share2

Additionally one needs to create the configuration files for aws: ... as described here

I've chosen to use the package gocloud for the access to the S3 API. This has the benefit that also other cloud storage systems are automatically supported, by using a different URL. It also allows to actually use local filesystem for storage, in case where no cloud itegration is desired. E.g.

:virtual:file:///home/uli/nosync-data/Rfuse_share2_BlockStorage:mount_at:/home/uli/nosync-data/Rfuse_share2

Currently, the directories need to be created manually beforehand. This will be improved soon.

The storage backend is not used to store the data as content of regular files as syncthing traditionally does it. Instead, it is more efficient to store each of the data blocks separately, identified by its hash-value. This way, there is no need for regular scanning/listing of the S3 bucket content, which has been proven to be very slow and thus inefficient.

Screenshots

Nautilus shows mounts from syncthing:

Mount cmd line tool shows syncthing mounts:

Listing the s3-bucket content:

Documentation

TODO

[calmh]

Awesome, cool stuff, I did not look at the details at all yet, but one question: does this inherently need to be a new folder type, or could it just be a filesystem type?

[cre4ture]

Awesome, cool stuff, I did not look at the details at all yet, but one question: does this inherently need to be a new folder type, or could it just be a filesystem type?

Thanks for the quick first feedback. Regarding your question: Yes+No. I still need to split the PR into two as it contains two seperate features. One of it can be handled more or less completely by using a new filesystem type. The other one can be handled more or less completely by using a new folder type.

When I started with my experiment, I was a bit undecided about which is the best. Thats why there are two on this PR branch. But I understood that this is confusing. So I will split the two features into two PRs.

[imsodin]

As usual, I am starting with the important stuff (attention, irony :P ): Just opened the PR and first thing github put in front of me is a diff removing a copyright header: https://github.com/syncthing/syncthing/pull/9691/files/ef2b3ba7e74bf78f024bb9059b238742828b9b32..61635016b1e5b6244386c7af7a3d223a91896bcc#diff-7e6f85da36a4c31f5d425c55f3a89f46827757bd225c5749ec0d556ca11364b2 I assume you started from their code and adjust it, heavily potentially. Nevertheless that means you used their code, so you can't remove copyright. I'd need to check what exactly you got to do for BSD, but not much afaik - copyright header though are sacrosanct on almost any license.

[cre4ture]

As usual, I am starting with the important stuff (attention, irony :P ): Just opened the PR and first thing github put in front of me is a diff removing a copyright header: https://github.com/syncthing/syncthing/pull/9691/files/ef2b3ba7e74bf78f024bb9059b238742828b9b32..61635016b1e5b6244386c7af7a3d223a91896bcc#diff-7e6f85da36a4c31f5d425c55f3a89f46827757bd225c5749ec0d556ca11364b2 I assume you started from their code and adjust it, heavily potentially. Nevertheless that means you used their code, so you can't remove copyright. I'd need to check what exactly you got to do for BSD, but not much afaik - copyright header though are sacrosanct on almost any license.

Thanks for pointing this out. You where luky to see the diff of my last change. Its actually true. I copied one of the example of go-fuse. I've seen the CI complaining about missing copyright statements in some of my files. So I started to add them. I was a bit in a rush and thus I didn't realise that the previous copyright statement is actually not from syncthing but from go-fuse. Sorry for this.

Meanwhile i cleaned up some of the files as they are not needed. For the remaining files, that I really did modify a lot I'm not sure what to do. Shall I keep the original and additionally place the new?

[imsodin]

Meanwhile i cleaned up some of the files as they are not needed. For the remaining files, that I really did modify a lot I'm not sure what to do. Shall I keep the original and additionally place the new?

Yep that's the way (in absence of more specific requirements by the license, which afaik isn't the case with BSD).

[imsodin]

Just because I already commented, to avoid incorrect expectations: I don't personally have capacity to review this for the foreseeable future. I am aware that's not very motivating statement to make, to put it mildly, and I am sorry about that - I feel it's still better than to just not review. Obviously there are others that can pick this up, this isn't blocked on my review.

Also this is definitely the kind of change (large and impactful) that first requires a high level review, mainly evaluating the broad design and it's suitability/maintainability within the Syncthing project. So anyone wanting to review, please don't jump into details yet until that's cleared. Otherwise a lot of time/effort might be invested, and then the verdict is still that it can't be included.

[calmh]

I can take point on reviewing. For that to happen I would like it to become a bit more focused. Currently I see blob storage (though called block storage, which I think is generally the opposite, though perhaps it's because we store blocks as blobs?), plus some sort of bitrot scrubbing, plus some sort of fuse integration, which are three different things where I think only the first one has a good chance to be integrated as-is.

For the blob storage it would ideally be "just" a new filesystem type -- albeit one with a bunch of settings and possibly a local cache area and whatnot, however one wants to handle that.

[cre4ture]

I can take point on reviewing. For that to happen I would like it to become a bit more focused. Currently I see blob storage (though called block storage, which I think is generally the opposite, though perhaps it's because we store blocks as blobs?), plus some sort of bitrot scrubbing, plus some sort of fuse integration, which are three different things where I think only the first one has a good chance to be integrated as-is.

For the blob storage it would ideally be "just" a new filesystem type -- albeit one with a bunch of settings and possibly a local cache area and whatnot, however one wants to handle that.

Thanks in advance for your review. To make it easier I followed your proposal and reverted the parts not related to the S3 supporting block storage. The reverted parts are fully about FUSE mount support. To add bit-rot scrubbing is my plan, but I didn't start with this yet.

BUT I have to point out that the solution I've chosen is probably different to what you currently expect. I've seen the code that exist in the code base about handling all the different corner-cases for dealing correctly with the complexity of a filesystem that is used for storing the data-blocks as parts of copies of the original remote-files. This complexity is one aspect that makes syncthing so powerful and convenient on end-user devices. But its not needed when we chose a different method to store the data-blocks. E.g. as part of a database. Or, as suggested in this PR, as part of a S3 block storage that stores block data addressed by hash rather than filename.

Additionally, for me, who doesn't know the code that well, it was much easier to implement this S3 block-storage solution by introduction of new folder instead of new filesystem type. And I still have the feeling that this was a good choice.

Best regards, cre4ture

[calmh]

Yeah I can see there will be some assumption mismatches between file systems and blob storage. Most likely some folder code needs to change. However I think it would make sense to be able to have folders be regular, receive-only, or send-only with blob storage as well. I don't think it would be nice to have three new folder types send-receive-blobs, receive-only-blobs and send-only-blobs. Hence I think some effort should go into not making this a new folder type.

[cre4ture]

I don't think it would be nice to have three new folder types send-receive-blobs, receive-only-blobs and send-only-blobs.

Please help me to understand this point of view. Why exactly would it be a problem to have these 3 new folder types?

Is it something in the GUI? Would it be OK if we hide de three new folder types behind a checkbox "virtual"(yes/no)?

Or is it about an issue on source code level? Code duplication? Maintenance effort?

Other topic: I created a use-case map on the Miro board here. And I added some diagrams that should give a better insight into the currently proposed solution.

One important aspect on the usecase map is, that in a first step the "encrypted receive only" virtual folder would be most important. The other variants could be omitted (low prio) for the S3 feature.

The other diagrams should point on how to implement the "multiple node using same storage" - feature.

Hope you find them useful.

Br cre4ture

[calmh]

Please help me to understand this point of view. Why exactly would it be a problem to have these 3 new folder types? Is it something in the GUI? Would it be OK if we hide de three new folder types behind a checkbox "virtual"(yes/no)? Or is it about an issue on source code level? Code duplication? Maintenance effort?

All of the above. The behavior is one dimension, the kind of storage is another dimension, mixing them should not be required.

[cre4ture]

Please help me to understand this point of view. Why exactly would it be a problem to have these 3 new folder types? Is it something in the GUI? Would it be OK if we hide de three new folder types behind a checkbox "virtual"(yes/no)? Or is it about an issue on source code level? Code duplication? Maintenance effort?

All of the above. The behavior is one dimension, the kind of storage is another dimension, mixing them should not be required.

I think I understood the general problem. But what I actually tried to achieve with this more detailed question, is that you would try to understand that the advantages of the proposed solution might be worth accepting the disadvantages of introduction of a new folder type. So its about putting the weight in relation.

What I get from your rather short answer is, that one is not willing to to generate a common understanding. This makes me feel sad. :-/

[calmh]

Again, I see no advantage in adding new different folder types -- folder types are about behaviour, not underlying storage mechanisms. I realise some code in all folder types may need to be modified to take the different behaviour of a blob storage backend into account, though. If this is not possible then of course we can look at why that's the case, but so far I haven't seen anything to indicate that would be the case?

[cre4ture]

Again, I see no advantage in adding new different folder types -- folder types are about behaviour, not underlying storage mechanisms. I realise some code in all folder types may need to be modified to take the different behaviour of a blob storage backend into account, though. If this is not possible then of course we can look at why that's the case, but so far I haven't seen anything to indicate that would be the case?

• The complexity of the existing code, especially of the SendReceive folder, which seems to be the base for all currently existing folder types, is daunting, at least for me. I don't see any reason why it would make sense to build the new functionality ontop of such a large and actually unneeded complexity. My proposed changes in the PR should demonstrate that this perspective is realistic. I would be glad that you argue on a more concrete abstraction level.

• Additionally, we got already a bit of feedback from another mmaintainer, Insodin. What I got from his input is that my approach is actually going into the right direction. Here are the links to input and response: https://github.com/syncthing/syncthing/pull/9698#issuecomment-2340218467 https://github.com/syncthing/syncthing/pull/9698#issuecomment-2340565330 Please have a look and consider this in the solution rating process.

By the way, meanwhile i could make the encrypted virtual folder working on this branch in my fork: https://github.com/cre4ture/syncthing/pull/2 Apart of the missing automated test, I see the feature as almost complete. Qualtiy improvments are still needed, a proof of concept is done.

I'm glad for your time spending to respond on the PR. Hope we can find a solution that makes sense to us both.

Br cre4ture

[acolomb]

Just to give you some perspective, I too value the clean separation of responsibility in the Syncthing code base. The folder types define different logic how changes are propagated and whether encryption / decryption happens. The lower layer, the abstracted file system, deals with different methods of storing file data. These two are orthogonal and should stay separated. Your feature clearly belongs in the storage layer and should be compatible with (and independent from) all of the existing folder types.

I understand that means more work for rearchitecting your proof of concept. But please understand that from our perspective, maintainability is more important. That means if someone knows the folder code very well, they should not have to deal with another four types involving block storage backends when investigating issues or changes. It should not matter what type of storage is used on the lower layer. On the other side, optimizing storage code should not have to touch several folder type implementations.

The interface between the two is the Filesystem abstraction, which might need some adaptations to fully support block-based storage. That may lead to necessary adjustments in the folder code and tests as well, so you'll need to familiarize yourself with some of it. But I don't see a way of this getting merged without reworking to separate concerns and put it on the right layer. I hope you understand and are still willing to invest the time needed. I'm sure we can help you out if some things are hard to understand. Best open a topic on the forum, in the Development category, for such discussions.

[cre4ture]

How can one connect a interface like this:

type HashBlockStorageI interface {
	io.Closer
	Get(hash []byte) (data []byte, ok bool)
	Set(hash []byte, data []byte)
	Delete(hash []byte)
}

to an interface like that:

// The Filesystem interface abstracts access to the file system.
type Filesystem interface {
	Chmod(name string, mode FileMode) error
	Lchown(name string, uid, gid string) error // uid/gid as strings; numeric on POSIX, SID on Windows, like in os/user package
	Chtimes(name string, atime time.Time, mtime time.Time) error
	Create(name string) (File, error)
	CreateSymlink(target, name string) error
	DirNames(name string) ([]string, error)
	Lstat(name string) (FileInfo, error)
	Mkdir(name string, perm FileMode) error
	MkdirAll(name string, perm FileMode) error
	Open(name string) (File, error)
	OpenFile(name string, flags int, mode FileMode) (File, error)
	ReadSymlink(name string) (string, error)
	Remove(name string) error
	RemoveAll(name string) error
	Rename(oldname, newname string) error
	Stat(name string) (FileInfo, error)
	SymlinksSupported() bool
	Walk(name string, walkFn WalkFunc) error
	// If setup fails, returns non-nil error, and if afterwards a fatal (!)
	// error occurs, sends that error on the channel. Afterwards this watch
	// can be considered stopped.
	Watch(path string, ignore Matcher, ctx context.Context, ignorePerms bool) (<-chan Event, <-chan error, error)
	Hide(name string) error
	Unhide(name string) error
	Glob(pattern string) ([]string, error)
	Roots() ([]string, error)
	Usage(name string) (Usage, error)
	Type() FilesystemType
	URI() string
	Options() []Option
	SameFile(fi1, fi2 FileInfo) bool
	PlatformData(name string, withOwnership, withXattrs bool, xattrFilter XattrFilter) (protocol.PlatformData, error)
	GetXattr(name string, xattrFilter XattrFilter) ([]protocol.Xattr, error)
	SetXattr(path string, xattrs []protocol.Xattr, xattrFilter XattrFilter) error

	// Used for unwrapping things
	underlying() (Filesystem, bool)
	wrapperType() filesystemWrapperType
}

? @acolomb @calmh @imsodin

[calmh]

Clearly some form of middle layer is required. In practice you probably need to do something similar to how fuse interfaces towards blob storage works, that is, cache files on disk and upload on close etc. In any case some form of system needs to be in place to store filenames, directories, metadata, etc in blob storage, or the data would be useless without the Syncthing database to hold metadata.

[cre4ture]

Clearly some form of middle layer is required. In practice you probably need to do something similar to how fuse interfaces towards blob storage works, that is, cache files on disk and upload on close etc. In any case some form of system needs to be in place to store filenames, directories, metadata, etc in blob storage, or the data would be useless without the Syncthing database to hold metadata.

References

• rclone - command line tool that also allows fuse mount of cloud storage

• s3fs - fuse based filesystem from mounting S3 buckets

• CSI for S3 - Kubernetes CSI driver

• goofys - similar to s3fs but has better performance and less POSIX compatibility

• s3backer - mount an S3 bucket as a single file

• S3Proxy - combine with s3fs to mount Backblaze B2, EMC Atmos, Microsoft Azure, and OpenStack Swift buckets

• s3ql - similar to s3fs but uses its own object format

• YAS3FS - similar to s3fs but uses SNS to allow multiple clients to mount a bucket

In issue #8113 are some of these existing solutions mentioned and disadvantages are listed. I personally, and it seems also @imsodin can confirm that these solutions mainly suffer from bad performance regarding listing m´folder content.. I didn't test all of these exsiting solutions. Maybe one could have a look also on "s3ql".

I dont't think that the solution for syncthing is to replicate the already existing solutions. We will probably run into the same already known issues. I think there is no significant advantage of a solution like "s3fs" directly integrated into syncthing.

[calmh]

So, what exactly is it you want to accomplish, use-case wise? I think it would be useful to be able to store a Syncthing folder in S3. The way I see it, such a thing would have approximately these properties:

• It's something you can enter as a path for a folder in Syncthing somehow, similar to how we can enter a local filesystem path today
• Objects in the object storage correspond roughly to files. That is, a file called foo/bar.txt on disk would be synced to an object called foo/bar.txt. This has two desirable sub-properties:
  • If the bucket is public, files written there by Syncthing are accessible directly on a predictable URL, so Syncthing can be used to publish things to the web
  • Things written to the object storage by someone else / something else become discovered by Syncthing and synced to other devices
• If my Syncthing instance breaks, I can just set up another Syncthing instance anywhere and point it at the same object endpoint and it'll be able to sync those files to other devices.
• In fact, I could probably point multiple Syncthing instances at the same object storage, and changes by one would be discovered by the other(s).

Perhaps you're thinking of something else?

[acolomb]

AIUI, the envisioned solution was to just store what Syncthing calls blocks, with the block hash as name / key and the block content as value. That is pretty efficient, as it leads to automatic deduplication on block level. However, it leaves the metadata problem unsolved.

Thus the plan sketched by @calmh sounds more reasonable to me (although so far it doesn't solve the metadata storage either). Any idea how that would compare regarding data deduplication?

As for the metadata storage, an easy solution would be to split the object store into a files/ prefix and a metadata/ prefix with the same tree structure in object names, but custom protobuf-encoded content.

[calmh]

Right, the precise storage format can be determined, but I think the first question is the what and why. That is, what feature are we adding, for which purposes/use cases. I'm not sure that storing blocks individually by hash and relying on the database for metadata, for example, is generally useful. At least not as useful as it could be.

[cre4ture]

Regarding the use-cases, I'm orienting myself on the use-cases listed in issue #8113. Short summary:

• cheaper storage
• higher bandwidth
• redundancy of nodes (with re-using same cloud bucket if possible)
• untrusted node (encrypted store and receive)

What is NOT mentioned:

• publish things to the web
• writes to the object storage by someone/something else than syncthing
• adding a syncthing node to the cluster after every existing node failed

With the solution proposed in this PR, we can cover all the explicitly mentioned use-cases with a minimum effort and a minimum of additional code. So, from a maintenance perspective this should be quite close to ideal. Yes, I agree that there are limitations for other not mentioned use-cases. These limitations are partially solvable by adding the Syncthing-FUSE-Feature that was initially also part of this PR. But which is currently reverted.

• FUSE mount can be used to access the file-data in a regular way using the syncthing integrated database to lookup all needed block hashes and all needed metadata.
• As an extension, FUSE mount can also be used to change/add files to the database if this is desired and the data is not encrypted. But I see this as low priority feature.

Finally, the proposed solution has additional bonus-features:

• automatic data de-duplication as a block with the same hash is only stored once
• with multiple nodes: automatic skipping of redundant download of the same data, as first we check the S3 bucket if some other node already placed a block with same hash.
• data integrity (bit-rot) can be checked and repaired with regular re-hashing and hash-validation of the stored blocks (useful especially for storage that is not already bit-rot protected)

I don't see it mandatory to store also the metadata in S3 as its already, even redundantly stored on the synchting nodes. For the exceptional case where all nodes fail, one could implement with low effort a regular backup of the syncthing-databases (the folder part) of the nodes to the S3 storage bucket.

There is only one still challenging point for the multi-node use-case: Safe deletion of unneeded blocks. Some kind of cloud garbage collection will be needed.

[calmh]

Short summary:

cheaper storage higher bandwidth redundancy of nodes (with re-using same cloud bucket if possible) untrusted node (encrypted store and receive)

These are not really "use cases" as I see it, rather just technical aspects of a solution. Except maybe for number three (redundancy of nodes), which is valid. When I speak of use cases I mean what things does it enable us to do that we cannot currently do with Syncthing. "Behave exactly the same as today, but with cheaper storage for cloud setups" is potentially one such, but it's not terribly compelling to be honest. If we're going to add cloud storage as a thing in Syncthing I'd rather we looked a bit wider and thought "ok, if we do that, what more can we enable if we do it properly?". Things like the points I mentioned earlier:

• Proper node redundancy and failover, Kubernetes style: if a node fails or you need more, just spin up another one and point it to the same cloud storage, and it will scan it and start serving. (The scanning step is not ideal here, it would be better to get the metadata loaded faster, but it would at least be possible.)
• Interaction with other systems, who can either read or put blobs into the cloud storage and have them be part of what Syncthing sees.
• A better story for backup and restore, with whatever functionality the cloud service has for versioning and soft delete of objects. This means Syncthing must be able to see and use the data when it's reverted etc.
• ... probably other things my imagination is not grabbing at the moment.

Implementing something that is technically S3 storage but doesn't take the opportunity to address these things would be missing the goal, imho.

[cre4ture]

"Behave exactly the same as today, but with cheaper storage for cloud setups"

I read it differently. And I personally need it the way I understood it. Probably everyone of us has this bias. How do we figure out what is really needed?

[calmh]

I think this discussion here is figuring it out, but we could also do the same as a forum thread or in the original issue.

[cre4ture]

I updated the code to not require a new folder type in the user interface nor the config file. Instead its decided just based on the path-url. when it starts with ":virtual:" a folder using the new hash-block storage is instantiated.

In future, I might look into a variant that uses the cloud storage rather like a normal filesystem. But for the encrypted case, I think the hash-block storage is better suited in any case.

[cre4ture]

I started with a implementation for using go-cloud as "normal" filesystem storage. I stumbled over the issue that S3 and other cloud storage types do not allow to edit existing files. This limitation makes the following operations very costly:

1. Truncate a file to a smaller size (file content needs to be downloaded, cut and uploaded completely again)
2. "WriteAt" a position of the file expecting that other parts of the file will not be changed. this also causes a complete upload of the file contents again.

While this is not an issue for small files, considering files with the size of many GBytes would be a severe issue, I guess. Of course, caching changes locally and upload the file content only after all changes are complete could make it somewhat feasible again. Still, considering the idea of having multiple syncthing instances using the same cloud storage would not work out of the box as the uploaded data is only available to others when the whole file content is uploaded completely. So one would need to have a synchronization mechanism that prevents other instances from changing the file while one is uploading data.

The hash-block storage idea does not suffer from this at all.

Are there any ideas on how to deal which those issues in a smarter way?

[bt90]

Shoehorning S3 into a filesystem interface is a really bad idea. It's just an object store, and lacks most of the functionality that a real filesystem would offer.

I agree with @cre4ture that we should use a different abstraction for it.

And I'm not alone in my opinion:

https://charemza.name/blog/posts/s3/s3-is-not-a-filesystem/ https://calpaterson.com/s3.html

[acolomb]

Are there any ideas on how to deal which those issues in a smarter way?

Not really thought through yet, but I want to shout out the idea early for discussion. If we want to implement a separate folder type, it basically needs a Filesystem abstraction to use, for integrating with the existing model and folder code. But for various reasons stated above, abusing the object storage as a filesystem (file = object) might be a bad idea. And we don't want to rely solely on the local database to hold the file tree to blocks mapping. So let's look at these concerns separately:

1. Store blocks as objects in the S3 backend, under their hash.
2. Mirror parts of the local database to the S3 backend as well in some way, to have a persistent folder tree somewhere else than the local DB (which might get lost easily).
3. Implement a new virtual Filesystem type that combines the two sources into a coherent file access API. Reading a file means collecting blocks as enumerated in the mapping. Writing means uploading the blocks first and then creating a new blocks-to-file mapping entry and persisting that to the DB and backend as well. Changing a file happens anyway by replacing individual blocks in the associated index entry. Accessing metadata touches only the file's mapping entry.
4. Have a new folder type that only accepts this new type of virtual Filesystem.

In the current PR state, I understand that the metadata and blocks mapping is stored only in the local index DB, so point 2 is not solved. We could even decouple the mapping storage from our local DB, and use some other mechanism to store that data in S3 or elsewhere. The index would then be handled just as for regular local folders again.

To integrate this nicely into Syncthing's architecture, I guess point 3 above is the most important: It needs to live in the fs layer, which obviously creates difficult dependency relations when the Filesystem in turn needs the index DB for its metadata storage. If we decouple that and ignore the index DB in context of the backend-blobs to file tree mapping, then there is no reason anymore to lift the feature up onto the folder abstraction layer. That's about where point 4 (new folder type) becomes unnecessary.

Sorry if I got some things mixed up, I'm a bit fuzzy on the existing Sycnthing architecture as well. But hopefully this will still give some inspiration.

[bt90]

The naïve approach would be to have an object which stores all the filesystem metadata and maybe even precomputed block hashes. If we really want to mimic a POSIX filesystem.