Making ecryptfs work, or how to reuse all ready open file handles

[bgemmill]

Hello all, I'm trying to make ecryptfs (note: not encfs) work on top of acd_cli, and noticed that it has a way of writing to files that makes acd_cli unhappy.

To reproduce, I'm using a setup like this:

acd_cli mount /tmp/acd-raw
mount -t ecryptfs /tmp/acd-raw /tmp/acd-decrypted

And attempts to copy files in give offset errors:

root@f3c6ad3a9f88 /tmp/acd-decrypted # cp ~/fnord.txt .
cp: error writing './fnord.txt': Illegal seek
cp: failed to close './fnord.txt': Remote I/O error

Now this is funny, because ecryptfs works just fine on my home directory. Looking at the acd logs show the funny business when we grep for write, open, and release:

16-08-01 17:10:10.378 [DEBUG] [acdcli.acd_fuse] - -> open /ECRYPTFS_FNEK_ENCRYPTED.hashed-file-name-here--- ('0x8002',)
16-08-01 17:10:10.380 [DEBUG] [acdcli.acd_fuse] - <- open 42
16-08-01 17:10:10.384 [DEBUG] [acdcli.acd_fuse] - -> write /ECRYPTFS_FNEK_ENCRYPTED.hashed-file-name-here--- (8192, 0, 42)
16-08-01 17:10:10.385 [DEBUG] [acdcli.acd_fuse] - <- write 8192
16-08-01 17:10:10.387 [DEBUG] [acdcli.acd_fuse] - -> open /ECRYPTFS_FNEK_ENCRYPTED.hashed-file-name-here--- ('0x8002',)
16-08-01 17:10:10.389 [DEBUG] [acdcli.acd_fuse] - <- open 43
16-08-01 17:10:10.390 [DEBUG] [acdcli.acd_fuse] - -> release /ECRYPTFS_FNEK_ENCRYPTED.hashed-file-name-here--- (42,)
16-08-01 17:10:10.397 [DEBUG] [acdcli.acd_fuse] - -> write /ECRYPTFS_FNEK_ENCRYPTED.hashed-file-name-here--- (4096, 8192, 43)
16-08-01 17:10:10.397 [ERROR] [acdcli.acd_fuse] - Wrong offset for writing to fh 43.
16-08-01 17:10:10.398 [DEBUG] [acdcli.acd_fuse] - <- write '[Errno 29] Illegal seek'

We see file handle 42 trying to write the first 8192 bytes, and before it's closed, we see file handle 43 trying to write the next 4096 bytes. Without looking at ecryptfs code, this feels like mmap crawling along.

Because both 42 isn't closed yet, and the next write begins with the proper offset, it looks like one way to solve this would be open/close reference counting on the file handles. That way 42 and 43 would both get the same buffer and it's offset, and this would work as intended.

I'm happy to have a go at the python if this sounds like a good idea.

[cyberbalsa]

Its a Object based store over the internet, the most we can try to do it buffer everything and prepare it all

or have encryption built into acd_cli

Also, may I suggest smoothfs and how it functions

[bgemmill]

@jrwr we can do a bit better than that with ecryptfs's ecryptfs_xattr flag. The checksum is computed and stored in the file's user.ecryptfs xattr, rather than the file contents.

This lets us write the encrypted file all at once front to back, if we can be lazy about when xattrs are sent. I currently have them waiting until either all file handles are closed or fuse unloads.

[bgemmill]

PR is in https://github.com/yadayada/acd_cli/pull/374