mergerfs
mergerfs copied to clipboard
trapexit
Mac updates
Thanks. Not sure if I'll refactor first and repush or accept and refactor after but this helps a lot. I should probably setup travis-ci to ensure in the future things build.
I'd advise updating it on this branch before merge. One thing I'm noticing with the changes I made is that file creation is broken due to a failure to resolve a file descriptor. I'll keep looking at it, though.
Is there some kind of test suite you use?
Unfortunately not. It's been a TODO for quite a while. It's not all that easy given dealing with the filesystem is nothing but side effects and relying on state held by the fuse library and OS. Would need to mock a lot of things.
I have just added basic Travis-CI integration but that'd only confirm building works right now. (Which for OSX it doesn't in the main branch but if you rebase to master it'll build for you.)
Found the source of the descriptor problem. OS X doesn't like AT_FDCWD in fcntl, but has no problem with opendir(".")/dirfd() to get the same value. ಠ_ಠ
I'll poke around some more.
Fixed the issues with the missing API (wrote a version of it) and fixed xattrs and Finder copies as well.
A side effect is that the xattr APIs in mergerfs now have a position argument to them, but it's immediately defaulted to zero for non-Apple platforms and omitted in the final call to the OS.
Every other solution I looked at just filled the code with tons of ifdefs and/or foo_osx.icpp files for very small API changes.
Oops. I've been refactoring your last commit into a more abstract interface and aligned to the overall style/layout. Let me know when you're finished and I'll tweak it if necessary.
If you'd like you can give me access to the PR.
The current version should be good. I've tested it with "rsync -aHAX" and with Finder copies and it checks out (I even mixed HFS+ and APFS-CI backing drives).
When you have it looking how you want it, let me know and I'll run those tests again.
Sure. Thanks for the effort. My one concern though is the copying of xattrs. You've hardcoded XATTR_SHOWCOMPRESSION and while I'm not positive yet... from my random readings it appears that you'd need to do with and without it to get the compressed resource forks and more traditional xattrs.
Right now I'm kindof leaning toward having xattr disabled on OSX for a first patch and focus on getting it right later.
For the xattrs, the Finder uses them a lot for normal operation on the files (things like labels, and display preferences) and other apps keep metadata in them on a semi-regular basis. Not having them limits the usefulness of the resultant mount. :\
The different API with FUSE and the OS is a small pain, but I don't think there's a clean way around having this FS be a superset of functionality in order to pass the right data down.
You're right about the compression flag; that was in there for a test and I missed it. The consumer of the API will pass that if it's expected and there will be side effects if it's there without their knowing/expecting.
There are two xattr uses. One is the FUSE interface. The other is the clonepath and clonefile components. Until the clone'ing usage has to be figured out else the copy will be incomplete.
For OSX, if you're copying a file from one backing store to another, use copyfile(3) with COPYFILE_ALL. The OS will handle all the details for you, including xattrs, resource forks, Finder info, etc.
https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man3/copyfile.3.html
Then all fs::xattr stuff used by copyfile aren't needed for Mac. I'll take a look. Thanks.
There isn't a clear separation in these commits. The type stuff, bug fixes, and osx changes.
I'm going to go through it all and pick out the general fixes and abstractions and put them in the main branch first. Leave out the type stuff (what did you run to discover them? clang's c++ analyzer isn't reporting anything.). Then rebase this branch and focus solely on osx changes.
Any update on this PR? I'm quite interested in trying this out in macOS. If not, is there a particular commit in the PR that I could compile and try out, like the most recent one?
This PR was incomplete and the code has changed quite a bit since it was created. libfuse on OSX is different from Linux (and other platforms) and since I embedded libfuse into mergerfs to simplify installation and normalize behavior its now more complicated to port.
I now have access to an OSX machine but I'm not sure when I'll have time to work on working on a port.
Would it be possible or at all advisable for me to try compiling using the latest commit in this PR's branch?
You can obviously do whatever you like but as I recall this PR was incomplete and IIRC had some bugs.
Just checking in again to see if there's any progress on this. @ahknight is this still something you're working on?
I have a renewed interest in trying to get mergerfs working in macOS and was thinking of seeing if I could port it.
I was going through the up-to-date codebase to try and get a sense of what's involved. The way certain low level functions are factored out makes this easier in comparison to when @ahknight was working on it.
But I can see how it's more difficult now that libfuse is included in the project. There's a few places where you're reaching into fuse's internal structures (e.g. fuse_config_set_attr_timeout and fuse_config_get_attr_timeout) and of course those functions aren't available in osxfuse. And osxfuse differs in a few places compared to libfuse; namely there are some constants mergerfs refers to that are missing from osxfuse, and osxfuse is missing some member variables such as fuse_file_info.auto_cache.
If it's not too much trouble, could you share any thoughts on how to proceed?
It's going to be a lot of trouble. Unless OSXFuse is very similar in protocol... you aren't going to be able to just use the high level libfuse API. And my intent is to further hack away at libfuse. If the protocol is similar enough then it might be doable but it's not going to get more API friendly anytime soon. Even if I hadn't vendored libfuse I had intended to move to the low level API which isn't cross platform.
Not knowing anything about osxfuse I really can't comment further.
Ah, I see now that some of the differences I pointed out are in fact significant changes you've made to libfuse, and so in order to port to mac it's necessary to make equivalent changes to osxfuse. That's certainly more work than I'm willing to take on right now.
It does look like osxfuse includes the low-level API, so if and when you transition mergerfs to that then maybe it'd be worth looking into a mac port again. I'd certainly love to see it on the mac platform as this seems to be the most full featured union filesystem currently available.
It's a big change to move to the low level API. My plan is to slowly take out or collapse the abstractions in libfuse to get it down to the LLAPI or maybe even replace libfuse all together. That's why I'm mentioning the protocol and setup. As I understand osxfuse is a fork of the original libfuse so I wouldn't think the protocol has changed too much. Maybe those changes could be incorporated into mergerfs. I should probably just take a look at the osxfuse code.