--exclude and --exclude-from [BOUNTY $100]

[fulldecent]

Because --reverse's only practical use case is archiving, I request that --exclude and --exclude-from features be added to be consistent with other archiving tools.

Work plan

• [ ] Use fnmatch(3) with FNM_PATHNAME for pattern matching
• [ ] Add a --exclude option to specify a pattern
• [ ] Add a --exclude-from option to specify a file which contains multiple patterns (see https://github.com/jmwhittaker/dotfiles/blob/master/backup_excludes.txt for an example)

Current workarounds:

• Use rofs-filtered and then encfs on top of it
  • THIS IS NOW A VIABLE WORKAROUND ON MAC/WIN/NIX, see https://github.com/gburca/rofs-filtered/issues/1
• Use rsync --link-dest to filter, then encfs on top of it
  • Works great, see https://gist.github.com/fulldecent/6271e06aa4cfc7271777
  • Completely allows rsync-style --exclude and --exclude-from options
• Whitelist files to transfer then use encfsctl to get encrypted names
  • This is implemented here https://gist.github.com/fulldecent/910fcd42350d59cb8558
  • Problem: it will not delete extraneous file on the destination (rsync's --files-from does not work with --delete).

[rfjakob]

encfsctl encode takes filenames on stdin. This should do it:

encfsctl encode /my/encfs/dir < excludes.txt > excludes-encrypted.txt

[fulldecent]

The problem is ._* and **/.DS_Store

[fulldecent]

I have updated the issues to include the current workarounds I am working on. Anyone can star or "+1" issue if you care.

[Aikhjarto]

Why don't you use filterfs?

[rfjakob]

Looks like this could be implemented in EncFS without too much hassle.... You'd just skip the files you want to exclude at https://github.com/vgough/encfs/blob/master/encfs/DirNode.cpp#L102 .

[rfjakob]

In reverse mode, de->d_name contains the plaintext name (in normal mode: the encrypted name). Is glob matching on the filename good enough? -> http://linux.die.net/man/3/fnmatch

[fulldecent]

@Aikhjarto filterfs does not support --exclude-from so the command line quickly gets ridiculous. Also, it's not available on brew.

@rfjakob yes, fnmatch will accomplish everything needed here

[tovrstra]

#99 May be relevant. In my case encfs --reverse on top of rofs-filtered requires a silly workaround.

[fulldecent]

FYI, I have added a complete workaround for backup use cases. This uses rsync itself to filter the files with --exclude and --exclude-from options (with --list-dest for speed), then encfs is able to encrypt the filtered directory.

Implementation at https://gist.github.com/fulldecent/6271e06aa4cfc7271777

[tovrstra]

@fulldecent : Thanks for sharing this. Which scripting language do you use in this workaround? (A shebang line would be good.) It seems that you have to run rsync twice on all the data: once for filtering and once for the backup. Does this double the time needed for the backup? If that isn't a concern, the workaround is nice. Using --link-dest for filtering is a clever trick. One should just make sure that the filtered copy is put on the same filesystem.

[fulldecent]

@tovrstra thanks. This is a rakefile and runs with the rake command. Running without arguments provides usage instructions. Sorry, no shebang for rake files. The first rsync is pretty quick, about 30s for 100gb/50k files on spinning USB HDD on 2014 MacBook.

Yes, your note is important about same filesystem. I am backing up a complete external drive but I needed to put the filtered die on that drive too. To avoid recursion, the solution was to ensure the name of the filtered directory matched a rule in the excludes.

[fulldecent]

Ok, I have added a bounty, which closes #77.

[dsteinkopf]

I am having the same problem and found a variante of the workaround using rsync and --link-dest: I am not able to place the links on the same filesystem as the source - so I could not use link-dest. But --files-from seems to be the solution. I did successfully play around with something like

rsync -a --info=name1 --dry-run --exclude-from foo /misc/bdata-all/ /tmp \
    | grep -v '/$' | sed 's/ -> .*$//' >files
encfsctl encode ... <files >filesEnc
rsync -av --files-from=filesEnc /misc/bdata-all/ /backup/

Does anyone have tried something similiar and would like to share experience?

Edit: Forgot the encoding of the filename listed in "files". Added encfsctl...

[fulldecent]

@dsteinkopf I have updated the original issue here. The rofs workaround is now fully usable. (Previously it did not work on macOS).

[dsteinkopf]

Thank you for the information. As I don't like installing self compiled file systems (although it's really easy in this case), I prefer the above method using --files-from....

[fulldecent]

How self compiled? This is an open source project https://github.com/gburca/rofs-filtered and anyone can compile.

Also regarding --files-from, I have also tried this approach. You may have saw the note above. This does NOT work with --delete. So the backup drive gets cluttered.

[dsteinkopf]

Yes, you're right: When using --files-from the --delete options seems to be ignored :-( Just did a litte test...

self compiled: I have to compile and install it myself and am not able to add it on the list of required packages to be installed on a server (I am not using mac in this case but ubuntu).

But rofs-filtered seems to be the best remaining option for me anyway. Unfortunately the exclude syntax is different to rsync, so I have to "translate" the excludes.

(BTW. Thank you for this post and your replies.)

[dsteinkopf]

Another comment to using rsync with --delete: In this post I found a hint that deleting works when using --include-from=filesEnc --exclude='*' --delete instead of --files-from=filesEnc will acutally do file deletion.