thheller
Tell macOS Time Machine to ignore cached/generated files
It would help reduce backup sizes on macOS for Time Machine and 3rd party backup tools if Shadow's generated files, caches, and target directories were marked to not be backed up. There is a command that can check any path for you to see if it is backed up or not:
$ tmutil isexcluded .shadow-cljs/
[Included] [...]/.shadow-cljs
$ tmutil isexcluded /Users/<myuser>/Developer/clojure/clj-rethinkdb/target
[Included] /Users/<myuser>/Developer/clojure/clj-rethinkdb/target
Apple provides an API CSBackupSetItemExcluded to mark directories as excluded by Time Machine, and other Mac backup software respects the same markings. There is also a command line tool tmutil which lets you accomplish the same task with tmutil addexclusion <file path>.
Relates to boot-clj/boot#628, https://github.com/technomancy/leiningen/issues/2292.
IMO generated directories shouldn't be backed up by default, though I could see this being a user setting (not a per-project setting).
Sounds reasonable but I'm uncertain whether this is something shadow-cljs should be doing?
I'm assuming time machine "remembers" this setting, so you calling tmutil addexclusion .shadow-cljs once, additionally maybe your :output-dir folders, doesn't seem unreasonable?
I just checked the tmutil isexcluded and it is telling me that the folder is included, even though I'm not using time machine at all. Not a super fan of running commands that aren't exactly needed.
I'm assuming time machine "remembers" this setting, so you calling tmutil addexclusion .shadow-cljs once, additionally maybe your :output-dir folders, doesn't seem unreasonable?
There's three ways to tell Time Machine not to back up files:
- The user specifies exclusions by path in Time Machine settings
- The program marks files or folders to be excluded by path with
CSBackupSetItemExcluded(not sure where this lives) - The program marks files or folders to be excluded by setting an attribute on the file with
CSBackupSetItemExcluded
The benefit of doing the third option is that other backup tools like Backblaze are also able to read the attribute and prevent backing up those files/folders.
More detailed info here: https://tommyang.github.io/pondini.org/TM/Works4.html
I just checked the tmutil isexcluded and it is telling me that the folder is included, even though I'm not using time machine at all. Not a super fan of running commands that aren't exactly needed.
tmutil looks at Time Machine settings (if any) + attributes on the files to tell you if it is included/excluded.
My proposed solution would be to set the attribute by calling the C API CSBackupSetItemExcluded from Java/Clojure. That way it would communicate to Time Machine and all other backup programs that these files don't need to be backed up.
An alternative option is for me to exclude all of the locations where files are generated by Shadow. That works for projects I work with regularly, but for projects I occasionally check out, I am unlikely to remember to do this.
Native code complicates distribution by a whole lot, so not a fan of that.
For the .shadow-cljs folder I guess its not a problem to add this, although calling it every time shadow-cljs launches seems like overkill. Could probably just create a .shadow-cljs/ignore-is-set file or something so it only runs if this doesn't exist.
:output-dir is a different story, as shadow-cljs does not enforce this directory being empty. So there is no real way to ensure this wouldn't ignore files that shouldn't be when set for the folder. Not a fan of setting this per file.
Wouldn't a tool that sets this based on .gitignore make more sense? I still don't quite see shadow-cljs as the best fit for doing this.
Ok, I recently moved to a mac mini and just now setup time machine. So, I'm going to spend some time on this. Here is how far I got with Claude. Since I got frustraded with claude messing up clojure code I started with Java got this
import java.lang.foreign.*;
import java.lang.invoke.MethodHandle;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
public class TimeMachineExclusion {
private static final Linker LINKER = Linker.nativeLinker();
private static final SymbolLookup LIBC;
private static final MethodHandle SETXATTR;
private static final MethodHandle REMOVEXATTR;
private static final MethodHandle GETXATTR;
// Time Machine exclusion attribute name
private static final String TM_EXCLUDE_ATTR = "com.apple.metadata:com_apple_backup_excludeItem";
static {
try {
// Use libc for xattr functions - try multiple possible locations
SymbolLookup libc = null;
String[] libcPaths = {
"/usr/lib/libc.dylib",
"/usr/lib/libSystem.dylib", // macOS system library that includes libc
"libc.dylib",
"libSystem.dylib"
};
for (String path : libcPaths) {
try {
libc = SymbolLookup.libraryLookup(path, Arena.global());
break;
} catch (IllegalArgumentException e) {
// Try next path
continue;
}
}
if (libc == null) {
throw new RuntimeException("Could not find libc/libSystem");
}
LIBC = libc;
// int setxattr(const char *path, const char *name, const void *value, size_t size, u_int32_t position, int options)
FunctionDescriptor setxattrDesc = FunctionDescriptor.of(
ValueLayout.JAVA_INT, // return int
ValueLayout.ADDRESS, // const char *path
ValueLayout.ADDRESS, // const char *name
ValueLayout.ADDRESS, // const void *value
ValueLayout.JAVA_LONG, // size_t size
ValueLayout.JAVA_INT, // u_int32_t position
ValueLayout.JAVA_INT // int options
);
// int removexattr(const char *path, const char *name, int options)
FunctionDescriptor removexattrDesc = FunctionDescriptor.of(
ValueLayout.JAVA_INT, // return int
ValueLayout.ADDRESS, // const char *path
ValueLayout.ADDRESS, // const char *name
ValueLayout.JAVA_INT // int options
);
// ssize_t getxattr(const char *path, const char *name, void *value, size_t size, u_int32_t position, int options)
FunctionDescriptor getxattrDesc = FunctionDescriptor.of(
ValueLayout.JAVA_LONG, // return ssize_t
ValueLayout.ADDRESS, // const char *path
ValueLayout.ADDRESS, // const char *name
ValueLayout.ADDRESS, // void *value
ValueLayout.JAVA_LONG, // size_t size
ValueLayout.JAVA_INT, // u_int32_t position
ValueLayout.JAVA_INT // int options
);
SETXATTR = LINKER.downcallHandle(
LIBC.find("setxattr").orElseThrow(), setxattrDesc);
REMOVEXATTR = LINKER.downcallHandle(
LIBC.find("removexattr").orElseThrow(), removexattrDesc);
GETXATTR = LINKER.downcallHandle(
LIBC.find("getxattr").orElseThrow(), getxattrDesc);
} catch (Exception e) {
throw new RuntimeException("Failed to initialize xattr bindings", e);
}
}
/**
* Exclude a file or directory from Time Machine backups
* @param path The path to exclude
* @return true if successful, false otherwise
*/
public static boolean excludeFromBackup(String path) {
return setExclusionAttribute(path, true);
}
/**
* Include a file or directory in Time Machine backups (remove exclusion)
* @param path The path to include
* @return true if successful, false otherwise
*/
public static boolean includeInBackup(String path) {
return setExclusionAttribute(path, false);
}
/**
* Check if a file or directory is excluded from Time Machine backups
* @param path The path to check
* @return true if excluded, false if included or on error
*/
public static boolean isExcludedFromBackup(String path) {
try (Arena arena = Arena.ofConfined()) {
MemorySegment pathSeg = arena.allocateFrom(path);
MemorySegment attrSeg = arena.allocateFrom(TM_EXCLUDE_ATTR);
// First call to get the size needed
long size = (long) GETXATTR.invoke(pathSeg, attrSeg, MemorySegment.NULL, 0L, 0, 0);
return size > 0; // If attribute exists, item is excluded
} catch (Throwable e) {
return false; // Assume not excluded on error
}
}
private static boolean setExclusionAttribute(String path, boolean exclude) {
// Verify path exists
if (!Files.exists(Paths.get(path))) {
System.err.println("Path does not exist: " + path);
return false;
}
try (Arena arena = Arena.ofConfined()) {
MemorySegment pathSeg = arena.allocateFrom(path);
MemorySegment attrSeg = arena.allocateFrom(TM_EXCLUDE_ATTR);
if (exclude) {
// Set the attribute with a binary plist value
// This is the standard value that Time Machine expects
byte[] plistValue = createExclusionPlist();
MemorySegment valueSeg = arena.allocateFrom(ValueLayout.JAVA_BYTE, plistValue);
int result = (int) SETXATTR.invoke(
pathSeg, // path
attrSeg, // attribute name
valueSeg, // value
(long) plistValue.length, // size
0, // position (not used on macOS)
0 // options
);
if (result != 0) {
System.err.println("Failed to set exclusion attribute, error code: " + result);
return false;
}
} else {
// Remove the attribute
int result = (int) REMOVEXATTR.invoke(
pathSeg, // path
attrSeg, // attribute name
0 // options
);
if (result != 0) {
// It's okay if the attribute doesn't exist (error -93)
System.err.println("Note: Could not remove exclusion attribute (may not exist), error code: " + result);
}
}
return true;
} catch (Throwable e) {
System.err.println("Error setting exclusion attribute: " + e.getMessage());
return false;
}
}
/**
* Creates a minimal binary plist matching tmutil's exact format
*/
private static byte[] createExclusionPlist() {
// Minimal binary plist that matches tmutil addexclusion exactly
// Just "bplist00" + the string "com.apple.backupd" with minimal plist structure
String binaryPlist = "bplist00_\u0010\u0011com.apple.backupd";
return binaryPlist.getBytes(java.nio.charset.StandardCharsets.ISO_8859_1);
}
public static void main(String[] args) {
if (args.length < 1) {
System.out.println("Usage: java TimeMachineExclusion <path> [exclude|include|check]");
System.out.println(" exclude - exclude from Time Machine (default)");
System.out.println(" include - include in Time Machine (remove exclusion)");
System.out.println(" check - check if currently excluded");
return;
}
String path = args[0];
String action = args.length > 1 ? args[1].toLowerCase() : "exclude";
try {
switch (action) {
case "exclude":
boolean excluded = excludeFromBackup(path);
System.out.println("Exclude '" + path + "': " + (excluded ? "SUCCESS" : "FAILED"));
break;
case "include":
boolean included = includeInBackup(path);
System.out.println("Include '" + path + "': " + (included ? "SUCCESS" : "FAILED"));
break;
case "check":
boolean isExcluded = isExcludedFromBackup(path);
System.out.println("Path '" + path + "' is " + (isExcluded ? "EXCLUDED" : "INCLUDED") + " from Time Machine");
break;
default:
System.err.println("Unknown action: " + action);
System.err.println("Use: exclude, include, or check");
}
} catch (Exception e) {
System.err.println("Error: " + e.getMessage());
}
}
}
I went with your CSBackupSetItemExcluded variant first, but that just fails with
This process is attempting to exclude an item from Time Machine by path without administrator privileges. This is not supported.
So, that was out. tmutil via the command line bugged me because I cannot check whether a file is already excluded, but I might still go with that route. Given that Java21+ is now required anyway going with panama should be fine though.
I'll ponder this for a bit longer, but seems promising.
The native approach is out as well I guess.
WARNING: A restricted method in java.lang.foreign.SymbolLookup has been called
WARNING: java.lang.foreign.SymbolLookup::libraryLookup has been called by TimeMachineExclusion in an unnamed module (file:/Users/thheller/code/tmp/TimeMachineExclusion.java)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled
tmutil actually has isexluded. Thanks ChatGPT for making me believe this didn't exist ...
So, probably going with just tmutil and no native gimmicks.