Whiteout portability

[dmcgowan]

Continuity has the potential to be used to store filesystem "diffs" in the same way tar files are used in Docker today. In order to support these diffs "whiteout" files should have first class support in the manifest. This will allow a portable way to use whiteouts between different copy on write file systems despite where it was originally generated. As a research item, how are whiteout files handled in the tar file diffs used by docker and are there any issues across filesystems (i.e export from AUFS import to overlay).

[AkihiroSuda]

+1. Any update on this? Can we implement this as just a single boolean field?

[stevvooe]

I think one aspect that needs to be considered here is that a diff is very much contextual. So, not only does it require the ability whiteout files, but it should also provide the ability to specify the manifest to which it applies. Without that, it is not safe.

Can we implement this as just a single boolean field?

Do you have an example?

[AkihiroSuda]

Do you have an example?

message Resource {
    repeated string path = 1;
    ...
    // If whiteout is true and the resource specified by path already exists, 
    // the resource will be removed on apply.}
    // Even when the file mode specified in the manifest and the actual file mode differs (e.g. regular file vs directory), 
    // the resource will be removed.
    bool whiteout = 14;
}

[stevvooe]

@AkihiroSuda What about the issue with parentage?

I would also think that a whiteout would have no other fields set.

[AkihiroSuda]

@stevvooe

WDYT about defining digest for directories, and requiring ParentDigest in Manifest?

If we can define digest for directories, it might be also useful for buildkit (https://github.com/moby/buildkit/issues/38 cc @tonistiigi )

[AkihiroSuda]

I want this feature for mixing up multiple https://github.com/AkihiroSuda/filegrain layers / OCI layers

[stevvooe]

@AkihiroSuda Rather than propagate a hack, let's define a proper diff format in continuity. Whiteout is part of aufs/overlay. We can define the format to be whatever we want for diff.