Potential incompatibility with bsdtar's mtree format?

[Earnestly]

I'm attempting to solve the problem of reapplying permissions and other metadata stored in an mtree generated by bsdtar but I can't seem to get mtree-port to accept it as input.

The mtree itself is generated with bsdtar using the following options:

--format=mtree --options='!all,use-set,type,uid,gid,mode,time,size,md5,sha256,link'

The pacman package manager stores the mtree in a gzipped format under /var/lib/pacman/local/<package>/mtree.

Here's a short but hopefully representational example of one for afl:

% gzip -cd /var/lib/pacman/local/afl-1.93b-1/mtree
#mtree
/set type=file uid=0 gid=0 mode=644
./.PKGINFO time=1441844344.171719430 size=743 md5digest=99e51b623d26d248aa08a37ddf6ea657 sha256digest=cc273f9a50a62cf84488c0864999615603423f778ebdb4b89d2621649790a53a
/set mode=755
./usr time=1441844344.8386312 type=dir
./usr/bin time=1441844344.28386285 type=dir
./usr/bin/afl-clang time=1441844344.28386285 mode=777 type=link link=afl-gcc

When attempting to use this mtree via mtree I get the following error:

# mtree -U < <(gzip -cd /var/lib/pacman/local/afl-1.93b-1/mtree)
mtree: line 3: slash character in file name

This port of mtree seems to directly contradict the keys supported by FreeBSD's mtree and bsdtar such as sha256 vs. sha256digest.

Is there any way to reconsile these differences or am I missing something important?

Thanks

[archiecobbs]

This port was created from a snapshot of FreeBSD's mtree as of several years ago (2010). There are probably several incompatibilities that have crept in since then.

To be honest my personal use of this mtree port is very simplistic, so I'm not surprised that it doesn't work with more serious use. Unfortunately that means I'm not sufficiently up to date on what "more serious use" exactly entails or the current specification for mtree files (if such a thing exists).

Feel free to do some investigation and suggest way we can update this port.

[Earnestly]

I'm not sure what to suggest other than perhaps following libarchive's implementation which is what bsdtar uses.

[archiecobbs]

Just curious... Try adding the flag --options mtree:use-set when generating the mtree output and see what happens.

[Earnestly]

As I'm not exactly sure what you mean adding a flag, I reduced the flags to just be --options mtree:use-set instead.

While this certainly changed the output of bsdtar's mtree, it didn't resolve the issue:

# mtree -U < <(gzip -cd /var/lib/pacman/local/afl-1.93b-2/mtree)
mtree: line 3: slash character in file name

% gzip -cd /var/lib/pacman/local/afl-1.93b-2/mtree
#mtree
/set type=file uname=root uid=0 gname=root gid=0 mode=644
./.PKGINFO time=1442079949.353061312 size=719
/set mode=755
./usr time=1442079949.213062257 type=dir
./usr/bin time=1442079949.233062121 type=dir
./usr/bin/afl-clang time=1442079949.229728811 mode=777 type=link link=afl-gcc
./usr/bin/afl-clang++ time=1442079949.233062121 mode=777 type=link link=afl-gcc
./usr/bin/afl-cmin time=1442079949.219728879 size=11056

Edit: For what's worth, the bsdtar I'm using is the latest stable release: bsdtar - libarchive 3.1.2

[archiecobbs]

Hmm, too bad. I thought this might work around the "slash character in file name" error.

[archiecobbs]

FYI, one could argue this is a bsdtar bug, not an mtree bug :)

And in fact one has...

https://github.com/libarchive/libarchive/issues/254

[Earnestly]

To be fair, the mtree format in libarchive hasn't had the greatest story with regard to stability either.

However that issue is at the latest 5 years old and it's worth considering that the current FreeBSD's mtree also supports their own bsdtar mtree format according to the mtree(8) manual.

It appears also that they've incorporated a new -F option for selecting the "flavour" of mtree to use, this could indeed highlight the turbulence in the mtree format over the years...

[archiecobbs]

Try this branch:

https://github.com/archiecobbs/mtree-port/tree/slash-in-filename

Completely untested.

[Earnestly]

Oh thank you for this, I have tried it just now but with a new error:

% mtree -U < <(gzip -cd /var/lib/pacman/local/afl-1.94b-1/mtree)
mtree: .: No such file or directory

Here provided is the mtree from the afl package in full.

After playing around a bit, I tried removing the leading ./ and this appears to have found more success but with less than ideal results. It will recreate the files and their permissions in the current working directory just fine, however if I apply -p / it will provide a few extra warnings:

# mtree -u < <(gzip -cd /var/lib/pacman/local/afl-1.94b-1/mtree| sed 's:^\./::g') -p /
. extra
.PKGINFO missing
usr missing (directory not created: File exists)
usr/bin missing (directory not created: File exists)
usr/bin/afl-clang missing (symlink not created: File exists)
usr/bin/afl-clang++ missing (symlink not created: File exists)
usr/bin/afl-cmin missing
usr/bin/afl-fuzz missing
usr/bin/afl-g++ missing (symlink not created: File exists)
usr/bin/afl-gcc missing
...

Of course the files it claims are missing do exist relative to the / directory on my system, but nothing appears to have been affected :-(

Having this functionality would solve a small problem really cleanly. The issue is when a user changes the permissions of a file owned by a package and wishes to revert back. rpm does this with --setperms <package> which will reapply the permissions and so forth according to the package.

How nice it would be if one could just use chmod and reuse existing functionality! But for complex packages this might be unwieldy and error prone, so I thought, why not reuse the mtree we already create?

Ideally if this could work it would be quite a nice solution to the problem of having to check a directory if it is only owned by a single package.

[Earnestly]

Oh, also I just learned that there may be a solution for this in the form of pacfix which was recently created...

[archiecobbs]

See also this newer mtree port: https://github.com/archiecobbs/nmtree