borg
borg copied to clipboard
Borg2 complains about extended attributes, when importing Solaris TAR archives
Have you checked borgbackup docs, FAQ, and open GitHub issues?
Yes
Is this a BUG / ISSUE report or a QUESTION?
Bug
System information. For client/server mode post info for both machines.
Your borg version (borg -V).
borg 2.0.0b12
Operating system (distribution) and version.
Ubuntu 22.04.5 (ARM64)
Hardware / network configuration, and filesystems used.
How much data is handled by borg?
Around 100KB
Full borg commandline that lead to the problem (leave away excludes and passwords)
borg import-tar --list --info -r BX2 SolarisEAs_FIFO_SymLink Download/SolarisEAs_FIFO_SymLink.tar
Describe the problem you're observing.
When trying to import some test TAR archives, created under Solaris 10, I receive the following, regarding the extended attributes contained within:
root@localhost:/sdcard# borg import-tar --list --info -r BX2 SolarisEAs_FIFO_SymLink Download/SolarisEAs_FIFO_SymLink.tar /usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported " /usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks. warnings.warn( d SolarisMetadata s SolarisMetadata/rcm_daemon_door f SolarisMetadata/SolarisFIFO /dev/null/..hdr: Unsupported tarinfo type b'E' E /dev/null/..hdr /dev/null/.: Unsupported tarinfo type b'E' E /dev/null/. /dev/null/ANewAttribute.hdr: Unsupported tarinfo type b'E' E /dev/null/ANewAttribute.hdr /dev/null/ANewAttribute: Unsupported tarinfo type b'E' E /dev/null/ANewAttribute /dev/null/copy-SUNWattr_ro.hdr: Unsupported tarinfo type b'E'
E /dev/null/copy-SUNWattr_ro.hdr /dev/null/copy-SUNWattr_ro: Unsupported tarinfo type b'E'
E /dev/null/copy-SUNWattr_ro /dev/null/copy-SUNWattr_rw.hdr: Unsupported tarinfo type b'E' E /dev/null/copy-SUNWattr_rw.hdr /dev/null/copy-SUNWattr_rw: Unsupported tarinfo type b'E' E /dev/null/copy-SUNWattr_rw
I believe that Borg 2.0.0b12 is ostensibly supposed to support "SunOS tar with extended attributes" files, but I am unsure if this is supposed to extend to Solaris 10-originated archives.
The archives are available at https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/understand/SolarisEAs_FIFO_SymLink.tar, and https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/understand/SolarisEAs1.tar
Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
Include any warning/errors/backtraces from the system logs
It looks like IBM i pax archives with extended attributes get a similar treatment:
root@localhost:/sdcard# borg import-tar --list --info -r BX2 test2 test2.tar /usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported " /usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks. warnings.warn( A /QSYS.LIB/ZZZZ.DEVD .SUBJECT: Unsupported tarinfo type b'E' E .SUBJECT .CODEPAGE: Unsupported tarinfo type b'E' E .CODEPAGE .TYPE: Unsupported tarinfo type b'E'
)
E .TYPE A /QSYS.LIB/ZZZZ.MSGQ .SUBJECT: Unsupported tarinfo type b'E' E .SUBJECT .CODEPAGE: Unsupported tarinfo type b'E' E .CODEPAGE .TYPE: Unsupported tarinfo type b'E' E .TYPE root@localhost:/sdcard#
[test2.tar.gz](https://github.com/user-attachments/files/17402173/test2.tar.gz
Looks like this comes from borg's filetype dispatcher:
tar = tarfile.open(fileobj=tarstream, mode="r|", ignore_zeros=args.ignore_zeros)
while True:
tarinfo = tar.next()
if not tarinfo:
break
if tarinfo.isreg():
status = tfo.process_file(tarinfo=tarinfo, status="A", type=stat.S_IFREG, tar=tar)
archive.stats.nfiles += 1
elif tarinfo.isdir():
status = tfo.process_dir(tarinfo=tarinfo, status="d", type=stat.S_IFDIR)
elif tarinfo.issym():
status = tfo.process_symlink(tarinfo=tarinfo, status="s", type=stat.S_IFLNK)
elif tarinfo.islnk():
# tar uses a hardlink model like: the first instance of a hardlink is stored as a regular file,
# later instances are special entries referencing back to the first instance.
status = tfo.process_hardlink(tarinfo=tarinfo, status="h", type=stat.S_IFREG)
elif tarinfo.isblk():
status = tfo.process_dev(tarinfo=tarinfo, status="b", type=stat.S_IFBLK)
elif tarinfo.ischr():
status = tfo.process_dev(tarinfo=tarinfo, status="c", type=stat.S_IFCHR)
elif tarinfo.isfifo():
status = tfo.process_fifo(tarinfo=tarinfo, status="f", type=stat.S_IFIFO)
else:
status = "E"
self.print_warning("%s: Unsupported tarinfo type %s", tarinfo.name, tarinfo.type)
self.print_file_status(status, tarinfo.name)
# This does not close the fileobj (tarstream) we passed to it -- a side effect of the | mode.
tar.close()
BTW, both code sections of your posts seem badly formatted, can you fix them?
https://github.com/python/cpython/blob/3.13/Lib/tarfile.py#L87
It doesn't look like tarinfo type "E" being supported even by tarfile stdlib module.
But a week ago (after b12) I added code to support extended attributes in PAX headers (of PAX tar archive members), see #2521.
Looks like the bad formatting is from trying to paste out of Termux. In the meantime, I ended up redoing it, with "script", and another repository:
Script started on 2024-10-17 00:32:29+01:00 [TERM="xterm-256color" TTY="/proc/6418/fd/2" COLUMNS="75" LINES="33"]
# borg import-tar --list --info -r BX3 Solaeus risEAs1.tar Solarus isEAs1.tar
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
d SolarisMetadata
/dev/null/..hdr: Unsupported tarinfo type b'E'
E /dev/null/..hdr
/dev/null/.: Unsupported tarinfo type b'E'
E /dev/null/.
/dev/null/ANewAttribute.hdr: Unsupported tarinfo type b'E'
E /dev/null/ANewAttribute.hdr
/dev/null/ANewAttribute: Unsupported tarinfo type b'E'
E /dev/null/ANewAttribute
/dev/null/copy-SUNWattr_ro.hdr: Unsupported tarinfo type b'E'
E /dev/null/copy-SUNWattr_ro.hdr
/dev/null/copy-SUNWattr_ro: Unsupported tarinfo type b'E'
E /dev/null/copy-SUNWattr_ro
/dev/null/copy-SUNWattr_rw.hdr: Unsupported tarinfo type b'E'
E /dev/null/copy-SUNWattr_rw.hdr
/dev/null/copy-SUNWattr_rw: Unsupported tarinfo type b'E'
E /dev/null/copy-SUNWattr_rw
# exit
Script done on 2024-10-17 00:33:24+01:00 [COMMAND_EXIT_CODE="1"]
Script started on 2024-10-17 00:25:17+01:00 [TERM="xterm-256color" TTY="/proc/6418/fd/2" COLUMNS="75" LINES="33"]
# borg import-tar --list --info -r BX3 lynx2-8-3.tar lynx2-8-3.tar t est2.tar test2.tar
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
A /QSYS.LIB/ZZZZ.DEVD
.SUBJECT: Unsupported tarinfo type b'E'
E .SUBJECT
.CODEPAGE: Unsupported tarinfo type b'E'
E .CODEPAGE
.TYPE: Unsupported tarinfo type b'E'
E .TYPE
A /QSYS.LIB/ZZZZ.MSGQ
.SUBJECT: Unsupported tarinfo type b'E'
E .SUBJECT
.CODEPAGE: Unsupported tarinfo type b'E'
E .CODEPAGE
.TYPE: Unsupported tarinfo type b'E'
E .TYPE
# exit
Script done on 2024-10-17 00:25:51+01:00 [COMMAND_EXIT_CODE="1"]
FWIW, I found https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/tar/tar.c, which documents some of the Solaris extensions
Maybe try if you can create PAX tar archives (POSIX.1-2001 (pax) format) - that seems to be standard, while I suspect that "E" header is a solaris specific thing?
I ended up installing a random build of Solaris 11, and it looks like there's a bunch of different ways of generating a TARBall, with extended attributes, with both "tar", and "pax", as they ship, out of the box.
Not sure about what the "extended headers" are, if you request the inclusion of (system) extended attributes, with "tar", but it doesn't seem possible to change the format, that it generates:
Their "pax" implementation has an "-x format" option, according the manual, which can take "cpio", "pax", "ustar", and "xustar". If not specified, "The default output format is tar(1)", according to that manual.
I'll probably try to generate a bunch of archives, with various Solaris features, to see what I get...
FWIW, there are some system-generated extended attributes ("SUNWattr_ro", and "SUNWattr_rw"), that files get, by default - but, I don't know if they're added to archives, by pax/tar, yet.
Doing some tests, with tar, you can store extended attributes, both with, and without extended headers:
(An archive is seemingly always USTAR format, by default).
It also seems that you don't get the system-generated attributes, unless you copy them, to another attribute, in a "runat" session:
Seems that XUSTAR, in pax is the same format, as used by tar's "extended header" mode.
Anyway, for each combination of format, and use of extended attributes, I've managed to generate a set of archives, containing some binary/text attributes:
I think Solaris also supports adding EAs to directories, but I haven't tested that, yet.
And, after a lot of messing around, with OpenSSH cipher settings, to be able to pull it out of my VM, here's a ZIP archive, containing all of those TAR archives, for testing:
And, if I try to import all of the archives into a new repository, with this script:
borg -r BT2 import-tar 1 TARTest/ExtendedHeader-ExtendedAttributes.tar
borg -r BT2 import-tar 2 TARTest/USTAR-ExtendedAttributes2.tar
borg -r BT2 import-tar 3 TARTest/ExtendedHeader.tar
borg -r BT2 import-tar 4 TARTest/pax-PAX-Default.tar
borg -r BT2 import-tar 5 TARTest/PAX-Default.tar
borg -r BT2 import-tar 6 TARTest/pax-PAX-ExtendedAttributes.tar
borg -r BT2 import-tar 7 TARTest/PAX-ExtendedAttributes.tar
borg -r BT2 import-tar 8 TARTest/ustar-PAX-Default.tar
borg -r BT2 import-tar 9 TARTest/ustar-PAX-ExtendedAttributes.tar
borg -r BT2 import-tar 10 TARTest/USTAR-Default.tar
borg -r BT2 import-tar 11 TARTest/xustar-PAX-Default.tar
borg -r BT2 import-tar 12 TARTest/USTAR-ExtendedAttributes.tar
borg -r BT2 import-tar 13 TARTest/xustar-PAX-ExtendedAttributes.tar
I receive the following:
Script started on 2024-10-22 22:54:27+01:00 [TERM="xterm-256color" TTY="/dev/pts/1" COLUMNS="75" LINES="33"]
# ./Import.sh
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/dev/null/..hdr: Unsupported tarinfo type b'E'
/dev/null/.: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute.hdr: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy: Unsupported tarinfo type b'E'
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/dev/null/..hdr: Unsupported tarinfo type b'E'
/dev/null/.: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute.hdr: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy: Unsupported tarinfo type b'E'
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/dev/null/..hdr: Unsupported tarinfo type b'E'
/dev/null/.: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute.hdr: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy: Unsupported tarinfo type b'E'
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/dev/null/..hdr: Unsupported tarinfo type b'E'
/dev/null/.: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute.hdr: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy: Unsupported tarinfo type b'E'
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/dev/null/..hdr: Unsupported tarinfo type b'E'
/dev/null/.: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute.hdr: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy: Unsupported tarinfo type b'E'
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/dev/null/..hdr: Unsupported tarinfo type b'E'
/dev/null/.: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute.hdr: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy: Unsupported tarinfo type b'E'
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
warnings.warn(
/dev/null/..hdr: Unsupported tarinfo type b'E'
/dev/null/.: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute.hdr: Unsupported tarinfo type b'E'
/dev/null/SolarisAttribute: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_rw-copy: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy.hdr: Unsupported tarinfo type b'E'
/dev/null/SUNWattr_ro-copy: Unsupported tarinfo type b'E'
# exit
Script done on 2024-10-22 22:55:06+01:00 [COMMAND_EXIT_CODE="1"]
It seems that Solaris tar/pax will always create type-"E" objects, for extended attributes, regardless of the archive format selected...
So, would you think it is better if borg silently ignored the E type tarinfos?
It'd be desirable, to archive everything, that's part of, and associated with a file, in the name of fidelity - but, naturally, Borg probably can't special-case everything, from every platform.
I don't know much about that E type stuff, but currently it looks like a non-standard solaris tar specific thing, guess I rather would not implement that considering that there is also the standard PAX format available on solaris.