cibuildwheel icon indicating copy to clipboard operation
cibuildwheel copied to clipboard
verified publisher icon pypa

Linux from macOS issues

Open henryiii opened this issue 1 year ago • 4 comments

Description

Building a linux wheel from macOS is producing warnings and created broken files:

$ pipx run cibuildwheel --only cp313t-manylinux_x86_64
...

Copying project into container...

    + mkdir -p /project
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.FinderInfo'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.xcode.CreatedByBuildSystem'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.xcode.CreatedByBuildSystem'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.FinderInfo'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.FinderInfo'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'

We can hide that warning, I think, but I'm also getting junk in the files:

/project/src/._register_transforms.cpp:1:1: warning: null character(s) ignored
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      | ^
/project/src/._register_transforms.cpp:1:2: error: stray ‘\5’ in program
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      |  ^
/project/src/._register_transforms.cpp:1:3: error: stray ‘\26’ in program
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      |   ^
/project/src/._register_transforms.cpp:1:4: error: stray ‘\7’ in program
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      |    ^
/project/src/._register_transforms.cpp:1:5: warning: null character(s) ignored
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      |     ^
...

Using gnu-tar instead works fine. I don't see a --no-xattrs option (https://stackoverflow.com/questions/51655657/tar-ignoring-unknown-extended-header-keyword-libarchive-xattr-security-selinux).

Build log

No response

CI config

No response

henryiii avatar Aug 11 '24 22:08 henryiii

Can we go back to using docker cp rather than the tar-pipes now? I recall we had to change due to a bug in docker, but that was a while ago.

(One thing to be careful about would be file ownership/permissions if we go back)

joerick avatar Aug 11 '24 22:08 joerick

c.f. https://github.com/pypa/cibuildwheel/pull/1434 for other reference. Other than gnu-tar, the --format gnutar argument could be used on macOS.

mayeut avatar Aug 12 '24 05:08 mayeut 

--format {ustar|pax|cpio|shar}  Select archive format

I assume we could use pax on both ends, though. (Edit: doesn't seem to work)

Do you have a ref to the docker bug?

henryiii avatar Aug 13 '24 00:08 henryiii

This works:

diff --git a/cibuildwheel/oci_container.py b/cibuildwheel/oci_container.py
index 0a5fa325..39516f35 100644
--- a/cibuildwheel/oci_container.py
+++ b/cibuildwheel/oci_container.py
@@ -229,7 +229,7 @@ def copy_into(self, from_path: Path, to_path: PurePath) -> None:
         if from_path.is_dir():
             self.call(["mkdir", "-p", to_path])
             subprocess.run(
-                f"tar cf - . | {self.engine.name} exec -i {self.name} tar --no-same-owner -xC {shell_quote(to_path)} -f -",
+                f"tar -c --format ustar -f - . | {self.engine.name} exec -i {self.name} tar --format ustar --no-same-owner -xC {shell_quote(to_path)} -f -",
                 shell=True,
                 check=True,
                 cwd=from_path,

This seems to be a ref for the docker cp bug: https://github.com/pypa/cibuildwheel/issues/1782 (not the bug, but apparently this workaround kills Windows too, and there's some links there, including a mention that Docker server 24 should be fine)

henryiii avatar Aug 13 '24 00:08 henryiii