manylinux icon indicating copy to clipboard operation
manylinux copied to clipboard

Cannot build C++14 in docker images

Open YannickJadoul opened this issue 7 years ago • 59 comments

Hello. I'm in the situation where I have a C++14 extension (using the brilliant (pybind11)[https://github.com/pybind/pybind11] library), but the GCC version installed on the docker images are 4.8 and do not support the C++14 standard.

Is there a way I can get around this, and would still be able to produce manylinux1 wheels?

YannickJadoul avatar Jun 08 '17 19:06 YannickJadoul

Unfortunately there aren't any compilers that both support c++14 and can generate manylinux1-compatible (= centos 5 compatible) builds.

However, there is a plan to define manylinux2 and manylinux3 standards based on centos6 and centos7, respectively. That's the most likely path to getting c++14 support. Currently there's no one actively working on this, because we're a volunteer run project and none of the existing contributors have the time right now, but it would be a pretty straightforward thing for someone to help with if you want. There's a rough checklist of what needs to happen in this email.

njsmith avatar Jun 08 '17 21:06 njsmith

Thank you for this quick reply. Hmm, given that I've been learning quite a lot about all this, over the last few days, I would indeed find it interesting to help (after digging into the internals of the first manylinux).

Meanwhile, while lacking such a newer standard, what would be the best way forward, on the short term? 'Downgrading' my C++14 code to 11, or building on another system (e.g., Travis CI's Ubuntu 12.04 (precise))?

YannickJadoul avatar Jun 08 '17 22:06 YannickJadoul

If you can downgrade your code to c++11, then that'll be the most reliable option, yeah. If you build on another system, then your wheels won't be manylinux1 compatible, and only manylinux1 wheels are allowed on pypi currently. Technically I suppose you could lie and label your wheels as being manylinux1-compatible when they aren't, but (a) that would be naughty, and (b) since pip uses the platform tag to determine which systems are compatible with which wheels, lying to it sets up a situation where some of your users may get broken installs. (Of course if you're not distributing these wheels publicly on pypi, then this matters much less.)

I think the manylinux2/3 to-do list is actually pretty approachable if you want to go that way; most of it should be possible to copy directly from the manylinux1 equivalents.

In any case, good luck!

njsmith avatar Jun 08 '17 23:06 njsmith

If you're really careful, it might be possible to use a newer version of g++ with tricks like http://archive.is/wkKoy#selection-129.0-144.0 to generate manylinux1-compatible code, but it requires a pretty high degree of compiler kung fu.

rmcgibbo avatar Jun 09 '17 14:06 rmcgibbo

Can't you (1) install a new gcc in the docker container and then (2) just vendor libstdc++

mpharrigan avatar Jun 09 '17 22:06 mpharrigan

@rmcgibbo Thanks, not sure how easy that is (though I'll pay around a little bit, next week), but it's a very nice read on the actual problem!

@mpharrigan The problem is that I can't find a GCC >= 5 for CentOS 5, so are you suggesting to build from source?

YannickJadoul avatar Jun 10 '17 07:06 YannickJadoul

Building a recent toolchain and then vendoring libstdc++ would indeed work AFAIK, though I don't know how easy it is.

njsmith avatar Jun 10 '17 07:06 njsmith

are you suggesting to build from source?

Yeah

mpharrigan avatar Jun 11 '17 00:06 mpharrigan

Right, I'll give these different options a shot, over the course of the next week, and see what works best in my case.

YannickJadoul avatar Jun 12 '17 08:06 YannickJadoul

I've built multilib GCC 6.3 for Centos 5 for Porable PyPy project: https://github.com/squeaky-pl/centos-devtools/releases

If you manage to convince manylinux docker image to use this compiler instead it might work.

You would need to untar it to your root and then use /opt/devtools-6.3/bin/<whatever>, possibly by tweaking PATH:

wget https://github.com/squeaky-pl/centos-devtools/releases/download/6.3/gcc-6.3.0-binutils-2.27-x86_64.tar.bz2 -O - | tar -C / -xj

squeaky-pl avatar Jun 14 '17 15:06 squeaky-pl

@squeaky-pl: that's impressive. i'm eager to hear if it works for the manylinux case. if so, that would be fantastic.

rmcgibbo avatar Jun 14 '17 17:06 rmcgibbo

Wow, this is brilliant! Unpacking and compiling seems to be working like a charm, on my local docker image.

However, I'm still running into a problem with auditwheel

[root@e46be390efb1 dist]# auditwheel show praat_parselmouth-0.1.0-cp27-cp27m-linux_x86_64.whl 

praat_parselmouth-0.1.0-cp27-cp27m-linux_x86_64.whl is consistent with
the following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.3.1',
'GCC_3.0'}, libstdc++.so.6 with versions {'CXXABI_1.3.5',
'CXXABI_1.3.9', 'CXXABI_1.3', 'GLIBCXX_3.4', 'GLIBCXX_3.4.18',
'GLIBCXX_3.4.20', 'CXXABI_1.3.3', 'GLIBCXX_3.4.21'}, libm.so.6 with
versions {'GLIBC_2.2.5'}, libpthread.so.0 with versions
{'GLIBC_2.2.5', 'GLIBC_2.3.2'}, libc.so.6 with versions
{'GLIBC_2.2.5'}

This constrains the platform tag to "linux_x86_64". In order to
achieve a more compatible tag, you would to recompile a new wheel from
source on a system with earlier versions of these libraries, such as
CentOS 5.

So, @squeaky-pl, is your GCC 6.3 also using a more recent version of CXXABI than the standard one on the manylinux1 CentOS 5?

YannickJadoul avatar Jun 14 '17 21:06 YannickJadoul

I am not quite sure because I don't work with C++ at all. The compiler was compiled itself on Centos 5 but of course C++ standard library evolved quite a lot since GCC 4.8.

The PEP itself says:

GLIBC <= 2.5 <-- fine
CXXABI <= 3.4.8 <-- fine
GLIBCXX <= 3.4.9 <-- this looks violated
GCC <= 4.2.0 <-- fine

So probably this is libstdc++.so.6 that needs to be shipped somehow because it's tightly coupled with compiler/C++ standard version I guess, and probably the resulting wheel references libstdc++.so.6 from that /opt/devtools-6.3 subdir.

squeaky-pl avatar Jun 14 '17 21:06 squeaky-pl

@YannickJadoul what about feeding -static-libstdc++ to the compiler/linker?

squeaky-pl avatar Jun 14 '17 21:06 squeaky-pl

Hmmm, I don't know, but I can give that a shot tomorrow. But I'm guessing there's a reason this isn't that standard approach, or not? Any experience with what the size increase of the wheel would be?

YannickJadoul avatar Jun 14 '17 22:06 YannickJadoul

I think the issue here is that auditwheel assumes that you want to use the system libstdc++, so when it sees that you're using a newer version it just says "nope, not manylinux!" instead of invoking its vendoring logic. Maybe there should be a way to tell it that no, you really do want to vendor libstdc++ (or similar).

-static-libstdc++ will make your wheel bigger, but maybe not as much as copying the entire libstdc++.so into your wheel, which is the other option. (Assuming we're sticking with manylinux1 here.) I guess if you have just one extension module using c++ then -static-libstdc++ should produce a smaller wheel than vendoring, and if you have multiple extension modules using c++ then it's hard to predict.

njsmith avatar Jun 14 '17 23:06 njsmith

That dóes work!

[root@485583349e21 Parselmouth]# auditwheel show dist/praat_parselmouth-0.1.0-cp27-cp27m-linux_x86_64.whl 

praat_parselmouth-0.1.0-cp27-cp27m-linux_x86_64.whl is consistent with
the following platform tag: "manylinux1_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0',
'GCC_3.3', 'GCC_3.3.1', 'GCC_4.2.0'}, libpthread.so.0 with versions
{'GLIBC_2.3.2', 'GLIBC_2.2.5'}, libc.so.6 with versions {'GLIBC_2.3',
'GLIBC_2.2.5'}, libm.so.6 with versions {'GLIBC_2.2.5'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib64/libc-2.5.so",
    "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
    "libm.so.6": "/lib64/libm-2.5.so",
    "libpthread.so.0": "/lib64/libpthread-2.5.so"
}

Gives me an increase in size from about 2.1 MB (2122551) to 2.5 MB (2510676), which seems rather reasonable for a 2017 library. On the other hand, it still feels somehow 'wrong' in principle, and like a 'temporary hack', or not?

YannickJadoul avatar Jun 15 '17 11:06 YannickJadoul

Sweet.

It's not really any more of a hack then anything else involved in shipping portable binaries – it's just that this time you got a peek behind the covers at the kind of thing that the docker image and auditwheel are trying to hide from you :-).

In the longer run switching to manylinux2/3 is probably a better solution (if only so you don't have to keep maintaining your own toolchain forever!), but I don't think you need to feel bad about it or anything.

Btw parselmouth looks super cool.

njsmith avatar Jun 15 '17 15:06 njsmith

I would say it would be cool (even in the future manylinux2) to let people use whatever C++ standard version they want especially when C++ goes so fast these days. It would be nice to show people that they can use newer compiler and get more manylinux packages, I don't mind maintaining that GCC because I've been doing that already for 3 years.

squeaky-pl avatar Jun 15 '17 16:06 squeaky-pl

@squeaky-pl: it's much less of an issue though for manylinux2/3, because for centos6/7 redhat is maintaining more up to date devtools compilers.

njsmith avatar Jun 15 '17 17:06 njsmith

...though actually it looks like the latest devtoolset compiler is GCC 5.3, so... maybe I'm wrong.

njsmith avatar Jun 15 '17 19:06 njsmith

@njsmith Thanks for the insight! And I dó like the fact that I've now got binary manylinux wheels :-)

I'd actually still be interested in helping with the manylinux2/3, on the longer term, though I don't think I have all the necessary practical knowledge and experience to do all this.

Also, the main 'magic' of Parselmouth is still in the pybind11 library, as far as I'm concerned; that thing is really awesome ;-)

YannickJadoul avatar Jun 15 '17 21:06 YannickJadoul

@squeaky-pl Just out of curiosity: did you actually have to come up with some tricks in order to get that CentOS 5 distribution of GCC 6, or was is 'just a matter of compiling' ?

YannickJadoul avatar Jun 15 '17 21:06 YannickJadoul

@YannickJadoul There are no tricks as far as I know. There is "standard GCC bootstraping pain" that might look tricky to somebody who never built GCC from source but it's no different than compiling GCC on a recent distribution in the end. https://github.com/squeaky-pl/centos-devtools you can look into build and build_deps if you are curious.

squeaky-pl avatar Jun 15 '17 22:06 squeaky-pl

I think it would be great if we could add some of the information in this thread as instructions for getting modern C++ working in the manylinux1 container to the wiki or readme or something. I think many people would benefit from them.

Sent from my iPhone

On Jun 15, 2017, at 2:23 PM, Yannick Jadoul [email protected] wrote:

@njsmith Thanks for the insight! And I dó like the fact that I've now got binary manylinux wheels :-)

I'd actually still be interested in helping with the manylinux2/3, on the longer term, though I don't think I have all the necessary practical knowledge and experience to do all this.

Also, the main 'magic' of Parselmouth is still in the pybind11 library, as far as I'm concerned; that thing is really awesome ;-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rmcgibbo avatar Jun 15 '17 22:06 rmcgibbo

Is there any possibility of getting runtime problems if a user imports two packages that both use compiled c++ code that statically links different versions of libstdc++?

Sent from my iPhone

On Jun 15, 2017, at 2:23 PM, Yannick Jadoul [email protected] wrote:

@njsmith Thanks for the insight! And I dó like the fact that I've now got binary manylinux wheels :-)

I'd actually still be interested in helping with the manylinux2/3, on the longer term, though I don't think I have all the necessary practical knowledge and experience to do all this.

Also, the main 'magic' of Parselmouth is still in the pybind11 library, as far as I'm concerned; that thing is really awesome ;-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rmcgibbo avatar Jun 15 '17 22:06 rmcgibbo

@rmcgibbo: If you have two extension modules that use C++ internally but that only communicate via python interfaces (i.e., they're not directly linking to each other or smuggling function pointers through pycapsules or something), then you should be OK with either static linking or vendoring libstdc++. (Watch out for cython's cimport functionality, which under the covers involves smuggling function pointers through pycapsules; otherwise I think you'll know if you're doing this.)

Also, that's assuming that you're using the standard Python interpreter. If you have Python embedded inside a C++ program then I think statically linking libstdc++ will be ok, but vendoring libstdc++ using auditwheel would not. So like, if you think someone might want to use your extension module from inside blender, then stick to statically linking libstdc++ or use the system libstdc++; don't try to vendor libstdc++.

njsmith avatar Jun 16 '17 00:06 njsmith

Actually that blender case is a pretty subtle gotcha – maybe that's an argument for not supporting libstdc++ vendoring in auditwheel, and recommending that people who need a newer libstdc++ do static linking.

njsmith avatar Jun 16 '17 00:06 njsmith

Side note. I pushed GCC 7.1 release: https://github.com/squeaky-pl/centos-devtools/releases/tag/7.1

squeaky-pl avatar Jun 17 '17 09:06 squeaky-pl

@squeaky-pl I've finally gotten to using your binaries, and they're great! Thanks! Things work out fine on 64-bit.

However, it seems as if that's ónly 64-bit, since my 32-bit builds are failing? Am I doing something wrong, or can't I use these binaries to build on the i686 manylinux docker image?

(EDIT: more specifically, the error seems to be centered around ccache: error: execv of /opt/devtools-7.1/bin/gcc failed: No such file or directory)

YannickJadoul avatar Jul 13 '17 16:07 YannickJadoul