botan
botan copied to clipboard
randombit
Botan 2.18.0 won't compile using ICC.
Hello.
I'm getting errors when I try to build and compile the library with ICC. make ends with errors, prompting several messages like:
[...]
src/lib/filters/threaded_fork.cpp(85): error #308: member "Botan::Filter::m_next" (declared at line 134 of "build/include/botan/filter.h") is inaccessible
n = m_next.size();
^
src/lib/filters/threaded_fork.cpp(97): error #308: member "Botan::Filter::m_next" (declared at line 134 of "build/include/botan/filter.h") is inaccessible
std::bind(&Threaded_Fork::thread_entry, this, m_next[i]))));
^
src/lib/filters/threaded_fork.cpp(104): error #308: member "Botan::Filter::m_write_queue" (declared at line 133 of "build/include/botan/filter.h") is inaccessible
if(m_write_queue.size())
^
src/lib/filters/threaded_fork.cpp(105): error #308: member "Botan::Filter::m_write_queue" (declared at line 133 of "build/include/botan/filter.h") is inaccessible
thread_delegate_work(m_write_queue.data(), m_write_queue.size());
^
src/lib/filters/threaded_fork.cpp(105): error #308: member "Botan::Filter::m_write_queue" (declared at line 133 of "build/include/botan/filter.h") is inaccessible
thread_delegate_work(m_write_queue.data(), m_write_queue.size());
^
src/lib/filters/threaded_fork.cpp(109): error #308: function "Botan::Filter::total_ports" (declared at line 109 of "build/include/botan/filter.h") is inaccessible
for(size_t j = 0; j != total_ports(); ++j)
^
src/lib/filters/threaded_fork.cpp(110): error #308: member "Botan::Filter::m_next" (declared at line 134 of "build/include/botan/filter.h") is inaccessible
if(m_next[j])
^
src/lib/filters/threaded_fork.cpp(114): error #308: member "Botan::Filter::m_write_queue" (declared at line 133 of "build/include/botan/filter.h") is inaccessible
m_write_queue += std::make_pair(input, length);
^
src/lib/filters/threaded_fork.cpp(116): error #308: member "Botan::Filter::m_write_queue" (declared at line 133 of "build/include/botan/filter.h") is inaccessible
m_write_queue.clear();
^
src/lib/filters/threaded_fork.cpp(126): error #308: function "Botan::Filter::total_ports" (declared at line 109 of "build/include/botan/filter.h") is inaccessible
m_thread_data->m_input_complete_barrier.wait(total_ports() + 1);
^
src/lib/filters/threaded_fork.cpp(127): error #308: function "Botan::Filter::total_ports" (declared at line 109 of "build/include/botan/filter.h") is inaccessible
m_thread_data->m_input_ready_semaphore.release(total_ports());
compilation aborted for src/lib/filters/threaded_fork.cpp (code 2)
make: *** [Makefile:344: build/obj/lib/filters_threaded_fork.o] Error 2
make: *** Waiting for unfinished jobs....
I'm using Botan 2.18.0 (cloned from https://github.com/randombit/botan/commit/550df679ce692a9461e0fc0022ada036c94132a8).
I'm executing the following commands to start the compilation:
$ export CXXFLAGS="-DNDEBUG -g2 -O0"
$ export CXX="icc -m32" (or -m64, neither of them works)
$ make -j 4
After setting and launching make, the resultant command is of the like of:
icpc -fPIC -std=c++17 -O1 -m64 -DBOTAN_IS_BEING_BUILT -w1 -Ibuild/include -Ibuild/include/external -c src/lib/hash/checksum/crc24/crc24.cpp -o build/obj/lib/hash_checksum_crc24.o
The version of ICC I'm using is:
$ icc -v
icc version 2021.2.0 (gcc version 8.3.0 compatibility)
Last but not least, this is the environment I'm working on:
$ uname -a
Linux rdp05 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux
$ lsb_release -a
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 36 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz
Stepping: 9
CPU MHz: 2892.725
CPU max MHz: 3200.0000
CPU min MHz: 1600.0000
BogoMIPS: 6385.53
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm arat pln pts
So far I've tried using either -m32 or -m64 and none of them works. I also tried using simply make (with no -j 4) and it was unseccessful as well.
What can I do in order to be able to build and compile Botan with ICC?
Thanks for the report. I think this is a bug in ICC but is hopefully easy to work around. Can you try this patch?
diff --git a/src/lib/filters/filter.h b/src/lib/filters/filter.h
index 94b9c6ccd..93e4fe9ee 100644
--- a/src/lib/filters/filter.h
+++ b/src/lib/filters/filter.h
@@ -105,6 +105,7 @@ class BOTAN_PUBLIC_API(2,0) Filter
friend class Pipe;
friend class Fanout_Filter;
+ friend class Threaded_Fork;
size_t total_ports() const;
size_t current_port() const { return m_port_num; }
@@ -154,12 +155,6 @@ class BOTAN_PUBLIC_API(2,0) Fanout_Filter : public Filter
void set_next(Filter* f[], size_t n) { Filter::set_next(f, n); }
void attach(Filter* f) { Filter::attach(f); }
-
- private:
- friend class Threaded_Fork;
- using Filter::m_write_queue;
- using Filter::total_ports;
- using Filter::m_next;
};
You're most welcome @randombit.
Making the changes you suggested did indeed solve the problem. However, there are now other compiling errors:
src/lib/utils/cpuid/cpuid_x86.cpp(36): error: argument of type "uint32_t={__uint32_t={unsigned int}} *" is incompatible with parameter of type "int *"
__cpuid(out, type);
^
src/lib/utils/cpuid/cpuid_x86.cpp(57): error: identifier "__cpuidex" is undefined
__cpuidex((int*)out, type, level);
^
compilation aborted for src/lib/utils/cpuid/cpuid_x86.cpp (code 2)
make: *** [Makefile:1076: build/obj/lib/utils_cpuid_x86.o] Error 2
make: *** Waiting for unfinished jobs....
Just like before, it happens with both -m32 and -m64 and regardless of the optimization flag.
icpc -fPIC -std=c++17 -Os -m32 -DBOTAN_IS_BEING_BUILT -w1 -Ibuild/include -Ibuild/include/external -c src/lib/utils/ct_utils.cpp -o build/obj/lib/utils_ct_utils.o
The first problem is just a missing cast and easily remedied.
I'm really not sure what to make of __cpuidex missing. This used to work (though it's been many years since I used icc). I think following patch will address the build problems. But, it will have the side effect that for 32-bit builds, support for extensions like AVX2 and SHA-NI will be disabled, because the library will no longer be able to detect them. To fix this requires adding support for whatever icc has replaced __cpuidex with. It's also possible that the header we're supposed to use for it has changed and __cpuidex is still supported.
diff --git a/src/lib/utils/cpuid/cpuid_x86.cpp b/src/lib/utils/cpuid/cpuid_x86.cpp
index 88c8a9d8f..c54281161 100644
--- a/src/lib/utils/cpuid/cpuid_x86.cpp
+++ b/src/lib/utils/cpuid/cpuid_x86.cpp
@@ -29,12 +29,9 @@ namespace {
void invoke_cpuid(uint32_t type, uint32_t out[4])
{
-#if defined(BOTAN_BUILD_COMPILER_IS_MSVC)
+#if defined(BOTAN_BUILD_COMPILER_IS_MSVC) || defined(BOTAN_BUILD_COMPILER_IS_INTEL)
__cpuid((int*)out, type);
-#elif defined(BOTAN_BUILD_COMPILER_IS_INTEL)
- __cpuid(out, type);
-
#elif defined(BOTAN_TARGET_ARCH_IS_X86_64) && defined(BOTAN_USE_GCC_INLINE_ASM)
asm("cpuid\n\t"
: "=a" (out[0]), "=b" (out[1]), "=c" (out[2]), "=d" (out[3])
@@ -53,9 +50,6 @@ void invoke_cpuid_sublevel(uint32_t type, uint32_t level, uint32_t out[4])
#if defined(BOTAN_BUILD_COMPILER_IS_MSVC)
__cpuidex((int*)out, type, level);
-#elif defined(BOTAN_BUILD_COMPILER_IS_INTEL)
- __cpuidex((int*)out, type, level);
-
#elif defined(BOTAN_TARGET_ARCH_IS_X86_64) && defined(BOTAN_USE_GCC_INLINE_ASM)
asm("cpuid\n\t"
: "=a" (out[0]), "=b" (out[1]), "=c" (out[2]), "=d" (out[3]) \
OK this is weird - Godbolt supports ICC and there this compiles fine?!? https://godbolt.org/z/Pfnh4roGf what is different?
Can't test it at the moment, but I'll do it as fast as I can. Maybe it is a problem related to how I installed ICC.
- I downloaded the local version 2021.2.0 from here.
- I executed the downloaded binary.
- Every time I want to use it, I must source a script called setvars.sh:
. ./setvars.sh intel64
Could it be that there is something wrong with my installation?
I will pull all the changes you did and try to compile it again as soon as I can. I will provide feedback.
I have pulled all your changes and the problem still persists... The new error is:
src/lib/utils/cpuid/cpuid_x86.cpp(54): error: identifier "__cpuidex" is undefined
__cpuidex((int*)out, type, level);
^
compilation aborted for src/lib/utils/cpuid/cpuid_x86.cpp (code 2)
make: *** [Makefile:1076: build/obj/lib/utils_cpuid_x86.o] Error 2
make: *** Waiting for unfinished jobs...
I hadn't decided what to do about __cpuidex until just now. It still seems very strange that it works on godbolt but not for you. Can you try 4aa1a8b?
The problem with __cpuindex seems fixed now. However, the compilation is still unsuccessful. There are, literally, thousands of errors like:
[...]
ld: build/obj/lib/x509_x509path.o: in function `std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<Botan::PKIX::check_ocsp_online(std::vector<Botan::X509_Certificate, std::allocator<Botan::X509_Certificate> > const&, std::vector<Botan::Certificate_Store*, std::allocator<Botan::Certificate_Store*> > const&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> > >, std::chrono::duration<long long, std::ratio<1ll, 1000ll> >, bool, std::chrono::duration<long long, std::ratio<1ll, 1ll> >)::{lambda()#2}> >, std::optional<Botan::OCSP::Response> >::~_Deferred_state()':
x509path.cpp:(.text._ZNSt13__future_base15_Deferred_stateINSt6thread8_InvokerISt5tupleIJZN5Botan4PKIX17check_ocsp_onlineERKSt6vectorINS4_16X509_CertificateESaIS7_EERKS6_IPNS4_17Certificate_StoreESaISD_EENSt6chrono10time_pointINSI_3_V212system_clockENSI_8durationIxSt5ratioILx1ELx1000000000EEEEEENSM_IxSN_ILx1ELx1000EEEEbNSM_IxSN_ILx1ELx1EEEEEUlvE0_EEEESt8optionalINS4_4OCSP8ResponseEEED0Ev[_ZNSt13__future_base15_Deferred_stateINSt6thread8_InvokerISt5tupleIJZN5Botan4PKIX17check_ocsp_onlineERKSt6vectorINS4_16X509_CertificateESaIS7_EERKS6_IPNS4_17Certificate_StoreESaISD_EENSt6chrono10time_pointINSI_3_V212system_clockENSI_8durationIxSt5ratioILx1ELx1000000000EEEEEENSM_IxSN_ILx1ELx1000EEEEbNSM_IxSN_ILx1ELx1EEEEEUlvE0_EEEESt8optionalINS4_4OCSP8ResponseEEED0Ev]+0x22): undefined reference to `operator delete(void*, unsigned int)'
ld: build/obj/lib/x509_x509path.o: in function `std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<Botan::PKIX::check_ocsp_online(std::vector<Botan::X509_Certificate, std::allocator<Botan::X509_Certificate> > const&, std::vector<Botan::Certificate_Store*, std::allocator<Botan::Certificate_Store*> > const&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> > >, std::chrono::duration<long long, std::ratio<1ll, 1000ll> >, bool, std::chrono::duration<long long, std::ratio<1ll, 1ll> >)::{lambda()#1}> >, std::optional<Botan::OCSP::Response> >::~_Deferred_state()':
x509path.cpp:(.text._ZNSt13__future_base15_Deferred_stateINSt6thread8_InvokerISt5tupleIJZN5Botan4PKIX17check_ocsp_onlineERKSt6vectorINS4_16X509_CertificateESaIS7_EERKS6_IPNS4_17Certificate_StoreESaISD_EENSt6chrono10time_pointINSI_3_V212system_clockENSI_8durationIxSt5ratioILx1ELx1000000000EEEEEENSM_IxSN_ILx1ELx1000EEEEbNSM_IxSN_ILx1ELx1EEEEEUlvE_EEEESt8optionalINS4_4OCSP8ResponseEEED0Ev[_ZNSt13__future_base15_Deferred_stateINSt6thread8_InvokerISt5tupleIJZN5Botan4PKIX17check_ocsp_onlineERKSt6vectorINS4_16X509_CertificateESaIS7_EERKS6_IPNS4_17Certificate_StoreESaISD_EENSt6chrono10time_pointINSI_3_V212system_clockENSI_8durationIxSt5ratioILx1ELx1000000000EEEEEENSM_IxSN_ILx1ELx1000EEEEbNSM_IxSN_ILx1ELx1EEEEEUlvE_EEEESt8optionalINS4_4OCSP8ResponseEEED0Ev]+0x22): undefined reference to `operator delete(void*, unsigned int)'
ld: build/obj/lib/x509_x509self.o: in function `Botan::X509::create_self_signed_cert(Botan::X509_Cert_Options const&, Botan::Private_Key const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Botan::RandomNumberGenerator&)':
x509self.cpp:(.text+0x3e3): undefined reference to `operator new(unsigned int)'
ld: x509self.cpp:(.text+0x444): undefined reference to `operator new(unsigned int)'
ld: x509self.cpp:(.text+0x88a): undefined reference to `operator delete(void*, unsigned int)'
ld: x509self.cpp:(.text+0x8c5): undefined reference to `operator delete(void*, unsigned int)'
ld: build/obj/lib/x509_x509self.o: in function `std::_MakeUniq<Botan::Cert_Extension::Key_Usage>::__single_object std::make_unique<Botan::Cert_Extension::Key_Usage, Botan::Key_Constraints&>(Botan::Key_Constraints&)':
x509self.cpp:(.text._ZSt11make_uniqueIN5Botan14Cert_Extension9Key_UsageEJRNS0_15Key_ConstraintsEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_[_ZSt11make_uniqueIN5Botan14Cert_Extension9Key_UsageEJRNS0_15Key_ConstraintsEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_]+0x14): undefined reference to `operator new(unsigned int)'
ld: build/obj/lib/x509_x509self.o: in function `std::_MakeUniq<Botan::Cert_Extension::Basic_Constraints>::__single_object std::make_unique<Botan::Cert_Extension::Basic_Constraints, bool const&, unsigned int const&>(bool const&, unsigned int const&)':
x509self.cpp:(.text._ZSt11make_uniqueIN5Botan14Cert_Extension17Basic_ConstraintsEJRKbRKjEENSt9_MakeUniqIT_E15__single_objectEDpOT0_[_ZSt11make_uniqueIN5Botan14Cert_Extension17Basic_ConstraintsEJRKbRKjEENSt9_MakeUniqIT_E15__single_objectEDpOT0_]+0x15): undefined reference to `operator new(unsigned int)'
ld: build/obj/lib/x509_x509self.o: in function `std::_MakeUniq<Botan::Cert_Extension::Extended_Key_Usage>::__single_object std::make_unique<Botan::Cert_Extension::Extended_Key_Usage, std::vector<Botan::OID, std::allocator<Botan::OID> > const&>(std::vector<Botan::OID, std::allocator<Botan::OID> > const&)':
x509self.cpp:(.text._ZSt11make_uniqueIN5Botan14Cert_Extension18Extended_Key_UsageEJRKSt6vectorINS0_3OIDESaIS4_EEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_[_ZSt11make_uniqueIN5Botan14Cert_Extension18Extended_Key_UsageEJRKSt6vectorINS0_3OIDESaIS4_EEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_]+0x16): undefined reference to `operator new(unsigned int)'
ld: x509self.cpp:(.text._ZSt11make_uniqueIN5Botan14Cert_Extension18Extended_Key_UsageEJRKSt6vectorINS0_3OIDESaIS4_EEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_[_ZSt11make_uniqueIN5Botan14Cert_Extension18Extended_Key_UsageEJRKSt6vectorINS0_3OIDESaIS4_EEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_]+0x80): undefined reference to `operator delete(void*, unsigned int)'
ld: build/obj/lib/x509_x509self.o: in function `std::_MakeUniq<Botan::Cert_Extension::Subject_Alternative_Name>::__single_object std::make_unique<Botan::Cert_Extension::Subject_Alternative_Name, Botan::AlternativeName&>(Botan::AlternativeName&)':
x509self.cpp:(.text._ZSt11make_uniqueIN5Botan14Cert_Extension24Subject_Alternative_NameEJRNS0_15AlternativeNameEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_[_ZSt11make_uniqueIN5Botan14Cert_Extension24Subject_Alternative_NameEJRNS0_15AlternativeNameEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_]+0x16): undefined reference to `operator new(unsigned int)'
ld: x509self.cpp:(.text._ZSt11make_uniqueIN5Botan14Cert_Extension24Subject_Alternative_NameEJRNS0_15AlternativeNameEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_[_ZSt11make_uniqueIN5Botan14Cert_Extension24Subject_Alternative_NameEJRNS0_15AlternativeNameEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_]+0x80): undefined reference to `operator delete(void*, unsigned int)'
ld: build/obj/lib/asn1_alg_id.o: file class ELFCLASS32 incompatible with ELFCLASS64
ld: final link failed: file in wrong format
make: *** [Makefile:88: libbotan-3.so.0] Error 1
make: *** Waiting for unfinished jobs....
I don't know what's wrong. Maybe it's just my environment and my ICC version...
ld: build/obj/lib/asn1_alg_id.o: file class ELFCLASS32 incompatible with ELFCLASS64
You are mixing 32-bit and 64-bit object files. You probably need to do a make clean and rebuild.
I am not mixing architectures. I am building ICC for 64 bits and I want to use it with -m32 to compile for 32bits. In other words:
$ source ./setvars.sh intel64 # Setting ICC environment variables por intel64
$ ./configure.py --enable-modules=aes,ofb,ctr,cbc,cfb,des,chacha --cc=icc --cxxflags="O1" --cpu=x86_32 # Configuring BOTAN
$ make -j 4
I will try other ICC version, but it does not look promising to me.
Hi again.
After a lot of try-outs, I'm still unsuccessful when compiling the library.
I reinstalled the compiler but something is not right in the environment setup. I'm getting the following error:
In file included from /usr/lib/gcc/x86_64-linux-gnu/8/include/immintrin.h(41),
from src/lib/block/threefish_512/threefish_512_avx2/threefish_512_avx2.cpp(9):
/usr/lib/gcc/x86_64-linux-gnu/8/include/avxintrin.h(615): error: identifier "____m128d" is undefined
return (__m128d) __builtin_ia32_vpermilvarpd ((__v2df)__A,
^
In file included from /usr/lib/gcc/x86_64-linux-gnu/8/include/immintrin.h(41),
from src/lib/block/threefish_512/threefish_512_avx2/threefish_512_avx2.cpp(9):
/usr/lib/gcc/x86_64-linux-gnu/8/include/avxintrin.h(615): error: expected a ")"
return (__m128d) __builtin_ia32_vpermilvarpd ((__v2df)__A,
^
In file included from /usr/lib/gcc/x86_64-linux-gnu/8/include/immintrin.h(41),
from src/lib/block/threefish_512/threefish_512_avx2/threefish_512_avx2.cpp(9):
/usr/lib/gcc/x86_64-linux-gnu/8/include/avxintrin.h(615): internal error: assertion failed at: "sys_predef.c", line 490 in enter_builtin_function
return (__m128d) __builtin_ia32_vpermilvarpd ((__v2df)__A,
^
compilation aborted for src/lib/block/threefish_512/threefish_512_avx2/threefish_512_avx2.cpp (code 4)
make: *** [Makefile:200: build/obj/lib/block_threefish_512_threefish_512_avx2.o] Error 4
make: *** Waiting for unfinished jobs....
As you can see, regardless of executing INFO: ./configure.py invoked with options "--enable-modules=aes,ofb,ctr,cbc,cfb,des,chacha --cc=icc --cxxflags=-O1 -m32 --cpu=x86_32 --disable-cc-tests" it is trying to access gcc headers rather than ICC's...
I have run out of ideas and it is not a Botan problem anymore. Now it seems to be a problem of the so-called Intel's oneapi and its unintuitive scripts.
Thank you for your help.
Closing since it seems anything that can be done on our side has been addressed already.