ursa
ursa copied to clipboard
fleek-network
SIGILL on some XEON CPU caused by prebuilt bin in Docker container
Description
The Ursa process finishes with exit code 132 (interrupted by signal 4: SIGILL) on some XEON CPUs.
A Docker image is provided, prebuilt on an x86_64 DigitalOcean Intel vCPU machine, which is not supported by some Intel processors that VPS providers might supply.
A few examples of CPU are:
- Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Here are a few examples of VPS provider's client VPS where this was observed:
- https://cloudcone.com
- https://pq.hosting
As a temporary solution, the user is asked to build the image instead of accepting the recommended "latest" or "nightly". This means that cargo build enables or disables some CPU features by considering the native CPU in which the build is launched.
Environment
- We're providing a prebuilt Docker image, cargo built by a GitHub workflow
- On the prebuilt binary, run as a Docker container
- The CI runs Ubuntu's latest
Demo
As discussed and observed in: https://discord.com/channels/965698989464887386/1047823650541277194/1090335532669735053
Another user: https://discord.com/channels/965698989464887386/1047823650541277194/1090966733130580008
For clarity, this has nothing to do with Docker. Here we see a binary built natively DO Intel Ubuntu latest x86_64, attempt run on the XEON E5 VPS:
Notes
Debugging details
Starting program: /usr/local/bin/ursa
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fb8b1615700 (LWP 627)]
[New Thread 0x7fb8b1414700 (LWP 628)]
[New Thread 0x7fb8b120d700 (LWP 629)]
[New Thread 0x7fb8b1009700 (LWP 630)]
2023-03-30T18:47:59.665085Z INFO ursa::config: Loading config from: "/root/.ursa/config.toml"
at crates/ursa/src/config.rs:36
2023-03-30T18:47:59.679817Z INFO ursa::ursa::identity: Loaded identity `default` (12D3KooWSUuPHmNLj3z9TxjdHjHpEdb1Pr49cWbogdC1M6ZjH2r1)
at crates/ursa/src/ursa/identity.rs:155
2023-03-30T18:47:59.680409Z INFO ursa_consensus::keys: Creating '"/root/.ursa/keystore/consensus/primary.key"' because it doesn't exists.
at crates/ursa-consensus/src/keys.rs:43
Thread 1 "ursa" received signal SIGILL, Illegal instruction.
0x000056316b498a2f in ?? ()
Asssembly
multi-thre Thread 0x7fce3fdb63 In: L?? PC: 0x5573c33eea2f
┌────────────────────────────────────────────────────────────────────────┐
│ >0x5573c33eea2f mulx %r14,%rax,%r11 │
│ 0x5573c33eea34 call 0x5573c33eeae0 │
│ 0x5573c33eea39 mov 0x8(%rsp),%r15 │
│ 0x5573c33eea3e mov 0x10(%rsp),%r14 │
│ 0x5573c33eea43 mov 0x18(%rsp),%r13 │
│ 0x5573c33eea48 mov 0x20(%rsp),%r12 │
│ 0x5573c33eea4d mov 0x28(%rsp),%rbx │
│ 0x5573c33eea52 mov 0x30(%rsp),%rbp │
│ 0x5573c33eea57 lea 0x38(%rsp),%rsp │
│ 0x5573c33eea5c repz ret │
│ 0x5573c33eea5e xchg %ax,%ax │
│ 0x5573c33eea60 endbr64 │
│ 0x5573c33eea64 push %rbp │
│ 0x5573c33eea65 push %rbx │
│ 0x5573c33eea66 push %r12 │
│ 0x5573c33eea68 push %r13 │
│ 0x5573c33eea6a push %r14 │
│ 0x5573c33eea6c push %r15 │
│ 0x5573c33eea6e sub $0x8,%rsp │
│ 0x5573c33eea72 mov %rsi,%rbx │
│ 0x5573c33eea75 mov %rcx,%r8 │
│ 0x5573c33eea78 mov %rdx,%rcx │
│ 0x5573c33eea7b mov (%rsi),%rdx │
│ 0x5573c33eea7e mov 0x8(%rsi),%r15 │
└────────────────────────────────────────────────────────────────────────┘
multi-thre Thread 0x7fce3fdb63 In: L?? PC: 0x5573c33eea2f
(gdb)
A mulx described here https://docs.oracle.com/cd/E37838_01/html/E61064/gnyak.html seems to be a CPU feature not available on older intel CPU such like the XEON E5 family above.
Obs: A cargo build with target x86_64-unknown-linux-gnu was attempted, but that seems to be used by default when no --target is specified. This was tested in a Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz VPS shared by a community member " LinKong| Sui Global" (on Discord).
Checklist
- [x] I have ensured that my version is up-to-date
- [x] I have ensured that my issue is reproducible
- [x] I have ensured that my issue is not a duplicate
Another CPU where this happens, as reported by a different user.
🤖 CPU details
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
stepping : 1
microcode : 0x1
cpu MHz : 2594.122
cache size : 16384 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 8
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5188.24
clflush size : 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
stepping : 1
microcode : 0x1
cpu MHz : 2594.122
cache size : 16384 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 8
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5188.24
clflush size : 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
stepping : 1
microcode : 0x1
cpu MHz : 2594.122
cache size : 16384 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 8
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5188.24
clflush size : 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
stepping : 1
microcode : 0x1
cpu MHz : 2594.122
cache size : 16384 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 8
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5188.24
clflush size : 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
stepping : 1
microcode : 0x1
cpu MHz : 2594.122
cache size : 16384 KB
physical id : 0
siblings : 8
core id : 4
cpu cores : 8
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5188.24
clflush size : 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
stepping : 1
microcode : 0x1
cpu MHz : 2594.122
cache size : 16384 KB
physical id : 0
siblings : 8
core id : 5
cpu cores : 8
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5188.24
clflush size : 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
stepping : 1
microcode : 0x1
cpu MHz : 2594.122
cache size : 16384 KB
physical id : 0
siblings : 8
core id : 6
cpu cores : 8
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5188.24
clflush size : 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
stepping : 1
microcode : 0x1
cpu MHz : 2594.122
cache size : 16384 KB
physical id : 0
siblings : 8
core id : 7
cpu cores : 8
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5188.24
clflush size : 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
Another user report
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz
stepping : 4
microcode : 0x42e
cpu MHz : 1799.979
cache size : 10240 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm arat pln pts md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 3599.95
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz
stepping : 4
microcode : 0x42e
cpu MHz : 1799.970
cache size : 10240 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm arat pln pts md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 3599.95
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz
stepping : 4
microcode : 0x42e
cpu MHz : 1799.975
cache size : 10240 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm arat pln pts md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 3599.95
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz
stepping : 4
microcode : 0x42e
cpu MHz : 1799.972
cache size : 10240 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm arat pln pts md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 3599.95
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management: