milvus icon indicating copy to clipboard operation
milvus copied to clipboard

[Bug]: Search failed with SIGILL: illegal instruction

Open MikeVL opened this issue 1 year ago • 7 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Environment

- Milvus version: 2.3.3
- Deployment mode: cluster
- MQ type: pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2): attu v2.3.4
- GPU: no
- Others: Kubernetes helm setup

Current Behavior

Vector Search query

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

[2023/12/14 11:14:21.691 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=0] [newVersion=1702552371687878643] [growingSegmentNum=0] [sealedSegmentNum=1]
SIGILL: illegal instruction
PC=0x7fb407333e71 m=19 sigcode=2
signal arrived during cgo execution
instruction bytes: 0xc4 0xe2 0x69 0x99 0x5 0x96 0xe0 0x38 0x0 0xc4 0xc1 0xe2 0x2a 0xce 0xc5 0xf2
[milvus-querynode.log](https://github.com/milvus-io/milvus/files/13672569/milvus-querynode.log)

milvus-querynode.log

Anything else?

No response

MikeVL avatar Dec 14 '23 11:12 MikeVL

What is your cpu arch? can you lscpu to see if it has avx2 instruction set.

chasingegg avatar Dec 14 '23 11:12 chasingegg

/assign @chasingegg /unassign

yanliang567 avatar Dec 14 '23 11:12 yanliang567

Flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology
tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm
cpuid_fault pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust smep arat md_clear flush_l1d arch_capabilities

We have two type CPU

  • Intel(R) Xeon(R) CPU E5-2696 v2
  • Intel(R) Xeon(R) CPU E5-2697 v4

MikeVL avatar Dec 14 '23 11:12 MikeVL

@MikeVL We have fixed it in 2.3.4 for whom does not have avx2 machine, could you try compile milvus from master or 2.3 branch or wait for the coming 2.3.4 release. And avx2 machines should work.

chasingegg avatar Dec 14 '23 12:12 chasingegg

I'll wait for the next release, thanks.

MikeVL avatar Dec 14 '23 12:12 MikeVL

@chasingegg Any updates on this issue? The milestone is being pushed further and further :(

nurmukhametdaniyar avatar Feb 21 '24 16:02 nurmukhametdaniyar

as comments from @chasingegg it was fixed, please retry and if it keeps failing, please attache new log files.

yanliang567 avatar Feb 22 '24 01:02 yanliang567

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

stale[bot] avatar Jun 16 '24 04:06 stale[bot]