faiss icon indicating copy to clipboard operation
faiss copied to clipboard
verified publisher icon facebookresearch

Error when compiling GPU version for aarch64

Open firekind opened this issue 3 years ago • 5 comments

Summary

Hey everyone, I am trying to compile the gpu version for aarch64 (specifically, for Nvidia's AGX Xavier) with compute capability 7.2 and with CUDA 10.2. The compilation fails with the following error:

ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 424; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 430; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 436; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 442; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 448; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 455; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 458; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 485; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 490; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 505; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 512; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 519; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 526; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 533; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 544; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 550; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 556; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 562; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 568; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 577; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 592; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 595; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 865; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 871; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 877; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 883; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 889; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 896; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 899; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 926; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 931; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 945; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 952; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 959; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 966; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 973; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 984; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 990; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 996; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1002; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1008; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1017; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1032; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1035; error   : Instruction 'vote' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1060; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1063; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1066; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1069; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1072; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1076; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1120; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1302; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1305; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1308; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1311; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1314; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1318; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4
ptxas /tmp/IVFUtils-7663dd/IVFUtils-sm_72.s, line 1353; error   : Instruction 'shfl' without '.sync' is not supported on .target sm_70 and higher from PTX ISA version 6.4

I also get a bunch of warnings:

ptxas /tmp/PQScanMultiPassPrecomputed-92abb0/PQScanMultiPassPrecomputed-sm_72.s, line 1662; warning : ld
.
.
.
ptxas /tmp/IVFPQ-f161db/IVFPQ-sm_72.s, line 20236; warning : ld
.
.
.

These warnings go on for a while, so just pasted a small snippet. Can these warning be ignored? And what do they mean? The compilation succeeds if I change the compute capability from 7.2 to 6.2, but the above error just becomes bunch of deprecation warnings, and the weird ld warnings still remain, but I need to compile it with compute capability 7.2

Platform

Docker image: nvcr.io/nvidia/deepstream:6.0.1-devel (sha256:a825db4ef06341d1c90f67d243f471a27ee29ae28bc39f92a555ca3daa67d8e8)

Faiss version: tag 1.7.2

Installed from: Compiled using clang 15 and bazel

Faiss compilation options: I translated the CMakeLists.txt to bazel BUILD files. Not compiling with MKL

Running on:

  • [ ] CPU
  • [x] GPU

Interface:

  • [x] C++
  • [ ] Python

firekind avatar Mar 20 '22 12:03 firekind

@mdouze any updates on this?

firekind avatar Mar 25 '22 03:03 firekind

Not sure if this helps in any way or if its right, but it looks like the problem is at the thrust::inclusive_scan call of runCalcListOffsets in faiss/gpu/impl/IVFUtils.cu. If I comment that line, compilation succeeds

firekind avatar Mar 26 '22 18:03 firekind

Unfortunately we don't have this combination of GPU and CPU, so we can't investigate.

mdouze avatar Mar 31 '22 10:03 mdouze

Will it help If I send a reproducible example? I'm trying to cross compile FAISS

firekind avatar Apr 03 '22 04:04 firekind

im also getting this while compiling faiss, but it seems to be just a warning? my builds fail later.

wesboyt avatar Nov 26 '22 20:11 wesboyt