Build error with CUDA 10.2 Jetson Nano

[alexismailov2]

Please help how can I use CTranslate2 with CUDA 10.2? The problem is in the Nvidia Jetson Nano does not work with CUDA 11 and 12(installation of cuda 11 or cuda 12 help to build but it does not work due to driver issue)… Cuda 10.2 will work well but looks like CTranslate2 needs at least CUDA 11… Provide solution how to fix CTranslate2 to build it for CUDA 10.2…

[guillaumekln]

CUDA 10 support was dropped 1 year ago.

What build error do you get? I think the main incompatibility with CUDA 10 is the usage of the C++17 standard.

https://github.com/OpenNMT/CTranslate2/blob/fd93a1f5548d0bcf60ab756cca2b37c109d54592/CMakeLists.txt#L426

You can try using c++14 in this line but it may require to some changes in the code also.

[alexismailov2]

During compiling I have gotten errors chained with missed headers If you need to understand what actually I can add here this info. But I think from your side will be better to create a CI/CD runner for NVIDIA Jetson Nano and you will see that problems. Could you provide a fork to keep support of building for NVIDIA Jetson Nano, because that pcb is enough power compared to another pcb in that price range(54$ for NVIDIA Jetson Nano 2Gb version). With installed CUDA 12.2 on Jetson Nano it build but not run due to some problems with internal driver which looks like was not updated by Nvidia for Jetson Nano.

[juansebashr]

Sorry @alexismailov2, have you encountered a solution? I'm facing the same problem

[alexismailov2]

@juansebashr Unfortunately still no, currently using Nvidia Jetson Orin Nano with cuda 11 support(expensive board but enough efficient). In addition I plan to introduce support of rk3588 NPU(Orange Pi 5, 6 TOPS) hope it will not be very difficult but could not say now anything…

[juansebashr]

@alexismailov2 Sadly I can't change the board :( haha, I will give it a try and write a comment if I accomplish anything

[alexismailov2]

@juansebashr I tried to build with cuda 10.2 but it produced an error with missed includes, may be localization of this dependency can be reduced…

[MuhammadShifa]

Hello @alexismailov2 , I am also trying to build with cuda 11.4 on Jeson Orin Nano but facing issue doing cmake .. CMake Error at CMakeLists.txt:273 (message): Intel OpenMP runtime libiomp5 not found

Please help me if you have solved this issue.

[alexismailov2]

You can turn it off with cmake option, I have build without problem it… try to search what option of ctranslate turn off openmp

[alexismailov2]

-DWITH_MKL=OFF

[MuhammadShifa]

Thank you @alexismailov2 , Yes I need to add this option -DWITH_MKL=OFF, but I don't know where to add this option. Although in main directory of ctranslae2 CMakeLists.txt I have turn off: option(WITH_MKL "Compile with Intel MKL backend" OFF) but still it showing me this error.

[alexismailov2]

cmake -Bbuild_folder -DWITH_MKL=OFF cmake —build build_folder

[MuhammadShifa]

@alexismailov2 Please look into this: I tried multiple command making enabled with cuda but still facing issue. Cranslate2 is the repository clone while I have mkdir build && cd build command and then cmake facing issue

[alexismailov2]

Add option -DOPENMP_RUNTIME=NONE

cmake -Bbuild_folder -DWITH_MKL=OFF -DOPENMP_RUNTIME=NONE -DWITH_CUDA=ON -DWITH_CUDNN=ON cmake —build build_folder

1. cmake can create directory for you with option -ByourSuperPuperFolderName, not needed to do redundant mkdir and after that go into with cd
2. Cmake can run build system by option —build yourSuperPuperFolderName

[MuhammadShifa]

Thank you so much @alexismailov2 . I have run the above command successfully. After this I am running the sudo make install command and it also working fine now but the system is in progress in installation. Thanks a lot for your precious time.

[alexismailov2]

You can even create wheel python package, for installation

[alexismailov2]

And be ready to understand that nvidia orin nano give good comparison only on big models and medium, on tiny unfortunately OrangePi 5 will be faster… (I am about FasterWhisper🙂)

[MuhammadShifa]

I appreciate your help. I would be happy if you share me a resource to create a wheel. Installation done 100% but again I am facing issue: e.py", line 10, in <module> import ctranslate2 ModuleNotFoundError: No module named 'ctranslate2'

I also want to run FasterWhisper medium model. It's working fine on my window based system but in Jetson Orin Nano I am facing this issue.

[alexismailov2]

Give me a second I will try to remember

1. go to python folder in the root of project
2. Install requirements with file requirements.txt
3. Run python setup.py bdist_wheel

[MuhammadShifa]

Yes I also follow these steps from here. I will update you soon.

[MuhammadShifa]

I followed your every steps and installed successfully. But I have face issue of cuda: ValueError: This CTranslate2 package was not compiled with CUDA support

I think I need to pass -DWITH_CUDA=ON then I think it will work. If you have any suggestion, Please tell me I think that's the reason it showed me this error.

[alexismailov2]

cmake -Bbuild_folder -DWITH_MKL=OFF -DOPENMP_RUNTIME=NONE -DWITH_CUDA=ON -DWITH_CUDNN=ON cmake —build build_folder

[alexismailov2]

I need to open my nvidia jetson nano and remember how to do that… I remember that I habe spend on it some time…

[MuhammadShifa]

Thank you so much @alexismailov2, Finally I have followed all the steps and installed CUDA enabled CTranslate2 on Jetson Orin Nano: Conclusion of all the steps in a sequence, hope it will also help the community:

1. git clone --recursive https://github.com/OpenNMT/CTranslate2.git
2. cd CTranslate2
3. cmake -Bbuild_folder -DWITH_MKL=OFF -DOPENMP_RUNTIME=NONE -DWITH_CUDA=ON -DWITH_CUDNN=ON
4. cmake —build build_folder
5. cd build_folder
6. sudo make install
7. change directory to python folder located in CTranslate2 directory cd python
8. pip3 install -r install_requirements.txt
9. python3 setup.py bdist_wheel
10. pip3 install dist/*.whl

Thank you once again for making my day great @alexismailov2 .

[alexismailov2]

If you work in some company which might need my expertise you can suggest me as a potential contractor!