How to activate BLAS?

[BadisG]

Hello,

I've heard that I could get BLAS activated through my intel i7 10700k by installing this library.

Unfortunatly, nothing happened, after compiling again with Clung I still have no BLAS in llama.cpp

system_info: n_threads = 14 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

Maybe it's just not possible I don't know, I need someone to tell me the truth 😅

[gjmulder]

Please review and use the issue template before submitting new issues

[hungerf3]

You'll need to edit the Makefile; look at the LLAMA_OPENBLAS section. Make sure the correct library and include paths are set for the BLAS library you want to use.

You'll also need to set LLAMA_OPENBLAS when you build; for example, add LLAMA_OPENBLAS=yes to the command line when you run make.

You might not see much improvement; the limit is likely memory bandwidth rather than processing power, and shuffling data between memory and the GPU might slow things down, but it's worth trying.

[KerfuffleV2]

@hungerf3

You might not see much improvement; the limit is likely memory bandwidth rather than processing power, and shuffling data between memory and the GPU might slow things down, but it's worth trying.

OpenBLAS just uses the CPU. I think there are other, separate BLAS implementations that may use GPU but that wouldn't apply in this case.

I gave it a try. You'll probably need both openblas and cblas installed. You may also need to add -lcblas to the link flags in the Makefile (#634). At least on my system, using OpenBLAS seemed a little bit faster but it was a small difference, probably less than 5%.

[cmp-nct]

On Windows I just gave up, wasted 3 hours of my life trying to get that BLASmonster somehow working. Barely anything to be found online, most references are for linux.

The best amount of chaos I had with that:

set(OpenBLAS_INCLUDE_DIR "C:/Program Files (x86)/OpenBLAS/include/openblas")
set(OpenBLAS_LIBRARIES "C:/Program Files (x86)/OpenBLAS/bin/openblas.dll")
set(BLAS_LIBRARIES "C:/Program Files (x86)/OpenBLAS/bin/openblas.dll")
include_directories(${OpenBLAS_INCLUDE_DIR})
add_compile_definitions(GGML_USE_OPENBLAS)
add_library(openblas SHARED IMPORTED)
add_link_options("/DELAYLOAD:openblas.dll")
add_link_options("/OPT:REF")

-- the code to find blas is not working (find_package() is not locating it) so the above part manually adds it The above code was added to the CMakeLists.txt manually.

Though that still doesn't properly load the DLL "unresolved external symbol cblas_sgemm referenced in function ggml_compute_forward_mul_mat_f16_f32" Must be something wrong in linking the dll, for another day/year

[BadisG]

@cmp-nct it's ok I found how to make it work.

1 - Download this https://github.com/xianyi/OpenBLAS/releases/tag/v0.3.22 2 - Put the files on a folder (I'll call it OpenBlasFolder) 3 - On the windows searsh write "Edit the system environnement variables" 4 - Click on "Environnement variables" 5 - On "System Variables" find PATH and click edit 6 - Add a new path that has OpenBlasFolder\bin in it (in my case it was something like D:\Large Language Models\OpenBlasFolder\bin) 7 - On your CMakeList.txt enable OpenBLAS

option(LLAMA_OPENBLAS               "llama: use OpenBLAS"                                   ON)

8 - On your CMakeList.txt, add this line just after "# Build libraries"

include_directories("D:/Large Language Models/OpenBlasFolder/include") <- For you the path will be something different of course

And there you go... gosh why is it so complicated to install a library seriously 😞

[sicp-dnegvp]

@BadisG adding the folder to the %PATH% makes cmake find OpenBLAS? It doesn't change anything for me.

-- Could NOT find BLAS (missing: BLAS_LIBRARIES)

Switching the LLAMA_OPENBLAS option to ON just enables the broken cmake find_package() check for OpenBLAS, which keeps failing, no matter what I do. If you want to bypass that and just manually add the paths and flags, you also need to add this after your include_directories:

add_compile_definitions(GGML_USE_OPENBLAS)
add_link_options("path_to_your_openblas_/lib/libopenblas.lib")

That will actually make it build with openblas (it actually depends on libopenblas.dll now). It just crashes on startup for me though. I think I'm giving up on this, it doesn't seem to be worth the effort in Windows.

[cmp-nct]

@cmp-nct it's ok I found how to make it work.

1 - Download this https://github.com/xianyi/OpenBLAS/releases/tag/v0.3.22 2 - Put the files on a folder (I'll call it OpenBlasFolder) 3 - On the windows searsh write "Edit the system environnement variables" 4 - Click on "Environnement variables" 5 - On "System Variables" find PATH and click edit 6 - Add a new path that has OpenBlasFolder\bin in it (in my case it was something like D:\Large Language Models\OpenBlasFolder\bin) 7 - On your CMakeList.txt enable OpenBLAS

option(LLAMA_OPENBLAS               "llama: use OpenBLAS"                                   ON)

8 - On your CMakeList.txt, add this line just after "# Build libraries"

include_directories("D:/Large Language Models/OpenBlasFolder/include") <- For you the path will be something different of course

And there you go... gosh why is it so complicated to install a library seriously 😞

In addition to that I had to add the -DBLAS_LIBRARIES= flag into settings.json as configurationArgs entry.

Now it indeed is compiled, also detected as BLAS = 1 The speed is exactly the same so it was useless.

I guess it might help to use the Intel BLAS lib, I just dislike adding 3.5 GB of libraries for a math multiplication.

[BadisG]

@cmp-nct If I had to guess I don't think adding the Intel BLAS lib will change anything, I'm pretty sure it's similar to the github BLAS I found.

So same, I didn't see significant increase of performance I also wasted my time 😅

[cmp-nct]

@cmp-nct If I had to guess I don't think adding the Intel BLAS lib will change anything, I'm pretty sure it's similar to the github BLAS I found.

So same, I didn't see significant increase of performance I also wasted my time 😅

Intel has a couple benchmarks up, significant differences to the open blas. But it's 3.5 gig to install. People these days don't know what efficiency means.

The same crap with the makefileon VScode I need to get llama compiled (llama.cpp is written awesomely). That cmake stuff is more complex than the source code and it depends on more files than the source code. It should be 10 lines to compile and link those 5 binaries but we've kilobytes of Make config to work through. So I'll skip on Intel Blas

[tomsnunes]

@BadisG adding the folder to the %PATH% makes cmake find OpenBLAS? It doesn't change anything for me.

-- Could NOT find BLAS (missing: BLAS_LIBRARIES)

Switching the LLAMA_OPENBLAS option to ON just enables the broken cmake find_package() check for OpenBLAS, which keeps failing, no matter what I do. If you want to bypass that and just manually add the paths and flags, you also need to add this after your include_directories:

add_compile_definitions(GGML_USE_OPENBLAS)
add_link_options("path_to_your_openblas_/lib/libopenblas.lib")

That will actually make it build with openblas (it actually depends on libopenblas.dll now). It just crashes on startup for me though. I think I'm giving up on this, it doesn't seem to be worth the effort in Windows.

I was able to compile on Windows adding the following to CMakeLists.txt:

include_directories("C:/libs/OpenBLAS/include")
add_compile_definitions(GGML_USE_OPENBLAS)
add_link_options("C:/libs/OpenBLAS/lib/libopenblas.dll.a")

Now talking about potential performance gains, it may be my impression, but I notice some gain in the response time of the model in relation to the binary compiled without OpenBLAS, but I will still do more tests.

For now, using the llama 7b template using llama.cpp (master) my initial result is: OpenBLAS enabled: 3.0843 OpenBLAS disabled: 3.1641

[rbrisita]

I was able to turn it on using the following for Windows 11 WSL2 Ubuntu 20.04:

git clone https://github.com/xianyi/OpenBLAS.git
cd OpenBLAS
make
make PREFIX=/usr/local/include/openblas install # The directory is in llama.cpp's Makefile.

cd ..
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make LLAMA_OPENBLAS=1

Next time you run llama.cpp you'll have BLAS turned on.

[rbrisita]

It was a little different on Debian 11. There are different versions available: libopenblas[0 | 64[-openmp | -pthread | -serial]]-dev. For this example, let's choose libopenblas-dev:

1. sudo apt install libopenblas-dev
2. Find include path and the lib file path of the installed library with dpkg -L libopenblas-dev.
3. cd llama.cpp
4. Modify Makefile to point to the include path, -I, in the CFLAGS variable.
5. Modify Makefile to point to the lib .so file in the LDFLAGS variable.
6. make clean
7. make LLAMA_OPENBLAS=1

Next time you run llama.cpp you'll have BLAS turned on.

[countzero]

Hi,

first of all @ggerganov thank you for this awesome project!

Problem

As a Windows user I also struggled to build llama.cpp with CMake and BLAS acceleration via OpenBLAS. The underlying problem seems to be that find_package(BLAS) does not find the OpenBLAS library. Even if it is located within the ./build directory. The error message is "Could NOT find BLAS (missing: BLAS_LIBRARIES)".

I debugged the CMake problem a bit further and described my findings there:

• https://discourse.cmake.org/t/find-package-blas-does-not-find-openblas/8414

Workaround

Based on the description of @tomsnunes I have found a reliable way to build llama.cpp with an OpenBLAS library binding on Windows with CMake.

1. Download and extract the latest OpenBLAS release
2. Copy the libopenblas.dll file into the ./build/bin/Release directory next to the main.exe
3. Add the following code on top of the CMakeLists.txt

# This is a workaround for a CMake bug on Windows to build llama.cpp
# with OpenBLAS. The find_package(BLAS) call fails to find OpenBLAS,
# so we have to link the 'libopenblas.dll' shared library manually.
# 
# @see https://github.com/ggerganov/llama.cpp/issues/627
# @see https://discourse.cmake.org/t/8414
# 
if (LLAMA_BLAS AND DEFINED LLAMA_BLAS_VENDOR)
    if (${LLAMA_BLAS_VENDOR} MATCHES "OpenBLAS")
        set(LLAMA_EXTRA_INCLUDES ${LLAMA_EXTRA_INCLUDES} "C:/absolute/path/to/OpenBLAS/include")
        set(LLAMA_EXTRA_LIBS ${LLAMA_EXTRA_LIBS} "C:/absolute/path/to/OpenBLAS/lib/libopenblas.dll.a")
        add_compile_definitions(GGML_USE_OPENBLAS)
    endif()
endif()

You can then build llama.cpp with OpenBLAS acceleration on Windows via CMake:

cmake -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS ..

Automation

Since llama.cpp changes frequently, I automated the complete build process with PowerShell:

• https://github.com/countzero/windows_llama.cpp

This enables you to simply execute the following to keep up to date with the llama.cpp project:

./rebuild_llama.cpp.ps1 -blasAccelerator "OpenBLAS"

Next Steps

I think eventually this problem should be fixed in the scope of CMake to not clutter the CMakeLists.txt. A temporary workaround in this project scope might be a new optional option parameter to inject the path to the OpenBLAS library for Windows users.

@ggerganov what do you think?

[loretoparisi]

libopenblas-dev

It was a little different on Debian 11. There are different versions available: libopenblas[0 | 64[-openmp | -pthread | -serial]]-dev. For this example, let's choose libopenblas-dev:

1. sudo apt install libopenblas-dev
2. Find include path and the lib file path of the installed library with dpkg -L libopenblas-dev.
3. cd llama.cpp
4. Modify Makefile to point to the include path, -I, in the CFLAGS variable.
5. Modify Makefile to point to the lib .so file in the LDFLAGS variable.
6. make clean
7. make LLAMA_OPENBLAS=1

Next time you run llama.cpp you'll have BLAS turned on.

I do not see the library files here

dpkg -L libopenblas-dev
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libopenblas-dev
/usr/share/doc/libopenblas-dev/BACKERS.md
/usr/share/doc/libopenblas-dev/CONTRIBUTORS.md.gz
/usr/share/doc/libopenblas-dev/README.md.gz
/usr/share/doc/libopenblas-dev/USAGE.md.gz
/usr/share/doc/libopenblas-dev/changelog.Debian.gz
/usr/share/doc/libopenblas-dev/changelog.gz
/usr/share/doc/libopenblas-dev/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/libopenblas-dev

[rbrisita]

libopenblas-dev

It was a little different on Debian 11. There are different versions available: libopenblas[0 | 64[-openmp | -pthread | -serial]]-dev. For this example, let's choose libopenblas-dev:

1. sudo apt install libopenblas-dev
2. Find include path and the lib file path of the installed library with dpkg -L libopenblas-dev.
3. cd llama.cpp
4. Modify Makefile to point to the include path, -I, in the CFLAGS variable.
5. Modify Makefile to point to the lib .so file in the LDFLAGS variable.
6. make clean
7. make LLAMA_OPENBLAS=1

Next time you run llama.cpp you'll have BLAS turned on.

I do not see the library files here

dpkg -L libopenblas-dev
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libopenblas-dev
/usr/share/doc/libopenblas-dev/BACKERS.md
/usr/share/doc/libopenblas-dev/CONTRIBUTORS.md.gz
/usr/share/doc/libopenblas-dev/README.md.gz
/usr/share/doc/libopenblas-dev/USAGE.md.gz
/usr/share/doc/libopenblas-dev/changelog.Debian.gz
/usr/share/doc/libopenblas-dev/changelog.gz
/usr/share/doc/libopenblas-dev/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/libopenblas-dev

Try installing one of the other versions like libopenblas-pthread-dev.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.