Unable to run example Project

[Jafesu]

I tried to duplicate the example project in the README, but every time i run it, i get the following error

Unhandled exception. System.TypeInitializationException: The type initializer for 'LLama.Native.NativeApi' threw an exception.
 ---> LLama.Exceptions.RuntimeError: The native library cannot be correctly loaded. It could be one of the following reasons:
1. No LLamaSharp backend was installed. Please search LLamaSharp.Backend and install one of them.
2. You are using a device with only CPU but installed cuda backend. Please install cpu backend instead.
3. One of the dependency of the native library is missed. Please use `ldd` on linux, `dumpbin` on windows and `otool`to check if all the dependency of the native library is satisfied. Generally you could find the libraries under your output folder.
4. Try to compile llama.cpp yourself to generate a libllama library, then use `LLama.Native.NativeLibraryConfig.WithLibrary` to specify it at the very beginning of your code. For more informations about compilation, please refer to LLamaSharp repo on github.

   at LLama.Native.NativeApi..cctor()
   --- End of inner exception stack trace ---
   at LLama.Native.NativeApi.llama_max_devices()
   at LLama.Abstractions.TensorSplitsCollection..ctor()
   at LLama.Common.ModelParams..ctor(String modelPath)
   at Program.<Main>$(String[] args) in D:\projects\testing\Llama\LLamaTesting\Program.cs:line 7
   at Program.<Main>(String[] args)

D:\projects\testing\Llama\LLamaTesting\bin\x64\Debug\net8.0\LLamaTesting.exe (process 48408) exited with code -532462766.

I have LLamaSharp.Backend.Cpu installed

Here is my code I am using

using LLama;
using LLama.Common;

string modelPath = "./models/thespis-13b-v0.5.Q5_K_S.gguf";
var prompt = "Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\r\n\r\nUser: Hello, Bob.\r\nBob: Hello. How may I help you today?\r\nUser: Please tell me the largest city in Europe.\r\nBob: Sure. The largest city in Europe is Moscow, the capital of Russia.\r\nUser:"; // use the "chat-with-bob" prompt here.

var parameters = new ModelParams(modelPath)
{
    ContextSize = 1024,
    Seed = 1337,
    GpuLayerCount = 5

};
using var model = LLamaWeights.LoadFromFile(parameters);
using var context = model.CreateContext(parameters);
var executor = new InteractiveExecutor(context);

var session = new ChatSession(executor);

Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine("The chat session has started. In this example, the prompt is printed for better visual result.");
Console.ForegroundColor = ConsoleColor.White;

// show the prompt
Console.Write(prompt);
while (true)
{
    await foreach (var text in session.ChatAsync(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { "User:" } }))
    {
        Console.Write(text);
    }

    Console.ForegroundColor = ConsoleColor.Green;
    prompt = Console.ReadLine();
    Console.ForegroundColor = ConsoleColor.White;
}`

[martindevans]

If you clone this repo and run the example project does it work, or do you get the same error?

[Jafesu]

If you clone this repo and run the example project does it work, or do you get the same error?

Trying now, standby

[Jafesu]

If you clone this repo and run the example project does it work, or do you get the same error?

Could it be because I am running .NET 8.0? Examples are 7.0 (still working on running the example from the project)

[SanftMonster]

Could it be because I am running .NET 8.0? Examples are 7.0 (still working on running the example from the project)

The most possible reason is that your cpu does not support avx2. We've fixed it in #316 but it was after v0.8.0. If you found it's not clear to run the examples, please feel free to tell us to improve them.

[Jafesu]

If you clone this repo and run the example project does it work, or do you get the same error?

The example semi-ran input wasn't working, but not too worries about that, the model loaded up, and i did not get any errors

[SanftMonster]

Ok, please let us know if you cannot run the examples successfully. The support for non-avx2 cpu will be included in the next release.

[Jafesu]

Ok, please let us know if you cannot run the examples successfully. The support for non-avx2 cpu will be included in the next release.

The examples are running, do we have an eta on the next release?

[SanftMonster]

The examples are running, do we have an eta on the next release?

Perhaps this weekend, it's related with a major bug fix I'm working on now. I'd recommend you to work with master branch first because there's little difference between building from source and using nuget packages. :)

Besides, if you do prefer to use nuget package, there's still a quick fix for your problem with v0.8.0:

1. Download a library from here.
2. Add NativeLibraryConfig.Default.WithLibrary({YOUR_PATH}) to the very beginning of your code, in which YOUR_PATH is the path of the library downloaded.

[Jafesu]

If i work off the master branch do i have to compile it myself?

[SanftMonster]

If i work off the master branch do i have to compile it myself?

Please add LLamaSharp as a dependency of your project, then LLamaSharp will be compiled automatically when you compiled your project.

[Jafesu]

one more question, I'm sorry to keep bothering you.

is this correct?

string libPath = "./libllama.dll";
NativeLibraryConfig.Default.WithLibrary(libPath);

I get error:

System.TypeInitializationException
  HResult=0x80131534
  Message=The type initializer for 'LLama.Native.NativeApi' threw an exception.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
RuntimeError: Failed to load the native library [./libllama.dll] you specified.

[Joshhua5]

I get this with the .so version too. I'll try downgrading from dotnet 8

[Joshhua5]

Works with the dotnet-sdk-7.0 package, not with 8

[SanftMonster]

Works with the dotnet-sdk-7.0 package, not with 8

So in dotnet8 native library cannot be loaded while everything goes well in dotnet 7?

[Webslug]

Likewise, I can't get the example projects to load. Either in 7.0 or 8.0 - LlamaSharp used to work great for me in older versions.

[martindevans]

One of the things that has changed from 7->8 is that llamasharp will try to autodetect certain CPU features (AVX512) and use a better binary if it detects them. So it's possible that's failing for you.

To diagnose this:

1. Exactly what CPU model are you using?

2. What is logged if you run this:

NativeLibraryConfig.Default.Logging = true
NativeApi.llama_empty_call();

[DM-98]

One of the things that has changed from 7->8 is that llamasharp will try to autodetect certain CPU features (AVX512) and use a better binary if it detects them. So it's possible that's failing for you.

To diagnose this:

1. Exactly what CPU model are you using?
2. What is logged if you run this:

NativeLibraryConfig.Default.Logging = true
NativeApi.llama_empty_call();

I have exact same issue as the author - I can use LlamaSharp without any issues when I use the LLamaSharp.Backend.Cpu Version 0.8.1 nuget package, but when I instead use LLamaSharp.Backend.Cuda12 nuget package Version 0.8.1, I get the error described in this post (using Llama 2 7b chat: llama-2-7b-chat.Q4_0.gguf).

Now I just added the 2 lines of codes (I have an AMD Ryzen 7 3700X 8-Core Processor):

NativeLibraryConfig.Instance.WithLogs(true);
NativeApi.llama_empty_call();

And I get:

I did try building the libllama.dll and referencing it in the beginning of the code like this:

NativeLibraryConfig.Instance.WithLibrary("G:\\libllama.dll");

And now it looks like this:

I use .NET 8. Cuda version:

EDIT: I managed to get it working by installing CUDA Toolkit:

I then saw a related issue regarding this, which might be preventing this exception after PR: https://github.com/SciSharp/LLamaSharp/issues/350

[martindevans]

Has this been resolved with the release of v0.9.1?

[martindevans]

I'll close this for now, since the issue was hopefully resolved in 0.9.1. Please don't hesitate to re-open if you're still having issues!

[evolcano]

I have the same issue, so I cloned this repo, and debug into LLama.Native.NativeApi.TryLoadLibrary().TryLoad(string path, bool supported = true).

The reason is LLamaSharp what to load runtimes/win-x64/native/avx512/llama.dll:

But no such directory and file exist:

The code is called from WinForm.

CPU: 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz GPU: GPU 0 Intel(R) UHD Graphics GPU 1 NVIDIA RTX A3000 Laptop GPU Driver: NVIDIA-SMI 537.79 Driver Version: 537.79 CUDA Version: 12.2 Backend: LLamaSharp.Backend.Cuda12

After I switch backend to LLamaSharp.Backend.Cpu, "./runtimes/win-x64/native/avx512/llama.dll" exist

[evolcano]

@martindevans Please reopen this issue.

[martindevans]

Checking I understand the issue correctly:

• If you run it with just the CUDA backend, it crashes because it can't load the AVX512 llama.dll.
• If you run it with the CPU backend installed, it runs fine

Is that correct?

[SanftMonster]

It's weird because with cuda it should select untimes/win-x64/native/cuda12/llama.dll instead of untimes/win-x64/native/avx512/llama.dll, maybe a bug there

[SirDaveWolf]

I have the same issue. It works with CPU, but not with GPU CUDA12. I have tested in .Net 6 and .Net 8.

LlamaSharp Version: 0.13.0

GPU: RTX 4090 CPU: i9 13900k

OS: Windows 11

"One of the dependency of the native library is missed. " What are the dependencies of the C++ DLL? Which runtime does it use?

Dumpbin gave this output:

Microsoft (R) COFF/PE Dumper Version 14.39.33523.0 Copyright (C) Microsoft Corporation. All rights reserved.

Dump of file \runtimes\win-x64\native\cuda12\llama.dll

File Type: DLL

Image has the following dependencies:

cudart64_12.dll
cublas64_12.dll
nvcuda.dll
KERNEL32.dll
MSVCP140.dll
VCRUNTIME140.dll
VCRUNTIME140_1.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-filesystem-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-environment-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-utility-l1-1-0.dll

Summary

   8B000 .data
    1000 .nvFatBi
 236C000 .nv_fatb
    D000 .pdata
  161000 .rdata
    1000 .reloc
    1000 .rsrc
  1C9000 .text

EDIT:

It started working after installing the NVIDIA CUDA Toolkit.