Llava DLL issue in Unity

[Titibo26]

Description

Hello, I imported LLava_shared.DLL as it is not shipped with nuggets package. methods from Llava are recognized in my project but when i execute the script, it gives me the following issue :

DllNotFoundException: llava_shared assembly:<unknown assembly> type:<unknown type> member:(null) LLama.Native.SafeLlavaModelHandle.LoadFromFile (System.String modelPath, System.Int32 verbosity) (at <6722d4d228b04bddaef49d0d909117a2>:0) LLama.LLavaWeights.LoadFromFile (System.String mmProject) (at <6722d4d228b04bddaef49d0d909117a2>:0)

here is my code :

 async UniTaskVoid Start()
    {
        _cts = new CancellationTokenSource();
        SetInteractable(false);
        Submit.onClick.AddListener(() =>
        {
            _submittedText = Input.text;
            Input.text = "";
        });
        Output.text = "User: ";
        // Load a model
        var parameters = new ModelParams(Application.streamingAssetsPath + "/" + ModelPath)
        {
            ContextSize = 4096,
            Seed = 1337,
            GpuLayerCount = 35
        };

      

        // Switch to the thread pool for long-running operations
        await UniTask.SwitchToThreadPool();
        using var model = LLamaWeights.LoadFromFile(parameters);
        using var clipModel = LLavaWeights.LoadFromFile(Application.streamingAssetsPath + "/" + ModelPathmmproj);
        await UniTask.SwitchToMainThread();
        // Initialize a chat session
        using var context = model.CreateContext(parameters);
        var ex = new InteractiveExecutor(context, clipModel);
        Texture2D texture = Resources.Load<Texture2D>("tablImg");
        byte[] img = texture.EncodeToPNG();
        

        ex.Images.Add(img);

        // Save the empty state for cases when we need to switch to empty session
        _emptyState = ex.GetStateData();
        foreach(var option in SessionSelector.options)
        {
            var session = new ChatSession(ex);
            // This won't process the system prompt until the first user message is received
            // to pre-process it you'd need to look into context.Decode() method.
            // Create an issue on github if you need help with that.
            session.AddSystemMessage(SystemPrompt);
            _chatSessions.Add(session);
            _executorStates.Add(null);
        }
        SessionSelector.onValueChanged.AddListener(SwitchSession);
        _activeSession = 0;
        // run the inference in a loop to chat with LLM
        await ChatRoutine(_cts.Token);
        Submit.onClick.RemoveAllListeners();
    }

Have you any direction on how to properly add Llava to a unity project ? (Llama is working)

Thanks,

TD

[SignalRT]

Could you specify the nuget packages that you are using, including versions?

Llava should be included in the nuget packages.

[Titibo26]

Hi, I tried with the following configurations of nuget packages :

• LLamaSharp 0.12.0 + LLamaSharp.Backend.Cuda11 0.12.0
• LLamaSharp 0.12.0 +all backend with version 0.12.0

I also tried from the unity demo which comes with : LLamaSharp 0.10.0 Tried this solution + all backend in version 0.10.0.

(In a desesperate moment i also tried the nuggetPackages "LlamaCpp" version 24.5.49215) In all those test, Llava_shared was never here. Llama DLL is here and it is working.

I manually downloaded Llava_shared from this post : https://github.com/SciSharp/LLamaSharp/issues/686#issuecomment-2074150467 but it did not work.

Thanks,

[ltmx]

Hi, I tried with the following configurations of nuget packages :

• LLamaSharp 0.12.0 + LLamaSharp.Backend.Cuda11 0.12.0
• LLamaSharp 0.12.0 +all backend with version 0.12.0

I also tried from the unity demo which comes with : LLamaSharp 0.10.0 Tried this solution + all backend in version 0.10.0.

(In a desesperate moment i also tried the nuggetPackages "LlamaCpp" version 24.5.49215) In all those test, Llava_shared was never here. Llama DLL is here and it is working.

I manually downloaded Llava_shared from this post : #686 (comment) but it did not work.

Thanks,

This issue is not a LLamaSharp Issue, it is actually a NugetForUnity issue. The issue is not fixed on their side. This issue is valid for any native library trying to be imported via NugetForUnity. Issue here : https://github.com/GlitchEnzo/NuGetForUnity/issues/419

Workaround :

• Update LLamaSharp to 0.16.0 (or later) using Nuget4Unity
• Manually download the package via this source for cuda 12 in this example (download link on the right of the page
• then extract the nuget package files using 7zip
• Find llama.dll, llava.dll and ggml.dll
• place dlls in "Assets/" folder
• Make sure you have installed CUDA on your computer, and have registered the CUDA as a PATH variable (the installer should do it automatically)
• Then you need to make sure Your CUDA backend has the same version as your LLamaSharp version: https://www.nuget.org/packages/LLamaSharp.Backend.Cuda12/0.16.0 (current latest version is v 0.16.0)
• If it is still not working, update your NVIDIA GPU drivers

[github-actions[bot]]

This issue has been automatically marked as stale due to inactivity. If no further activity occurs, it will be closed in 7 days.