Inconsistent Model Initialization and Execution Failures in Dockerized Environment on Ubuntu

[Mukku27]

Environment

• OS: Ubuntu 22.04 LTS
• Hardware: Nvidia RTX 4070
• Software: Docker Desktop, Ollama (local installation)
• LLM Models Tested: Gemma2, Nomic Embedding, Custom fine-tuned models

Reproduction Steps

1. Set up an Ubuntu system with an Nvidia GPU and install Docker Desktop.
2. Install Ollama locally and download various LLM models, including Gemma2 and the Nomic Embedding model.
3. Attempt to run the models within a Docker container, using different models for chat and utility tasks.
4. Observe the behavior of the model initialization and execution.

Expected Behavior

• The LLM models should consistently initialize within the Docker container, utilizing the GPU for processing.
• The models should execute correctly without looping errors, providing accurate and relevant responses based on the input prompts.

Actual Behavior

• The LLM models fail to consistently initialize within the Docker container, sometimes bypassing GPU utilization.
• Execution is highly inconsistent, with the models often entering an infinite loop, providing incorrect or error-laden responses.
• The system occasionally initiates the correct model or tool, but this occurs infrequently (approximately 1 out of 40 prompts).