WSL doesn't seem to work (gpu or cpu) - always results in "Killed"

[achillez]

Running on: cpu load INSTRUCTOR_Transformer max_seq_length 512 Using embedded DuckDB with persistence: data will be stored in: /home/achillez/devel/localGPT/DB Killed

[PromtEngineer]

I will probably need to add smaller-size embeddings to use by default. What is your Hardware configuration?

[darrinh]

getting the same result:

python ./run_localGPT.py 
Running on: cuda
load INSTRUCTOR_Transformer
max_seq_length  512
Using embedded DuckDB with persistence: data will be stored in: /home/chatbot/localGPT/DB
Loading checkpoint shards:   0%|                                                                        | 0/2 [00:00<?, ?it/s]
Killed

I5/32G RAM/Nvidia Titan 12G

thanks Darrin

[movanet]

I concur, doesn't work with wsl

[joostshao]

same to me

[mindwellsolutions]

I'm getting the same issue here in Ubuntu 22.04.2 . However I installed the same build on Windows 10 (Powershell) and everything works there (Temp solution for those that want to try out localGPT).

Same specs work in Win10, but not Ubuntu: My Specs: i7-10700KF 3.8Ghz, RTX 3080, 48GB DDR 3200 RAM, 1TB NVME HD.

Notes: RTX 3080 is configured properly in Ubuntu, show up in nvidia-smi and the NVidia Server App.

[achillez]

I'm getting the same issue here in Ubuntu 22.04.2 . However I installed the same build on Windows 10 (Powershell) and everything works there (Temp solution for those that want to try out localGPT).

Same specs work in Win10, but not Ubuntu: My Specs: i7-10700KF 3.8Ghz, RTX 3080, 48GB DDR 3200 RAM, 1TB NVME HD.

Notes: RTX 3080 is configured properly in Ubuntu, show up in nvidia-smi and the NVidia Server App.

do you have to install visual studio to compile on windows/powershell?

I'm beginning to wonder if this is a ubuntu specific issue.

[achillez]

Fails on another Ubuntu setup (not WSL). Traced the python script, failing in torhc:

init.py(101): if nonlinearity in linear_fns or nonlinearity == 'sigmoid': init.py(103): elif nonlinearity == 'tanh': init.py(105): elif nonlinearity == 'relu': init.py(107): elif nonlinearity == 'leaky_relu': init.py(108): if param is None: init.py(110): elif not isinstance(param, bool) and isinstance(param, int) or isinstance(param, float): init.py(112): negative_slope = param init.py(115): return math.sqrt(2.0 / (1 + negative_slope ** 2)) init.py(409): std = gain / math.sqrt(fan) init.py(410): bound = math.sqrt(3.0) * std # Calculate uniform bounds from standard deviation init.py(411): with torch.no_grad(): --- modulename: grad_mode, funcname: init grad_mode.py(49): if not torch._jit_internal.is_scripting(): --- modulename: _jit_internal, funcname: is_scripting _jit_internal.py(1121): return False grad_mode.py(50): super().init() grad_mode.py(51): self.prev = False --- modulename: grad_mode, funcname: enter grad_mode.py(54): self.prev = torch.is_grad_enabled() grad_mode.py(55): torch.set_grad_enabled(False) --- modulename: grad_mode, funcname: init grad_mode.py(150): self.prev = torch.is_grad_enabled() grad_mode.py(151): torch._C.set_grad_enabled(mode) grad_mode.py(152): self.mode = mode init.py(412): return tensor.uniform(-bound, bound) Killed

[achillez]

I will probably need to add smaller-size embeddings to use by default. What is your Hardware configuration?

Can we make this change?

[zejefferson]

Same issue here with WSL2 / Ubuntu (launching "python ingest.py" from a Docker container):

Loading documents from /app/SOURCE_DOCUMENTS Loaded 1 documents from /app/SOURCE_DOCUMENTS Split into 72 chunks of text load INSTRUCTOR_Transformer Killed

[PromtEngineer]

The issue is related to the required RAM on the system. I was able to reproduce this. The current implementation uses ~40GB of memory. I have tested a few models but haven't made an unquantized model work yet. I will be exploring some 4-bit quantized models but have trouble making them work in LangChain.

[penthoy]

I have exact same issue, I have 64GB cpu ram, still crashes, 3080 16GB vram, on wsl ubuntu, exact message.

[mindwellsolutions]

I can confirm along with Penthoy. With the same machine I am able to run localGPT successfully in Windows 10 but get this exact error when running it through Ubuntu (Same machine.) For me, on Windows 10, Python.exe uses 32GB Max RAM (During Ingest.py) and 29GB Max RAM (During run_localGPT.py) with lots of headroom with my 48GB DDR4 3200mhz. RTX 3080 (10GB) hits a max of 6-7 GB of GPU usage when running Ingest.py or run_localGPT.py with decent headroom.

If its working on Windows, but not Ubuntu with the same machine, is it likely something other than a lack of hardware resources?

Machine Details: Dual booting Ubuntu 22.04.2 and Windows 10 (i7 10th Gen, RTX 3080, 48GB RAM)

On Ubuntu the proper 3rd Party Nvidia driver is installed and shows up properly in nvidia app. All CUDA requirements are installed and are functioning properly (to my knowledge).

[achillez]

I got it working in WSL2 with a 64GB swap file and 16GB of memory. This was a ubuntu instance.

With that said, this is with the cpu only mode. The gpu mode doesn't work quite yet. For some reason cuda is very finicky in wsl.

[mindwellsolutions]

I got it working in WSL2 with a 64GB swap file and 16GB of memory. This was a ubuntu instance.

With that said, this is with the cpu only mode. The gpu mode doesn't work quite yet. For some reason cuda is very finicky in wsl.

Yes CPU works on my end. It is CUDA GPU mode that has this issue in Ubuntu for me. Windows 10 CUDA GPU mode works though.

[achillez]

I got it working in WSL2 with a 64GB swap file and 16GB of memory. This was a ubuntu instance. With that said, this is with the cpu only mode. The gpu mode doesn't work quite yet. For some reason cuda is very finicky in wsl.

Yes CPU works on my end. It is CUDA GPU mode that has this issue in Ubuntu for me. Windows 10 CUDA GPU mode works though.

Gotcha. CUDA GPU ran but never returned anything. I installed the CUDA toolkit but now anything CUDA related segfaults.