feat: WIP: Adjust GPU Layers

[siddimore]

Description

1. Add GGUF Parser

TODO

1. Install nvidia-smi driver
2. Fetch GPU Device information
3. Offload layers to GPU baed on GGUF Parser metadata

This PR fixes #3541

Notes for Reviewers

Signed commits

• [ ] Yes, I signed my commits.

[netlify[bot]]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	cd1dc5dac4591bdf61dad25443f7f9bde40a2e4f
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6705d246290228000812ee36
😎 Deploy Preview	https://deploy-preview-3737--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[siddimore]

@mudler can u kindly check the PR approach and give some high level feedback when possible?

Next step that i will add is some sort of GPU_Layer estimator based on:

1. VRAM from GGUF Parsing
2. Noof GPU's on the device

[mudler]

@siddimore thanks for taking a stab at this, direction looks good here - just few minor nits here and there but definitely not blockers

[siddimore]

@siddimore thanks for taking a stab at this, direction looks good here - just few minor nits here and there but definitely not blockers

thanks much @mudler you are welcome!! i will improve the code and add some more testing. Appreciate the feedback and will fix the comments

[github-actions[bot]]

This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This PR was closed because it has been stalled for 10 days with no activity.