Sebastian Raschka

icon indicating a website link https://sebastianraschka.com

icon location AI Research Engineer working on LLMs.

Results 818 comments of Sebastian Raschka

Batched inference on a single node with multiple GPUs

I would even start with single GPU, and then we could think about implementing data or model parallelism later.

Overwrite system prompt on load/generate

Thanks for the suggestion. This is a reasonable extension. Unfortunately, I currently don't have the capacity to work on it, but maybe my colleagues can weigh in here.

How to convert and use huggingface checkpoint in litgpt ?

Hi there, if it is a new model not already supported by LitGPT, the general procedure is outlined here (it's a bit involved): https://github.com/Lightning-AI/litgpt/blob/main/tutorials/developer-docs/adding-models.md However, if it is a model...

Using custom data for `Continue pretraining an LLM`

Good question. Maybe it's too small so it can't generate the validation set. Does the same issue occur if you make the dataset larger, e.g., duplicating the sentence?

Using custom data for `Continue pretraining an LLM`

Thanks, this definitely sounds like an issue then to look into.

Using custom data for `Continue pretraining an LLM`

Thanks for reporting. I totally missed your follow-up @SimiPixel . Based on the error message, it kind of looks like a LitData issue. The LitData version was just upgraded a...

Using custom data for `Continue pretraining an LLM`

Oh I see. Could you try to upgrade (`pip install litdata==0.2.16`) and see if the error still persists then?

Using custom data for `Continue pretraining an LLM`

Arg, I was hoping this would fix it. I currently don't have any good explanation for this and would have to look into it. Thanks for sharing

Using custom data for `Continue pretraining an LLM`

@MaxGonzalezSaez-Diez Is it possible to attach the files here that are causing the problems?

Using custom data for `Continue pretraining an LLM`

Thanks for the super detailed write-up. So I suspect there are two potential issues 1. Short texts that are shorter than the `max_seq_length`. 2. Leftover files from testing > I...