LLMs-from-scratch

LLMs-from-scratch copied to clipboard

fix issue #664 - inverted token and pos emb layers

2

fixes #664

casinca

### Bug description Issue reported here: https://livebook.manning.com/forum?product=raschka&comment=578419 This is the affected code: https://github.com/rasbt/LLMs-from-scratch/blob/78bbcb364305f7f59e0954cbe3d5e24fd36ef249/ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb#L1215-L1233 This uses old en-/decoding variants for pieces (tokens) and token IDs. Newer (and more intuitive) would be:...

d-kleine

Bugs with def classify_review

1

### Bug description In page 201, ```python def classify_review( text, model, tokenizer, device, max_length=None, pad_token_id=50256): model.eval() input_ids = tokenizer.encode(text) supported_context_length = model.pos_emb.weight.shape[1] input_ids = input_ids[:min( max_length, supported_context_length )] input_ids +=...

Jessen-Li

### Bug description pos_embedding_layer and token_embedding_layer definition in exercise_solutions chapter 2 incorrect ### What operating system are you using? Linux ### Where do you run your code? Local (laptop, desktop)...

BaixiangZ

Typo / text length

2

### Bug description https://github.com/rasbt/LLMs-from-scratch/blob/3f93d73d6dd75aeff21158506afff9500f79c359/ch05/01_main-chapter-code/ch05.ipynb#L816 I assume "words" should be "characters". Also https://github.com/rasbt/LLMs-from-scratch/blob/3f93d73d6dd75aeff21158506afff9500f79c359/ch05/01_main-chapter-code/ch05.ipynb#L841-L842 and https://github.com/rasbt/LLMs-from-scratch/blob/3f93d73d6dd75aeff21158506afff9500f79c359/ch05/01_main-chapter-code/ch05.ipynb#L867-L868 both print 99 characters, not 100 as explained above ### What operating system are you using?...

d-kleine

Docker file issue

1

While building the docker image the Add command is failing change that part to this. RUN curl -fsSL https://astral.sh/uv/install.sh -o /uv-installer.sh && \ chmod +x /uv-installer.sh && \ /uv-installer.sh &&...

JibinES

Fix: Typo in `appendix_d.py` comments.

1

In `appendix_d.py` the phrase "which lead to" should be "which leads to". Applied fixes: --- a/pkg/llms_from_scratch/appendix_d.py +++ b/pkg/llms_from_scratch/appendix_d.py @@ -76,7 +76,7 @@ if global_step > warmup_steps: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) else: -...

PrinceSajjadHussain

Issue 1: Typo in function docstring

7

Body: In the `get_requirements_dict` function, the docstring has a typo. The word "packaging.requirements.Requirement" should be "packaging.requirements.Requirement".

PrinceSajjadHussain

Body: In `appendix_d.py`, there's a typo in the comment: ```python if global_step >= warmup_steps: # the book originally used global_step > warmup_steps, which lead to a skipped clipping step after...

PrinceSajjadHussain

The functions `find_highest_gradient` and `train_model` in `appendix_d.py` are missing docstrings. Adding docstrings would improve readability and explain the purpose, arguments, and return values of these functions.

PrinceSajjadHussain