`ValueError: Accessing value before it's been set.` at random intervals with `model.generate`

[apoorvkh]

Hi, wanted to share one more issue I'm seeing, in case I'm doing something wrong. Appreciate your help, thanks so much!

I am encountering an error at random intervals in my loop, even given the same inputs (e.g. at iteration 179, 91, 204, ...). Using an A6000 GPU, Python 3.12, nnsight==0.4.3, torch==2.6.0.

My code (given prompts.json) looks like this:

import json
prompts = json.load(open('prompts.json'))

from nnsight import LanguageModel
from tqdm import tqdm

model = LanguageModel('meta-llama/Llama-3.2-3B', device_map="auto", dispatch=True)
completions = []

for prompt in tqdm(prompts):
    new_tokens = []
    with model.generate(prompt, max_new_tokens=10):
        for _ in range(10):
            new_tokens.append(model.lm_head.output.argmax(dim=-1)[0][-1].save())
            _ = model.lm_head.next()
    new_tokens = [model.tokenizer.decode(_t.value) for _t in new_tokens]
    completions.append("".join(new_tokens))

And the error I get is:

Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "venv/lib/python3.12/site-packages/nnsight/tracing/graph/proxy.py", line 64, in value
    return self.node.value
           ^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/nnsight/tracing/graph/node.py", line 143, in value
    raise ValueError("Accessing value before it's been set.")
ValueError: Accessing value before it's been set

[apoorvkh]

It seems that .value raises this error if not Node().done. Is this expected after leaving the model.generate() contextmanager?

[apoorvkh]

Just fyi, also an issue on 0.4.6 :)

[AdamBelfki3]

Hey @apoorvkh, I'm not entirely sure what's breaking here. It may be related to "Frame Injection" of saved results...

To play it safely, I would define the list you're appending to inside the tracing context:

import json
prompts = json.load(open('prompts.json'))

from nnsight import LanguageModel
from tqdm import tqdm
import nnsight

model = LanguageModel('meta-llama/Llama-3.2-3B', device_map="auto", dispatch=True)
completions = []

for prompt in tqdm(prompts):
    with model.generate(prompt, max_new_tokens=10):
        new_tokens = nnsight.list().save()
        for _ in range(10):
            new_tokens.append(model.lm_head.output.argmax(dim=-1)[0][-1])
            _ = model.lm_head.next()
    new_tokens = [model.tokenizer.decode(_t) for _t in new_tokens]
    completions.append("".join(new_tokens))

I couldn't reproduce the error with this updated code. Let me know if this works for you.

[apoorvkh]

Sorry for the slow follow up, but I just tried your proposed change in my experiments at scale and I think it works just fine! Thank you!