Intermittent Key error when using Jupyter notebook

[ZeeMuFaSaa]

First of all, love this package..., first time using it.

When running the following code in a notebook cell in your example 'react.ipynb', intermittently getting a Key error. Running notebook in VSCode. If I re-run the cell it will work. Have tried different models as well. Don't let the 'mistral7' var name throw you, just changing model. The tracing functionality seems a little buggy, sometime there is useful output, other times just blank with no info. Here is the code.

lm = mistral7
lm += react_prompt_example("What is the logarithm of Leonardo DiCaprio's age?", tools)

Error report:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[8], line 2
      1 lm = mistral7
----> 2 lm += react_prompt_example("What is the logarithm of Leonardo DiCaprio's age?", tools)

File ~/.pyenv/versions/3.11.10/envs/_ai/lib/python3.11/site-packages/guidance/models/_model.py:1061, in Model.__add__(self, value)
   1051 """Adding is the primary mechanism for extending model state.
   1052 
   1053 Parameters
   (...)
   1056     The grammar used to extend the current model.
   1057 """
   1059 # create the new lm object we will return
   1060 # (we need to do this since Model objects are immutable)
-> 1061 lm = self.copy()
   1063 # find blocks that are now active, but haven't been opened by lm yet
   1064 enter_blocks = []

File ~/.pyenv/versions/3.11.10/envs/_ai/lib/python3.11/site-packages/guidance/models/_model.py:945, in Model.copy(self)
    943 new_lm._parent_id = self._id
    944 new_lm._trace_nodes = set()
--> 945 new_lm._update_trace_node(new_lm._id, new_lm._parent_id, None)
    946 self.engine.model_dict[new_lm._id] = new_lm
    947 new_lm.vis_chunk = None

File ~/.pyenv/versions/3.11.10/envs/_ai/lib/python3.11/site-packages/guidance/models/_model.py:1036, in Model._update_trace_node(self, identifier, parent_id, node_attr)
   1033 self._trace_nodes.add(trace_node)
   1035 if self.renderer is not None:
-> 1036     self.renderer.update(
   1037         TraceMessage(
   1038             trace_id=identifier,
   1039             parent_trace_id=parent_id,
   1040             node_attr=node_attr,
   1041         )
   1042     )

File ~/.pyenv/versions/3.11.10/envs/_ai/lib/python3.11/site-packages/guidance/visual/_renderer.py:489, in AutoRenderer.update(self, message)
    488 def update(self, message: GuidanceMessage) -> None:
--> 489     self._renderer.update(message)

File ~/.pyenv/versions/3.11.10/envs/_ai/lib/python3.11/site-packages/guidance/visual/_renderer.py:360, in JupyterWidgetRenderer.update(self, message)
    357     call_soon_threadsafe(self._recv_queue.put_nowait, serialize_message(started_msg))
    359 # Check if message has diverged from prev messages
--> 360 diverged, shared_ancestor_idx = self.has_divergence(message)
    361 if diverged:
    362     out_messages.append(ResetDisplayMessage())

File ~/.pyenv/versions/3.11.10/envs/_ai/lib/python3.11/site-packages/guidance/visual/_renderer.py:302, in JupyterWidgetRenderer.has_divergence(self, message)
    300 else:
    301     last_trace_message = prev_trace_messages[-1]
--> 302     last_trace_node = self._trace_handler[last_trace_message.trace_id]
    304     if last_trace_node not in message_trace_node.path():
    305         logger.debug(f"DIVERGENCE:curr:{message_trace_node}")

File ~/.pyenv/versions/3.11.10/envs/_ai/lib/python3.11/site-packages/guidance/trace/_trace.py:286, in TraceHandler.__getitem__(self, item)
    285 def __getitem__(self, item):
--> 286     return self.id_node_map[item]

File ~/.pyenv/versions/3.11.10/lib/python3.11/weakref.py:136, in WeakValueDictionary.__getitem__(self, key)
    134 if self._pending_removals:
    135     self._commit_removals()
--> 136 o = self.data[key]()
    137 if o is None:
    138     raise KeyError(key)

KeyError: 229

To Reproduce The notebook is your 'react.ipynb'. This is the last cell that make a call to the model. Here is the model loading, although I have tried other models. I am using the 'Phi-4-Q5' model.

import math

from huggingface_hub import hf_hub_download

import guidance
from guidance import models, gen, select

repo_id = "NikolayKozloff/phi-4-Q5_K_S-GGUF"
filename = "phi-4-q5_k_s.gguf"
model_kwargs = {"verbose": True, "n_gpu_layers": -1, "n_ctx": 4*1024}

downloaded_file = hf_hub_download(repo_id=repo_id, filename=filename)
mistral7 = guidance.models.LlamaCpp(downloaded_file, **model_kwargs)

System info:

• OS: Distributor ID: Ubuntu Description: Ubuntu 22.04.5 LTS Release: 22.04 Codename: jammy

• Guidance Version: 0.2.0

• VSCode version: 1.96.2

• Python version: Python 3.11.10 (main, Jan 6 2025, 10:15:28) [GCC 11.4.0] on llinux

[nopdive]

Hi @ZeeMuFaSaa, glad you like the package!

Let me take a look tomorrow, repro and get back to you. Looks like a memory bug on our side.

[nopdive]

I can confirm this is a bug with repro on our end. I'll try to get a fix in this week.