BUG Failure in CrescendoOrchestrator

[riedgar-ms]

Describe the bug

Running a Crescendo attack on a locally running model, I had a failure due to the following exception:

  File "C:\Users\riedgar\source\repos\PyRIT\build\__editable__.pyrit-0.4.1.dev0-py3-none-any\pyrit\orchestrator\crescendo_orchestrator.py", line 158, in apply_crescendo_attack_async
    self._prompt_target_conversation_id = await self._backtrack_memory(
  File "C:\Users\riedgar\source\repos\PyRIT\build\__editable__.pyrit-0.4.1.dev0-py3-none-any\pyrit\orchestrator\crescendo_orchestrator.py", line 319, in _backtrack_memory
    new_conversation_id = self._memory.duplicate_conversation_excluding_last_turn(
  File "C:\Users\riedgar\source\repos\PyRIT\build\__editable__.pyrit-0.4.1.dev0-py3-none-any\pyrit\memory\memory_interface.py", line 259, in duplicate_conversation_excluding_last_turn
    piece.orchestrator_identifier["id"] = new_orchestrator_id
TypeError: 'NoneType' object does not support item assignment

Steps/Code to Reproduce

Not easy to provide (and will have to back out my own retry logic to reproduce).

Expected Results

No exception

Actual Results

Stacktrace shown above

Screenshots

N/A

Versions

Win11 Python 3.10 PyRIT installed via pip install -e .[dev] from repository cloned on 2024-09-04

[nina-msft]

Attempting to reproduce now - will get back to you on my progress!

[nina-msft]

Reassigning to @jbolor21 :-)

[riedgar-ms]

Thanks. This isn't blocking right now, mainly because it's not reproducing all of a sudden. I do have some extra logging in my local copy of memory_interface and will share that if I encounter the issue again

[riedgar-ms]

Any updates at all? I'm still seeing this a lot. Based on an AzureML run I did over the weekend, the problem is particularly acute when there is a system prompt present.

[riedgar-ms]

Tag @nina-msft @jbolor21

[romanlutz]

Closing this issue since we've made considerable refactoring changes. If the same or similar happens again please reopen.