OpenAdapt
OpenAdapt copied to clipboard
Published
20 hours ago •
OpenAdaptAI
OpenAdaptAI
[Bug]: model generated JSON snippet exceeds output context length
Describe the bug
python -m openadapt.replay SegmentReplayStrategy "replay verbatim"
This happens when there are too many segments due to a lack of filtering, e.g.
('```json\n' 14:20:57 [36/95455]
'{\n'
' "descriptions": [\n'
' [1, "Calculator GUI"],\n'
' [2, "Dark grey background area"],\n'
' [3, "Dark grey background area with text"],\n'
' [4, "Dark grey background area"],\n'
' [5, "Dark grey background area with text"],\n'
' [6, "Button with \'7\'"],\n'
' [7, "Round button"],\n'
' [8, "Dark grey background area with text"],\n'
' [9, "Buttons with \'2\' and \'3\'"],\n'
' [10, "Dark grey background area with text"],\n'
' [11, "Dark grey background area with text"],\n'
' [12, "Dark grey background area with a line"],\n'
' [13, "Dark grey background area with a line"],\n'
' [14, "Button with icon or text"],\n'
' [15, "Button with \'8\'"],\n'
' [16, "Button with \'-\'"],\n'
' [17, "Dark grey background area"],\n'
' [18, "Dark grey background area with text"],\n'
' [19, "Dark grey background area"],\n'
' [20, "Dark grey background area"],\n'
' [21, "Button with icon or text"],\n'
' [22, "Round button"],\n'
' [23, "Dark grey background area with text"],\n'
' [24, "Dark grey background area with text"],\n'
' [25, "Dark grey background area with text"],\n'
' [26, "Round button"],\n'
' [27, "Button with text"],\n'
' [28, "Dark grey background area with text"],\n'
' [29, "Button with \'1\'"],\n'
' [30, "Dark grey background area"],\n'
' [31, "Button with text"],\n'
' [32, "Dark grey background area"],\n'
' [33, "Round button"],\n'
' [34, "Dark grey background area with text"],\n'
' [35, "Button with \'C\'"],\n'
' [36, "Dark grey background area"],\n'
' [37, "Dark grey background area with text"],\n'
' [38, "Text or icon on a dark grey background"],\n'
' [39, "Dark grey area with text"],\n'
' [40, "Dark grey background area"],\n'
' [41, "Column of buttons with icons"],\n'
' [42, "Dark grey background area with text"],\n'
' [43, "Text or icon on a dark grey background"],\n'
' [44, "Dark grey background area"],\n'
' [45, "Dark grey background area with text"],\n'
' [46, "Dark grey background area with text"],\n'
' [47, "Button with text or icon"],\n'
' [48, "Button with \'+\'"],\n'
' [49, "Dark grey background area with text"],\n'
' [50, "Dark grey background area with text"],\n'
' [51, "Dark grey background area with text"],\n'
' [52, "Dark grey background area text"],\n'
' [53, "Button with \'7\'"],\n'
' [54, "Dark grey background area"],\n'
' [55, "Dark grey background area with text or icon"],\n'
' [56, "Dark grey background area with icons"],\n'
' [57, "Orange button with \'÷\'"],\n'
' [58, "Text or icon"],\n'
' [59, "Dark grey background area with text or icon"],\n'
' [60, "Dark grey background area with text"],\n'
' [61, "Dark grey background area with icon or text"],\n'
' [62, "Dark grey background area"],\n'
' [63, "Dark grey background area with text"],\n'
' [64, "Column of buttons with icons"],\n'
' [65, "Dark grey background area with icons"],\n'
' [66, "Text or icon on a darker grey background"],\n'
' [67, "Dark grey area with text or icon"],\n'
' [68, "Dark grey background area"],\n'
File "/Users/abrichr/oa/OpenAdapt/openadapt/utils.py", line 601, in parse_code_snippet
rval = ast.literal_eval(json_string)
| | -> '{\n "descriptions": [\n [1, "A plus sign icon on an orange background"],\n [2, "A blank gray square"],\n [3, "The ...
| -> <function literal_eval at 0x104ba2ef0>
-> <module 'ast' from '/opt/homebrew/Cellar/[email protected]/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py'>
File "/opt/homebrew/Cellar/[email protected]/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py", line 64, in literal_eval
node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
| | -> <method 'lstrip' of 'str' objects>
| -> '{\n "descriptions": [\n [1, "A plus sign icon on an orange background"],\n [2, "A blank gray square"],\n [3, "The ...
-> <function parse at 0x104ba2cb0>
File "/opt/homebrew/Cellar/[email protected]/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py", line 50, in parse
return compile(source, filename, mode, flags,
| | | -> 1024
| | -> 'eval'
| -> '<unknown>'
-> '{\n "descriptions": [\n [1, "A plus sign icon on an orange background"],\n [2, "A blank gray square"],\n [3, "The ...
File "<unknown>", line 19
}
^
SyntaxError: closing parenthesis '}' does not match opening parenthesis '[' on line 2
To Reproduce
python -m openadapt.replay SegmentReplayStrategy "replay verbatim"
https://chatgpt.com/share/e92a6524-bb9a-4692-ac2e-9e10fe52aeff
For now we will focus on reducing the number of segments per prompt in upstream logic.
A regression was introduced in https://github.com/OpenAdaptAI/OpenAdapt/pull/679 in openadapt/adapters/ultralytics.py.
Before:
conf: float = 0.4,
iou: float = 0.9,
After:
# threshold below which boxes will be filtered out
conf: float = 0,
# discards all overlapping boxes with IoU > iou_threshold
iou: float = 0.05,
The regression has been fixed in https://github.com/OpenAdaptAI/OpenAdapt/pull/758 but the issue will crop up again for lower NMS threshold values.