OpenAdapt
OpenAdapt copied to clipboard
Published
20 hours ago •
OpenAdaptAI
OpenAdaptAI
[Bug]: VanillaReplayStategy returns KeyError
Describe the bug
VanillaReplayStategy returns KeyError I have pulled the latest upstream and tested it. here's the log:
(openadapt-py3.10) C:\Users\nsaip\Desktop\algora\OpenAdapt>python -m openadapt.replay VanillaReplayStrategy --instructions "calculate 9-8"
[2024-07-12 13:23:34.345] [recordLogger] [info] Created d3d11 device from 0
[2024-07-12 13:23:34.345] [recordLogger] [info] found 1 displays
C:\Users\nsaip\AppData\Local\pypoetry\Cache\virtualenvs\openadapt-28n1JpsV-py3.10\lib\site-packages\pywinauto\_init_.py:80: UserWarning: Revert to STA COM threading mode
warnings.warn("Revert to STA COM threading mode", UserWarning)
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:409 - Reading from C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\data\config.json
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - ROOT_DIR_PATH=C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - PRIVATE_AI_API_KEY=********Y>
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - REPLICATE_API_TOKEN=*******N>
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - DEFAULT_SEGMENTATION_ADAPTER=ultralytics
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - OPENAI_API_KEY=*****>
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - ANTHROPIC_API_KEY=******>
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - GOOGLE_API_KEY=*****>
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - CACHE_DIR_PATH=.cache
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - CACHE_ENABLED=True
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - CACHE_VERBOSITY=0
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - DB_ECHO=False
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - ERROR_REPORTING_ENABLED=True
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - OPENAI_MODEL_NAME=gpt-3.5-turbo
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - RECORD_WINDOW_DATA=False
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - RECORD_READ_ACTIVE_ELEMENT_STATE=False
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - RECORD_VIDEO=True
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - RECORD_AUDIO=True
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - RECORD_FULL_VIDEO=False
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - RECORD_IMAGES=False
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - LOG_MEMORY=False
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - REPLAY_STRIP_ELEMENT_STATE=True
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - VIDEO_ENCODING=libx264
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - VIDEO_PIXEL_FORMAT=yuv444p
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - VIDEO_DIR_PATH=C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\data\videos
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - STOP_SEQUENCES=[['o', 'a', '.', 's', 't', 'o', 'p'], ['ctrl', 'ctrl', 'ctrl']]
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - IGNORE_WARNINGS=False
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - MAX_NUM_WARNINGS_PER_SECOND=5
2024-07-12 13:23:35.690 | INFO | openadapt.config:print_config:413 - WARNING_SUPPRESSION_PERIOD=1
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - MESSAGES_TO_FILTER=['Cannot pickle Objective-C objects']
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - ACTION_TEXT_SEP=-
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - ACTION_TEXT_NAME_PREFIX=<
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - ACTION_TEXT_NAME_SUFFIX=>
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - PLOT_PERFORMANCE=True
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - APP_DARK_MODE=False
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SCRUB_ENABLED=False
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SCRUB_CHAR=*
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SCRUB_LANGUAGE=en
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SCRUB_FILL_COLOR=255
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SCRUB_KEYS_HTML=['text', 'canonical_text', 'title', 'state', 'task_description', 'key_char', 'canonical_key_char', 'key_vk', 'children']
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SCRUB_CONFIG_TRF={'nlp_engine_name': 'spacy', 'models': [{'lang_code': 'en', 'model_name': 'en_core_web_trf'}]}
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SCRUB_PRESIDIO_IGNORE_ENTITIES=[]
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - VISUALIZE_DARK_MODE=False
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - VISUALIZE_RUN_NATIVELY=True
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - VISUALIZE_DENSE_TREES=True
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - VISUALIZE_ANIMATIONS=True
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - VISUALIZE_EXPAND_ALL=False
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - VISUALIZE_MAX_TABLE_CHILDREN=10
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SAVE_SCREENSHOT_DIFF=False
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SPACY_MODEL_NAME=en_core_web_trf
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - DASHBOARD_CLIENT_PORT=5173
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - DASHBOARD_SERVER_PORT=8080
2024-07-12 13:23:35.706 | INFO | openadapt.config:print_config:413 - SOM_SERVER_URL=<SOM_SERVER_URL>
2024-07-12 13:23:35.721 | INFO | openadapt.config:print_config:413 - UNIQUE_USER_ID=b802d869-3605-490c-a3f6-019edb336876
2024-07-12 13:23:35.721 | INFO | openadapt.config:print_config:413 - REDIRECT_TO_ONBOARDING=True
2024-07-12 13:23:35.721 | INFO | openadapt.config:print_config:413 - DEFAULT_ADAPTER=openai
2024-07-12 13:23:35.721 | INFO | openadapt.config:print_config:420 - active_branch_name='main'
2024-07-12 13:23:35.721 | INFO | openadapt.config:print_config:424 - is_reporting_branch=True
2024-07-12 13:23:37.504 | INFO | _main_:replay:69 - strategy_name='VanillaReplayStrategy'
2024-07-12 13:23:48.097 | INFO | _main_:replay:82 - strategy_class=<class 'openadapt.strategies.vanilla.VanillaReplayStrategy'>
2024-07-12 13:23:49.301 | INFO | openadapt.events:get_events:74 - num_process_iters=0 num_action_events=504 num_window_events=6 num_screenshots=53
2024-07-12 13:23:49.301 | INFO | openadapt.events:process_events:825 - before num_action_events=504 num_window_events=6 num_screenshots=53 num_total=563
2024-07-12 13:23:49.317 | INFO | openadapt.events:merge_consecutive_action_events:757 - name='redundant_mouse_move' num_events_removed=93
2024-07-12 13:23:49.458 | INFO | openadapt.events:merge_consecutive_action_events:757 - name='keyboard' num_events_removed=0
2024-07-12 13:23:49.473 | INFO | openadapt.events:merge_consecutive_action_events:757 - name='mouse_move' num_events_removed=387
2024-07-12 13:23:49.473 | INFO | openadapt.events:merge_consecutive_action_events:757 - name='mouse_scroll' num_events_removed=0
2024-07-12 13:23:49.473 | INFO | openadapt.events:merge_consecutive_action_events:757 - name='mouse_click' num_events_removed=7
2024-07-12 13:23:49.473 | INFO | openadapt.events:process_events:873 - after num_action_events_=17 num_window_events_=5 num_screenshots_=17 num_total_=39
2024-07-12 13:23:49.473 | INFO | openadapt.events:process_events:877 - pct_action_events=0.03373015873015873 pct_window_events=0.8333333333333334 pct_screenshots=0.32075471698113206 pct_total=0.06927175843694494
2024-07-12 13:23:49.473 | INFO | openadapt.events:get_events:127 - duration=1.3763306140899658
2024-07-12 13:23:49.489 | ERROR | fire.core:_CallAndUpdateTrace:681 - An error has been caught in function '_CallAndUpdateTrace', process 'MainProcess' (16932), thread 'MainThread' (20464):
Traceback (most recent call last):
File "C:\Users\nsaip\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
| | -> {'_name': 'main', 'doc_': 'Replay recorded events.\n\nUsage:\n python -m openadapt.replay.py <strategy_name> [--...
| -> <code object <module> at 0x0000023D52F2C240, file "C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\replay.py", line 1>
-> <function _run_code at 0x0000023D52F0A3B0>
File "C:\Users\nsaip\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
| -> {'_name': 'main', 'doc_': 'Replay recorded events.\n\nUsage:\n python -m openadapt.replay.py <strategy_name> [--...
-> <code object <module> at 0x0000023D52F2C240, file "C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\replay.py", line 1>
File "C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\replay.py", line 126, in <module>
fire.Fire(replay)
| | -> <function replay at 0x0000023D6CE44280>
| -> <function Fire at 0x0000023D552CEDD0>
-> <module 'fire' from 'C:\\Users\\nsaip\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\openadapt-28n1JpsV-py3.10\\lib\\site-pac...
File "C:\Users\nsaip\AppData\Local\pypoetry\Cache\virtualenvs\openadapt-28n1JpsV-py3.10\lib\site-packages\fire\core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
| | | | | -> 'replay.py'
| | | | -> {}
| | | -> Namespace(verbose=False, interactive=False, separator='-', completion=None, help=False, trace=False)
| | -> ['VanillaReplayStrategy']
| -> <function replay at 0x0000023D6CE44280>
-> <function _Fire at 0x0000023D553AA5F0>
File "C:\Users\nsaip\AppData\Local\pypoetry\Cache\virtualenvs\openadapt-28n1JpsV-py3.10\lib\site-packages\fire\core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
| -> <function _CallAndUpdateTrace at 0x0000023D553AA710>
-> <function replay at 0x0000023D6CE44280>
> File "C:\Users\nsaip\AppData\Local\pypoetry\Cache\virtualenvs\openadapt-28n1JpsV-py3.10\lib\site-packages\fire\core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
| | -> {}
| -> ['VanillaReplayStrategy', False, None, None, None]
-> <function replay at 0x0000023D6CE44280>
File "C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\replay.py", line 84, in replay
strategy = strategy_class(recording, **kwargs)
| | -> {}
| -> Recording(id=2, timestamp=1720770391.9934068, monitor_width=1920, monitor_height=1080, double_click_interval_seconds=0.5, dou...
-> <class 'openadapt.strategies.vanilla.VanillaReplayStrategy'>
File "C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\strategies\vanilla.py", line 56, in _init_
self.recording_description = describe_recording(
| -> <function describe_recording at 0x0000023D13271A20>
-> <openadapt.strategies.vanilla.VanillaReplayStrategy object at 0x0000023D6F507310>
File "C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\strategies\vanilla.py", line 131, in describe_recording
window_dicts = [
File "C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\strategies\vanilla.py", line 132, in <listcomp>
action.window_event.to_prompt_dict(include_window_data)
| | -> False
| -> <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x0000023D6CCD34C0>
-> ActionEvent(name='singleclick', timestamp=1720770437.7697651, recording_timestamp=1720770391.9934068, screenshot_timestamp=17...
File "C:\Users\nsaip\Desktop\algora\OpenAdapt\openadapt\models.py", line 576, in to_prompt_dict
window_dict["state"].pop("data")
-> {'title': 'Search', 'left': 684, 'top': 282, 'width': 800, 'height': 798}
KeyError: 'state'
To Reproduce
[ ] Windows
python -m openadapt.replay VanillaReplayStrategy --instructions "calculate 9-8"
