metamap sometimes gets corrupted?

[edeutsch]

I think this also happened before on our own instance, but this morning Sarah reported that ARAX prod was not working. I see queries on there crashing with:

2023-02-06T21:05:43.221269 ERROR: An uncaught error occurred: 'utf-8' codec can't decode byte 0x95 in position 23: invalid start byte: ['Traceback (most recent call last):\n', ' File "/mnt/data/orangeboard/production/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/controllers/../../../../../ARAX/ARAXQuery/ARAX_query.py", line 735, in execute_processing_plan\n expander.apply(response, action[\'parameters\'], mode=mode)\n', ' File "/mnt/data/orangeboard/production/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/controllers/../../../../../ARAX/ARAXQuery/ARAX_expander.py", line 183, in apply\n kp_selector = KPSelector(valid_kps_override={"infores:rtx-kg2"}, log=log) if mode == "RTXKG2" else KPSelector(log=log)\n', ' File "/mnt/data/orangeboard/production/RTX/code/ARAX/ARAXQuery/Expand/kp_selector.py", line 36, in __init__\n self.meta_map = self._load_meta_map()\n', ' File "/mnt/data/orangeboard/production/RTX/code/ARAX/ARAXQuery/Expand/kp_selector.py", line 223, in _load_meta_map\n meta_map = self._refresh_meta_map()\n', ' File "/mnt/data/orangeboard/production/RTX/code/ARAX/ARAXQuery/Expand/kp_selector.py", line 254, in _refresh_meta_map\n meta_map = pickle.load(existing_meta_map_file)\n', "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 23: invalid start byte\n"]

I have a vague recollection of @amykglen once commenting about meta map cache corruption? It seems to be an issue. Maybe at minimum a try except block around this that triggers a rebuild? Or perhaps eventually a thread-safe version if we suspect that different threads are stomping on each other in the meta map cache?

[amykglen]

huh, yeah, that's not good. maybe we can address this as part of #1952 (@sundareswarpullela)