commaai
switch to new json library
Describe the bug
I updated logmessaged.py as below to see why I keep receiving an error message when I execute manager.py.
if level >= log_level:
print("record:::",record)
log_handler.emit(record)
Below is the result, the json message that is sent by logmessaged.py is not formatted correctly and https://github.com/commaai/openpilot/blob/master/common/logging_extra.py#L100 cannot parse it. (this part is not valid: {"created": 1652444715,3936491, ...)
((openpilot) ) hamid@hamid:~/openpilot/selfdrive/manager$ PASSIVE=0 NOSENSOR=1 USE_WEBCAM=1 ./manager.py
selfdrive/loggerd/bootlog.cc: bootlog to /home/hamid/.comma/media/0/realdata/boot/2022-05-13--14-25-13
missing public key: /home/hamid/.comma/persist/comma/id_rsa.pub
selfdrive/logmessaged.py: needs update
record::: {"created": 1652444715,3936491, "ctx": {"daemon": "soundd", "device": "pc", "dirty": true, "dongle_id": "UnregisteredDevice", "version": "0.8.14"}, "filename": "selfdrive/ui/soundd/sound.cc", "funcname": "Sound::Sound(QObject *)", "levelnum": 20, "lineno": 16, "msg": "default audio device: \"\""}
--- Logging error ---
Traceback (most recent call last):
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/logging/__init__.py", line 1085, in emit
msg = self.format(record)
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/logging/__init__.py", line 929, in format
return fmt.format(record)
File "/home/hamid/openpilot/common/logging_extra.py", line 100, in format
v = json.loads(record)
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 24 (char 23)
Call stack:
File "./manager.py", line 205, in <module>
main()
File "./manager.py", line 182, in main
manager_thread()
File "./manager.py", line 134, in manager_thread
ensure_running(managed_processes.values(), False, params=params, CP=sm['carParams'], not_run=ignore)
File "/home/hamid/openpilot/selfdrive/manager/process.py", line 313, in ensure_running
p.start()
File "/home/hamid/openpilot/selfdrive/manager/process.py", line 245, in start
self.proc.start()
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/hamid/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/hamid/openpilot/selfdrive/manager/process.py", line 43, in launcher
getattr(mod, 'main')()
File "/home/hamid/openpilot/selfdrive/logmessaged.py", line 29, in main
log_handler.emit(record)
Unable to print the message and arguments - possible formatting error.
Use the traceback above to help find the error.
record::: {"msg": "System time valid", "ctx": {}, "level": "INFO", "levelnum": 20, "name": "swaglog", "filename": "manager.py", "lineno": 31, "pathname": "./manager.py", "module": "manager", "funcName": "manager_init", "host": "hamid", "process": 11907, "thread": 140079618852672, "threadName": "MainThread", "created": 1652444713.9832933}
record::: {"msg": "missing public key: /home/hamid/.comma/persist/comma/id_rsa.pub", "ctx": {}, "level": "WARNING", "levelnum": 30, "name": "swaglog", "filename": "manager.py", "lineno": 83, "pathname": "./manager.py", "module": "manager", "funcName": "manager_init", "host": "hamid", "process": 11907, "thread": 140079618852672, "threadName": "MainThread", "created": 1652444714.1361263}
record::: {"msg": "starting process ui", "ctx": {"dongle_id": "UnregisteredDevice", "version": "0.8.14", "dirty": true, "device": "pc"}, "level": "INFO", "levelnum": 20, "name": "swaglog", "filename": "manager.py", "lineno": 171, "pathname": "./manager.py", "module": "manager", "funcName": "main", "host": "hamid", "process": 11907, "thread": 140079618852672, "threadName": "MainThread", "created": 1652444714.969782}
record::: {"msg": "preimporting selfdrive.locationd.calibrationd", "ctx": {"dongle_id": "UnregisteredDevice", "version": "0.8.14", "dirty": true, "device": "pc"}, "level": "INFO", "levelnum": 20, "name": "swaglog", "filename": "manager.py", "lineno": 102, "pathname": "./manager.py", "module": "manager", "funcName": "manager_prepare", "host": "hamid", "process": 11907, "thread": 140079618852672, "threadName": "MainThread", "created": 1652444714.9746444}
p.s: the manger seems to continue working without any issue.
OS Version
20.04
openpilot version or commit
No response
Additional info
No response
The create time in the cloudlog json is a double, which is converted to a string using snprintf here: https://github.com/commaai/openpilot/blob/master/third_party/json11/json11.cpp#L60
The problem is that this probably takes into account your current locale, producing output with a comma as a decimal separator. Which is not valid json.
a workaround would be to switch your system locale to something that uses a . as decimal separator.
relevant json11 issue: https://github.com/dropbox/json11/issues/38
@pd0wm thanks, I confirm that the issue is correctly identified and your workaround fixed the issue. Feel free to close the issue if the issue is too minor to be fixed. Thanks again.
Looks like json11 is no longer maintained. rapidjson seems like a good and mature alternative that may also give us a small speed improvement too.
Bounty is to fix this as simply as possible, and it seems like we'll need to switch libraries to do it (confirm this though). If we absolutely have to switch, https://github.com/nlohmann/json seems like the best options.