photonvision icon indicating copy to clipboard operation
photonvision copied to clipboard

Use protobuf for setting serialization

Open mcm001 opened this issue 2 years ago • 4 comments

JSON is nice and all, but it can struggle with weird things like serializing arrays of polymorphic types. I do hacky things to get around this in my SQL code. If our settings database was instead one big protobuf, we should be able to reduce the quantity of hacky stuff we do. This has the fun side effect of enabling us to support changing pipeline settings over NT (hello, ros services but worse!)

mcm001 avatar Nov 22 '23 00:11 mcm001

I'm not a big fan of sending full settings over NT and allowing the rio to control those becasue:

  1. It could become a race condition with both the UI and NT manipulating the same settings simultaneously
  2. There is no major use case for the rio needing to manipulate settings greater than driver mode and pipeline index. Everything else would require the dashboard cause you need to actually see the results.

srimanachanta avatar Nov 22 '23 01:11 srimanachanta

Yeah fair. Still think protobufs for internal only settings storage are good, though the value prop is basically just "it's better lol"

mcm001 avatar Nov 22 '23 03:11 mcm001

Yeah fair. Still think protobufs for internal only settings storage are good, though the value prop is basically just "it's better lol"

I agree with that, SerDe with Protobuf beats Jackson for me

srimanachanta avatar Nov 23 '23 17:11 srimanachanta

Making everything one big protobuf might also make https://github.com/PhotonVision/photonvision/issues/1875 an easy choice to make, as several config related classes are also duplicated on the frontend. We could really switch out JSON for protobuf almost everywhere. The primary benefit to using protobuf would be the ability to more strongly type all the data we use and store everywhere, as well as reducing the amount of hacky serialization code (could remove all of JacksonUtils, as well as unnamed raw HashMaps).

One downside to this is we lose the human readability of the SQLite database. I've opened it multiple times to extract chunks of the config out (like calibration) without needing to boot up PhotonVision, and I'd like to have some tooling around getting a human readable format back out (probably JSON again since protobuf can be converted to JSON and back.)

Gold856 avatar Aug 20 '25 02:08 Gold856