archivematica icon indicating copy to clipboard operation
archivematica copied to clipboard

Use JSON as the wire format in Gearman jobs

Open sevein opened this issue 10 months ago • 0 comments

This pull request replaces all uses of pickle with json. The use of pickle was introduced back in 2011 when MCP was integrated with Gearman due to its ability to round-trip Python objects, at the cost of lower security. However, apart from datetime.datetime and uuid.UUID, we are not encoding any complex structures that cannot be easily represented in JSON.

Please notice that the use of orjson was an afterthought and that's why it was submitted as an extra commit. orjson is significantly faster compared to the standard library json module, being faster than pickle protocol=0 that we were using previously. It also supports native serialization of datetime and UUID objects.

Fixes https://github.com/archivematica/Issues/issues/1673.

sevein avatar Apr 23 '24 03:04 sevein