OpenWPM icon indicating copy to clipboard operation
OpenWPM copied to clipboard

Consider using HTTP/WebSockets for client/platform communication

Open motin opened this issue 6 years ago • 1 comments

In #221, this was briefly attempted but ultimately reverted in favor of a (raw) sockets-based approach.

Using a more standard/generic approach could be more extensible, maintainable, not add technical debt, and open up possibilities for running clients remotely and so on

It would also open up support for running OpenWPM crawls on other browsers.

motin avatar Aug 16 '19 09:08 motin

WebSockets have a number of attractive qualities:

  1. They allow us to remove our own socket implementation both in the browser but also in the platform
  2. They can by default understand the difference between binary and unicode data meaning we can just send all files we want to save as binary data that get written directly to storage
  3. They have a well defined open and close sequence allowing us to shut down in an orderly manner
  4. All of their implementations are async by default allowing us to make more parts of our code base non blocking

Known problems:

  • Today we use our sockets to send pickled objects to different processes, which is functionality we'd need to implement for WebSockets, luckily the spec allows defining custom protocols (on the same level as the binary and unicode data) and the python WebSocket implementation allows us to define such a custom protocol. Meaning we could still send pickled objects and have them automatically unpickled on the other side.
  • We loose the ability to use the built-in Socket logger. However I think we should have a discussion about if the MPLogger really meets our needs at this point or if we should consider moving forward with something else. See #788

vringar avatar Oct 14 '20 00:10 vringar