ert icon indicating copy to clipboard operation
ert copied to clipboard

Stop pickling RecordTransmitters and serialize its configuration instead

Open jondequinor opened this issue 2 years ago • 3 comments

  • A storage session in a remote environment (e.g. a unix prefect task), assumes to find storage_server.json in order to know how to connect to storage. A RecordTransmitter should not rely on storage_server.json, it should inherently know how to connect.

  • RecordTransmitter is serialized using pickle. This is probably not a good idea for a IO-bound instance, as the same instance will likely have to operate in two wildly different environments (e.g. RGS vs docker in azure). The configuration for a RecordTransmitter should be serialized instead. Also, pickle will not call __init__, which might cause confusion in the long run.

Definition of done

  1. RecordTransmitters are no longer pickled
  2. RecordTransmitters inherently know how to connect

jondequinor avatar Mar 02 '22 11:03 jondequinor

An implementation was started here https://github.com/equinor/ert/pull/2050 but abandoned because it did not fix the issue it was trying to fix.

jondequinor avatar Mar 02 '22 11:03 jondequinor

Closing as related to ert3, which is no longer the direction taken by the project

eivindjahren avatar Sep 15 '22 13:09 eivindjahren

This is part of ert.data, and is still relevant

sondreso avatar Sep 16 '22 08:09 sondreso

There are is longer any class RecordTransmitter in ert.data so there is no pickling of this class. I am closing the issue. If there is some other class that should not be pickled, we can make a new issue for it.

eivindjahren avatar May 24 '24 10:05 eivindjahren