soorgeon icon indicating copy to clipboard operation
soorgeon copied to clipboard

working with unserializable objects

Open edublancas opened this issue 4 years ago • 3 comments

e.g., the notebook uses a db connection:

conn = open_db_connection()

We should not serialize this object, but rather embed this line in any of the tasks that use it. How do we know if serialize it or not? Easiest way is to have the user add a "preparation" cell, and then add it if any of the tasks use the variables as inputs

edublancas avatar Nov 17 '21 19:11 edublancas

In short, what happens today is that soorgeon throws an error on global variables. We want to:

  • Check if there are any (code exist already).
  • Check if the vars are serialzible (as mentioned above).
  • Instead of throwing an error embed those vars where they're being used.
  • Adding the necessary tests and doc updates

@edublancas clarification on the 2nd point, we don't serialize/reinstantiate the objects to maintain the state? (i.e in the db_conn we don't really care what happened before). Ideally the users won't need to take this extra step - it should happen for them.

idomic avatar Jun 16 '22 22:06 idomic

I changed the title since it wasn't accurate. this isn't about all types of global variables. but about variables defined in the notebook that do not support serialization.

A typical example is connections to a database or logging objects. The problem is that we won't know if the notebook contains non-serializable objects because soorgeon only performs static analysis, it never runs the code.

edublancas avatar Jun 17 '22 07:06 edublancas

Keeping this for a later stage, until we'll have a run function that can run the user's code

idomic avatar Jun 17 '22 15:06 idomic