Cannot un-pickle a pickled Operation instance 🥒

[chartpath]

Hey, I'm using sgqlc to introspect a schema and then generate queries from it and it is going well. However, I'm trying to parallelize tasks that contain an operation, which requires pickling it inside the task.

Here is the traceback that arises when trying to deserialize the (successfully) serialized op:

Traceback (most recent call last):
  File "/Users/myuser/myproj/venv/lib/python3.7/site-packages/sgqlc/types/__init__.py", line 657, in __getattr__
    return self.__kinds[key]  # .type, .scalar, etc...
  File "/Users/myuser/myproj/venv/lib/python3.7/site-packages/sgqlc/types/__init__.py", line 657, in __getattr__
    return self.__kinds[key]  # .type, .scalar, etc...
  File "/Users/myuser/myproj/venv/lib/python3.7/site-packages/sgqlc/types/__init__.py", line 657, in __getattr__
    return self.__kinds[key]  # .type, .scalar, etc...
  [Previous line repeated 486 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

I can reproduce this like so (using the generated python schema from sgqlc):

import my_python_schema
from cloudpickle import dumps, loads

schema_query = my_python_schema.Query
op = Operation(schema_query)

pickled_op = dumps(op) # works
unpickled_op = loads(pickled_op) # blows up with traceback above

Are there any options I'm missing that could maybe pre-flatten the types or specify the max depth?

Thanks for the sweet project 🙂

[barbieri]

I'm not sure it will be possible to autommatically serialize/load these types as they abuse some python protocols to make it easy to use (__getattr__, __setattr__ and the likes).

I'll think a bit about it when I have some time (currently busy with work), but what you want to parallelize is to generate the GraphQL DSL (text), the server execution or the parse of response data?

If you just want to parallelize the server execution, then you can convert the operation to bytes on the main thread/process and send that to the server -- HTTPEndpoint.__call__() accepts that as well. In that case you can return the JSON to the main process/thread and from there you can "add" the operation to get the Python objects.

[barbieri]

Hi, did the above comment help?

Do you still need support with this? Otherwise please close the issue :-)

[ar-ms]

Is there any development on that ? Is it possible to create an Operation instance from a bytes representation ?

[barbieri]

no, I couldn't look into this.

[rpmcginty]

I know this is quite a late follow up, but is there any ability to deserialize raw graphql? This would allow for "copying" by serializing and deserializing.

I am also looking into pagination and am not able to avoid modifying the original operation. variables seem to be difficult especially when there are multiple selections in the query.

[barbieri]

@rpmcginty not the classes, but you should really look into variables rather than this. Actually, in most of other frameworks you're not allowed to create queries on the fly, for instance TypeScript/Apollo requires the exact DSL so it will parse and generate the matching output type definitions. Java and Swift is also the same.

We work with GraphQL since 2017 and we always used variables for actual code. The only place where we don't use it is while playing in a playground console. If you match that with some directives, such as @skip(if: ) and @include(if: ) you can even control the body shape (ie: detail x simple queries). Fragments are also very helpful in that sense.

Pagination and filtering is really one case to use such things. But since we can't do nested/grouped variables, indeed the naming may become cumbersome if you have like 20 variables, then you need to use some convention, like a prefix with the "group" (ex: user_id, user_name, ... element_id, element_name...).

Another case to do variables is mutations, usually you include all the mutation arguments as query variables and also some extra to control result selection (maybe required if your query needs parameters, like lang or currency).