snowflake-connector-python icon indicating copy to clipboard operation
snowflake-connector-python copied to clipboard

SNOW-228474: Support converting VARIANT to python object

Open roi-meir opened this issue 5 years ago • 2 comments

What this pr does:

Adds the convert_arrow_variant_to_object parameter to SnowflakeConnection. If this parameter value is True columns of the types VARIANT, ARRAY and OBJECT are retuned as python object and not string.

To support this feature I added

  • The convert_arrow_variant_to_object parameter and property. The default value of this parameter is False to preserve the old behavior.
  • The method VARIANT_to_python in ArrowConverterContext that uses json.loads if the flag is True.
  • A cpp VariantConverter class that calls the method VARIANT_to_python.
  • Some test to tests the new functionality.

Why is this change necessary: I wrote about it in #544 :

When querying a variant (or array and object) column the results are returned as dumped json string. In the past before the Arrow format support one could've used the converter_class parameter to pass its own subclass of SnowflakeConverter that calls json.loads on the returned string. But when using the Arrow format this parameter has no affect.

I hope my changes make sense in the project and I would be happy to fix any comments you have.

roi-meir avatar Nov 21 '20 17:11 roi-meir

I see that some checks are failing but it doesn't seem to be related to my changes (looks like a missing environment variable). Should I change anything in order to resolve it ?

roi-meir avatar Nov 21 '20 19:11 roi-meir

I see that some checks are failing but it doesn't seem to be related to my changes (looks like a missing environment variable). Should I change anything in order to resolve it ?

No, don't worry about those. Once someone from the team reviews your code for malicious code (as an example dumping password) we will kick of the same tests for you with our secrets added.

sfc-gh-mkeller avatar Nov 23 '20 01:11 sfc-gh-mkeller