snowflake-connector-python
snowflake-connector-python copied to clipboard
SNOW-228474: Support converting VARIANT to python object
What this pr does:
Adds the convert_arrow_variant_to_object parameter to SnowflakeConnection.
If this parameter value is True columns of the types VARIANT, ARRAY and OBJECT are retuned as python object and not string.
To support this feature I added
- The
convert_arrow_variant_to_objectparameter and property. The default value of this parameter is False to preserve the old behavior. - The method
VARIANT_to_pythoninArrowConverterContextthat usesjson.loadsif the flag is True. - A cpp
VariantConverterclass that calls the methodVARIANT_to_python. - Some test to tests the new functionality.
Why is this change necessary: I wrote about it in #544 :
When querying a variant (or array and object) column the results are returned as dumped json string. In the past before the Arrow format support one could've used the converter_class parameter to pass its own subclass of
SnowflakeConverterthat callsjson.loadson the returned string. But when using the Arrow format this parameter has no affect.
I hope my changes make sense in the project and I would be happy to fix any comments you have.
I see that some checks are failing but it doesn't seem to be related to my changes (looks like a missing environment variable). Should I change anything in order to resolve it ?
I see that some checks are failing but it doesn't seem to be related to my changes (looks like a missing environment variable). Should I change anything in order to resolve it ?
No, don't worry about those. Once someone from the team reviews your code for malicious code (as an example dumping password) we will kick of the same tests for you with our secrets added.