snowpark-python
snowpark-python copied to clipboard
SNOW-1163211: Refactor StringType to allow variable max size
-
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-1163211
-
Fill out the following pre-review checklist:
- [ ] I am adding a new automated test(s) to verify correctness of my new code
- [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
- [ ] I am adding new logging messages
- [ ] I am adding a new telemetry message
- [ ] I am adding new credentials
- [ ] I am adding a new dependency
- [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes.
- [ ] I am adding a new automated test(s) to verify correctness of my new code
-
Please describe how your code solves the related issue.
This PR prepares snowpark for a future change to handling large objects. This mainly affects binary and string columns. Our current snowpark binary type columns do not have a concept for size so will be unaffected by this change. String type columns on the other had need to be made aware of the new max size. To implement support for this I have done the following:
- Added a new property to the server connection that provides the max string size when connecting. This is provided by the
VARCHAR_AND_BINARY_MAX_SIZE_IN_RESULT
parameter that is returned on successful connection of a session. This parameter is not always set so if it is missing we will default to the previous max string size of 16mb. Local testing connections will remain on the old max size for now. - Added a property to StringType that can be set to say that the type is of max size. This is set by default when a string length is not specified. It can also be set manually when both the length and the max size are known.
- When parsing schema metadata the max size from the connection is used to determine if StringType columns are of max size and if so the type flag is set.
- Normalized the repr for StringType so that max size strings always are displayed without length. This was inconsistent before. User defined max size strings would be empty while server side strings would always have the max size in the repr.
- For now I've updated the tests that are affected by this change and added a unit test to ensure the new functionality works as expected for StringType comparison and repr. LOB support is an account level parameter which makes integration tests difficult for now.
- Added a new property to the server connection that provides the max string size when connecting. This is provided by the