python-bigquery-pandas icon indicating copy to clipboard operation
python-bigquery-pandas copied to clipboard

Published 20 hours ago • verified publisher icon googleapis

Improve InvalidSchema exception by adding specific fields that do not match

Open artemrys opened this issue 4 years ago • 4 comments

I am talking about this InvalidSchema exception.

It may be useful to show at most 3 (for example) fields that have different types. It should help to debug a type mismatch faster.

artemrys avatar Dec 06 '20 13:12 artemrys

Great suggestion. I think we do a "set difference" in order to discover schema mismatch, so this should be possible to figure out and include in the exception.

tswast avatar Dec 17 '20 20:12 tswast

Great, I will start working on it then.

artemrys avatar Dec 17 '20 22:12 artemrys

I'm a bit wary of including this information in the actual exception message due to potential privacy issues around logging. Instead, I'm adding local_schema and remote_schema properties to InvalidSchema in https://github.com/googleapis/python-bigquery-pandas/pull/449

tswast avatar Dec 23 '21 23:12 tswast

So long as we leave out description (and other column metadata like IAM roles), which can contain arbitrary text (possibly, though unlikely to be PII) I do think it would be an improved experience if the string representation of InvalidSchema showed the diff of field names and the types.

tswast avatar Jul 26 '22 15:07 tswast