unexpected_rows returned None or errored out for BigQuery Datasource.

[vikramshinde12]

Describe the bug When checkpoint ran with request_format COMPLETE with include_unexpected_rows e.g. result_format={'result_format': 'COMPLETE', 'partial_unexpected_count': 20, 'include_unexpected_rows': True} It returns either Error or None.

To Reproduce For expectation_type = expect_column_values_to_not_be_null, it returns None/null For expectation_type = expect_column_values_to_be_between, it gives error It seems in core/util.py module, https://github.com/great-expectations/great_expectations/blob/develop/great_expectations/core/util.py#L249 For the LegacyRow type, there is no check hence it returns error. If I add return dict(data) for this type, I get the unexpected_rows.

Expected behavior It is expected to return unexpected_row

Environment (please complete the following information):

• Operating System: MacOS
• Great Expectations Version: 0.13.46

Additional context

[cdkini]

Hey @vikramshinde12 thanks for opening up this issue!

Would you mind providing both your great_expectations.yml file (with any sensitive details omitted) as well as the full stack trace of the error you're seeing? I'll use these details to try and diagnose what the root cause is. Thanks!

[vikramshinde12]

Hey @vikramshinde12 thanks for opening up this issue!

Would you mind providing both your great_expectations.yml file (with any sensitive details omitted) as well as the full stack trace of the error you're seeing? I'll use these details to try and diagnose what the root cause is. Thanks!

Hi @cdkini The full stack trace as below

Traceback (most recent call last): File "/Users/vikramshinde/PycharmProjects/ge_v3/great_expectations/run_stg_checkpoint2.py", line 91, in results = run_checkpoint() File "/Users/vikramshinde/PycharmProjects/ge_v3/great_expectations/run_stg_checkpoint2.py", line 15, in run_checkpoint results: CheckpointResult = data_context.run_checkpoint( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/core/usage_statistics/usage_statistics.py", line 295, in usage_statistics_wrapped_method result = func(*args, **kwargs) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/data_context/data_context.py", line 3361, in run_checkpoint return checkpoint.run(**checkpoint_run_arguments) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/core/usage_statistics/usage_statistics.py", line 295, in usage_statistics_wrapped_method result = func(*args, **kwargs) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/checkpoint/checkpoint.py", line 383, in run self._run_validation( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/checkpoint/checkpoint.py", line 279, in _run_validation async_executor.submit( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/core/async_executor.py", line 97, in submit return AsyncResult(value=fn(*args, **kwargs)) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/validation_operators/validation_operators.py", line 385, in run batch_actions_results = self._run_actions( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/validation_operators/validation_operators.py", line 470, in _run_actions raise e File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/validation_operators/validation_operators.py", line 449, in _run_actions action_result = self.actions[action["name"]].run( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/checkpoint/actions.py", line 63, in run return self._run( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/checkpoint/actions.py", line 780, in _run return_val = self.target_store.set( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/data_context/store/store.py", line 163, in set self.key_to_tuple(key), self.serialize(key, value), **kwargs File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/data_context/store/validations_store.py", line 168, in serialize return self._expectationSuiteValidationResultSchema.dumps( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/schema.py", line 586, in dumps serialized = self.dump(obj, many=many) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/schema.py", line 562, in dump result = self._serialize(processed_obj, many=many) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/schema.py", line 526, in _serialize value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/fields.py", line 309, in serialize return self._serialize(value, attr, obj, **kwargs) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/fields.py", line 697, in _serialize return [self.inner._serialize(each, attr, obj, **kwargs) for each in value] File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/fields.py", line 697, in return [self.inner._serialize(each, attr, obj, **kwargs) for each in value] File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/fields.py", line 564, in _serialize return schema.dump(nested_obj, many=many) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/schema.py", line 556, in dump processed_obj = self._invoke_dump_processors( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/schema.py", line 1076, in _invoke_dump_processors data = self._invoke_processors( File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/marshmallow__shade/schema.py", line 1233, in _invoke_processors data = processor(data, many=many, **kwargs) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/core/expectation_validation_result.py", line 253, in convert_result_to_serializable data.result = convert_to_json_serializable(data.result) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/core/util.py", line 159, in convert_to_json_serializable new_dict[str(key)] = convert_to_json_serializable(data[key]) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/core/util.py", line 166, in convert_to_json_serializable new_list.append(convert_to_json_serializable(val)) File "/Users/vikramshinde/PycharmProjects/ge_v3/venv/lib/python3.9/site-packages/great_expectations/core/util.py", line 254, in convert_to_json_serializable raise TypeError( TypeError: (Decimal('7.000000000'), 'Tahir', datetime.date(2000, 7, 1), 3000) is of type LegacyRow which cannot be serialized.

[vikramshinde12]

@cdkini great_expectations.yml

I could solve this issue by adding condition for LegacyRow in core/util.py

if isinstance(data, sqlalchemy.engine.row.LegacyRow): return dict(data)

[cdkini]

@vikramshinde12 thanks for providing me with the solution! Would you want to open up a PR around this by any chance? I'm happy to merge the changes but you might want a contributor mug 🎉

[github-actions[bot]]

Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity.

It will be closed if no further activity occurs. Thank you for your contributions 🙇