Multiple conditions doesnt work on row_condition field using SqlAlchemyExecutionEngine

[oguzAydin]

I'm using great_expectations 0.16.7. I'm facing a problem when i try to use multiple conditions in row_condition field. I have a vertica and mysql database. i tried both of them and got same result. in my case, in my table there are 408 rows as 'SUCCEEDED' and 4 rows as 'CANCELLED_BEFORE' values in completion_status column. when i use single condition it works well, but when i try to use two condition it only notices first one.

My expectation is like

expectation_configuration1 = ExpectationConfiguration(
    expectation_type="expect_table_row_count_to_equal",
    kwargs={
        "value": 412,
        "row_condition": 'col("completion_status")==\'SUCCEEDED\'', #  "observed_value": 408 as expected
        #  "row_condition": 'col("completion_status")==\'CANCELLED_BEFORE\'' , # observed_value: 4 as expected
        #  "row_condition": 'col("completion_status")==\'SUCCEEDED\' OR col("completion_status")==\'CANCELLED_BEFORE\'', #  "observed_value": 408 but it should be 412
        #  "row_condition": 'col("completion_status")==\'CANCELLED_BEFORE\' OR col("completion_status")==\'SUCCEEDED\'', #  "observed_value": 4 but it should be 412
        "condition_parser": "great_expectations__experimental__"
    }
)

my datasource yaml is like

example_yaml = f"""
name: {datasource_name}
class_name: Datasource
execution_engine:
  class_name: SqlAlchemyExecutionEngine
  credentials:
    host: {host}
    port: '{port}'
    username: {username}
    password: {password}
    database: {database}
    drivername: vertica+vertica_python
data_connectors:
  default_configured_data_connector_name:
    class_name: ConfiguredAssetSqlDataConnector
    assets:
      {table_name_1}:
        class_name: Asset
        schema_name: {schema_name_1}
"""

how can we use multiple conditions on SqlAlchemyExecutionEngine? Doesn't support great expectations multiple conditions for Sql based database?

[austiezr]

Hey @oguzAydin! Thanks for raising this. We'll investigate this behavior and be in touch.

[kujaska]

Hi @austiezr , any good news so far?..

[kujaska]

why not to allow SQL WHERE conditions to be accepted and passed directly to SQL Engine instead of this weird parse-wrapper?

[kujaska]

Is anybody there? The feature is really needed to avoid preparing a lot of different batches

[kujaska]

any good news so far?

[mkzbyrad91]

GX 0.18.12 Same issue here.... Im trying to apply two filters as you can see in the json extract

{
      "expectation_type": "expect_table_row_count_to_be_between",
      "kwargs": {
        "max_value": 530.0,
        "min_value": 490.0,
        "column": "ID_CARTERA",
        "condition_parser": "great_expectations__experimental__",
        "row_condition": "col('registro_actual')==True AND col('situacion')==\"A\""
},

then Im getting this error in the resullt json in the exception_message info : unable to parse condition: col('registro_actual')==True AND col('situacion')=="A" In more detail I attach exception_traceback extract where you can see the exact code that does not allow to use free SQL filtering...

parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected {{Combine:({Suppress:('col("') W:(A-Za-z, -.0-9A-Z_a-z) Suppress:('")')}) '.NOTNULL()'} ^ {Combine:({Suppress:('col("') W:(A-Za-z, -.0-9A-Z_a-z) Suppress:('")')}) {'>' ^ '<' ^ '>=' ^ '<=' ^ '==' ^ '!='} {Re:('[+-]?\d+(?:\.\d*)?(?:[eE][+-]?\d+)?') ^ {Suppress:('"') W:(	 !#-&(-~) Suppress:('"')} ^ {Suppress:("'") W:(	 !#-&(-~) Suppress:("'")} ^ {'date' Suppress:('(') {{Suppress:('"') W:(	 !#-&(-~) Suppress:('"')} ^ {Suppress:("'") W:(	 !#-&(-~) Suppress:("'")}} Suppress:(')')}}}}, found 'col'  (at char 0), (line:1, col:1)