[Spark] expect_column_pair_values_A_to_be_greater_than_B(parse_strings_as_datetimes=True) is not working properly

[mykola-yesypchuk-inflection]

https://github.com/great-expectations/great_expectations/blob/1f2bc4d3c33faa898f76ecd3bf267bd788bae9bc/great_expectations/dataset/sparkdf_dataset.py#L1362

Data: "col_a": ["07/11/1967", "04/07/1966", "01/03/1953"], "col_b": ["08/10/1960", "03/07/1966", "01/01/1900"], Engine: Spark Command: validator.expect_column_pair_values_A_to_be_greater_than_B("col_b", "col_a", parse_strings_as_datetimes=True) Expected: expectation result=Failed Now: expectation result=True

Details: When we compare dates fields(string fields in '%m/%d/%Y' format) using expect_column_pair_values_A_to_be_greater_than_B, all tests will be passed. I guess the reason that sparktypes.TimestampType() used in the code of expectation. The string value can't be parsed correctly and ignore all rows.

[austiezr]

Hey @mykola-yesypchuk-inflection ! Thanks for raising this; we'll investigate and be in touch.

[austiezr]

Hey @mykola-yesypchuk-inflection ! Unfortunately, we are no longer actively developing the V2 API. Additionally, Great Expectations is stepping back from performing any type of input transformation (we believe testing should be done on data as it is; see this blog post for further explanation).

If you're interested in contributing a fix in this area, I would be happy to provide review and guidance; however, at this time I'm closing out this issue.

[mykola-yesypchuk-inflection]

Thank you for your response.

On Mon, Nov 7, 2022 at 7:53 PM Austin Ziech Robinson < @.***> wrote:

Hey @mykola-yesypchuk-inflection https://urldefense.com/v3/__https://github.com/mykola-yesypchuk-inflection__;!!OaIjtfQ!JBLokSFyJLSRJMEF7PAUP8NS2-fyyiTojVoSy8iViTLboSEaTJ2w6p_NHijok1FnFpU6S_VkRqsveF3nE2SfR10brvqORS0OAlI$ ! Unfortunately, we are no longer actively developing the V2 API. Additionally, Great Expectations is stepping back from performing any type of input transformation (we believe testing should be done on data as it is; see this blog post https://urldefense.com/v3/__https://greatexpectations.io/blog/why_we_dont_do_transformations_for_expectations/__;!!OaIjtfQ!JBLokSFyJLSRJMEF7PAUP8NS2-fyyiTojVoSy8iViTLboSEaTJ2w6p_NHijok1FnFpU6S_VkRqsveF3nE2SfR10brvqOYNzpT7k$ for further explanation).

If you're interested in contributing a fix in this area, I would be happy to provide review and guidance; however, at this time I'm closing out this issue.

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/great-expectations/great_expectations/issues/5888*issuecomment-1305977636__;Iw!!OaIjtfQ!JBLokSFyJLSRJMEF7PAUP8NS2-fyyiTojVoSy8iViTLboSEaTJ2w6p_NHijok1FnFpU6S_VkRqsveF3nE2SfR10brvqOmP75QH8$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ATWS4SLB4HYDKKXNTJQCUSTWHE62RANCNFSM6AAAAAAQBODCIQ__;!!OaIjtfQ!JBLokSFyJLSRJMEF7PAUP8NS2-fyyiTojVoSy8iViTLboSEaTJ2w6p_NHijok1FnFpU6S_VkRqsveF3nE2SfR10brvqOE7unlV0$ . You are receiving this because you were mentioned.Message ID: @.***>