[source-shopify] Change data type of 'payment_terms' column from `string` to `object`

[icychocolate98]

Connector Name

source-shopify

Connector Version

2.0.4

What step the error happened?

Other

Relevant information

Currently, when using the Shopify connector in Airbyte, we have identified an issue related to the 'payment_terms' column within the 'orders' stream. According to the official Shopify documentation (https://shopify.dev/docs/api/admin-rest/2024-04/resources/order#resource-object), this column should be of type Object as it represents a JSON object with multiple fields, but it appears to be treated as a String during processing.

However, in our experience with the Airbyte connector, it seems that the 'payment_terms' column is being treated as a String instead of an Object. This can cause issues in data processing and incorrect interpretation of the information.

Relevant log output

Explanation of Impact:

This issue is leading to errors when moving the data into our ClickHouse database. While using the JSONExtractString() function seems to work, it's noteworthy that ClickHouse requires the JSON to be properly formatted with double quotes for keys and values to be considered valid.

Here's an example of the issue when using isValidJSON function:

Query id: 6cfa0bc3-3d06-4451-8451-2ddf56674d97

Row 1:
──────
payment_terms_obj:                                              {'id': 14597980446, 'created_at': '2024-02-28T04:02:37-05:00', 'due_in_days': 7, 'payment_schedules': [{'id': 15728345374, 'amount': '0.00', 'currency': 'EUR', 'issued_at': '2024-02-28T04:01:21-05:00', 'due_at': '2024-03-06T04:01:21-05:00', 'completed_at': '2024-03-26T09:00:37-04:00', 'created_at': '2024-02-28T04:02:37-05:00', 'updated_at': '2024-03-26T09:00:37-04:00'}], 'payment_terms_name': 'Net 7', 'payment_terms_type': 'net', 'updated_at': '2024-02-28T04:02:37-05:00'}

isValidJSON(JSONExtractString(_airbyte_data, 'payment_terms')): 0

Contribute

• [ ] Yes, I want to contribute

[marcosmarxm]

Hello @icychocolate98 I added the issue to the connector team backlog to further discussion and prioritization. Any update they will return to you here.

[icychocolate98]

That sounds great. Thank you so much! @marcosmarxm

[marcosmarxm]

@ellipsis-dev want to work on this pretty please? want to work on this pretty please?