airbytehq
[source-amazon-seller-partner] missing `order-id` in `GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL` stream
Connector Name
source-amazon-seller-partner
Connector Version
3.0.0
What step the error happened?
During the sync
Relevant information
I synchronized the stream GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL from source-amazon-seller-partner to destination-clickhouse:0.2.5. In the raw table in the Destination, the JSON does not contain the order-id, which results in the order-id field having no value in the table after the Normalized tabular data process. The raw data only contains the following keys: "purchase-date" "order-channel" "ship-service-level" "product-name" "sku" "currency" "item-price" "item-tax" "shipping-price" "gift-wrap-price" "gift-wrap-tax" "item-promotion-discount" "ship-city" "ship-state" "ship-postal-code" "ship-country" "is-business-order" "price-designation" "dataEndTime"
It's worth mentioning that when I directly requested the data for the same day through the seller-partner API's GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL, the returned data did contain the amazon-order-id field, and all had values.
Could you please advise if my configuration is incorrect, or if this is a bug?
It's worth mentioning that the following is my source configuration:
The "order-id" field in the table after "Normalized tabular data" is all empty.
Relevant log output
2023-12-26 09:39:17 platform > Docker volume job log path: /tmp/workspace/135/0/logs.log
2023-12-26 09:39:17 platform > Executing worker wrapper. Airbyte version: 0.50.35
2023-12-26 09:39:17 platform > Attempt 0 to save workflow id for cancellation
2023-12-26 09:39:17 platform >
2023-12-26 09:39:17 platform > Running with normalization version: airbyte/normalization-clickhouse:0.4.3
2023-12-26 09:39:17 platform > ----- START DEFAULT NORMALIZATION -----
2023-12-26 09:39:17 platform >
2023-12-26 09:39:17 platform > Checking if airbyte/normalization-clickhouse:0.4.3 exists...
2023-12-26 09:39:17 platform > airbyte/normalization-clickhouse:0.4.3 was found locally.
2023-12-26 09:39:17 platform > Creating docker container = normalization-clickhouse-normalize-135-0-qwxdf with resources io.airbyte.config.ResourceRequirements@4372fc89[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-12-26 09:39:17 platform > Preparing command: docker run --rm --init -i -w /data/135/0/normalize --log-driver none --name normalization-clickhouse-normalize-135-0-qwxdf --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e AIRBYTE_VERSION=0.50.35 airbyte/normalization-clickhouse:0.4.3 run --integration-type clickhouse --config destination_config.json --catalog destination_catalog.json
2023-12-26 09:39:17 normalization > Running: transform-config --config destination_config.json --integration-type clickhouse --out /data/135/0/normalize
2023-12-26 09:39:18 normalization > Namespace(config='destination_config.json', integration_type=<destinationtype.clickhouse:>, out='/data/135/0/normalize')
2023-12-26 09:39:18 normalization > transform_clickhouse
2023-12-26 09:39:19 normalization > Running: transform-catalog --integration-type clickhouse --profile-config-dir /data/135/0/normalize --catalog destination_catalog.json --out /data/135/0/normalize/models/generated/ --json-column _airbyte_data
2023-12-26 09:39:20 normalization > Processing destination_catalog.json...
2023-12-26 09:39:20 normalization > Truncating amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL (#68) to amz_spapi_bc_na2_GET_BY_ORDER_DATE_GENERAL (#43)
2023-12-26 09:39:20 normalization > Truncating amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL (#68) to amz_spapi_bc_na2_GET_BY_ORDER_DATE_GENERAL (#43)
2023-12-26 09:39:20 normalization > Generating airbyte_ctes/airbyte_spapi_bcna/amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL_ab1.sql from amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL
2023-12-26 09:39:20 normalization > Truncating amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL (#68) to amz_spapi_bc_na2_GET_BY_ORDER_DATE_GENERAL (#43)
2023-12-26 09:39:20 normalization > Generating airbyte_ctes/airbyte_spapi_bcna/amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL_ab2.sql from amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL
2023-12-26 09:39:20 normalization > Truncating amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL (#68) to amz_spapi_bc_na2_GET_BY_ORDER_DATE_GENERAL (#43)
2023-12-26 09:39:20 normalization > Truncating _airbyte_amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL_hashid (#59) to _airbyte_amz_spapi_b_R_DATE_GENERAL_hashid (#43)
2023-12-26 09:39:20 normalization > Truncating amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL (#68) to amz_spapi_bc_na2_GET_BY_ORDER_DATE_GENERAL (#43)
2023-12-26 09:39:20 normalization > Generating airbyte_ctes/airbyte_spapi_bcna/amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL_ab3.sql from amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL
2023-12-26 09:39:20 normalization > Truncating amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL (#68) to amz_spapi_bc_na2_GET_BY_ORDER_DATE_GENERAL (#43)
2023-12-26 09:39:20 normalization > Truncating _airbyte_amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL_hashid (#59) to _airbyte_amz_spapi_b_R_DATE_GENERAL_hashid (#43)
2023-12-26 09:39:20 normalization > Truncating amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL (#68) to amz_spapi_bc_na2_GET_BY_ORDER_DATE_GENERAL (#43)
2023-12-26 09:39:20 normalization > Adding drop table hook for amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL_scd to amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL
2023-12-26 09:39:20 normalization > Generating airbyte_tables/airbyte_spapi_bcna/amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL.sql from amz_spapi_bc_na2_GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL
2023-12-26 09:39:20 normalization > detected no config file for ssh, assuming ssh is off.
2023-12-26 09:39:30 normalization > 09:39:30 Running with dbt=1.4.6
2023-12-26 09:39:30 normalization > 09:39:30 Unable to do partial parsing because saved manifest not found. Starting full parse.
2023-12-26 09:39:34 normalization > 09:39:34 [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
2023-12-26 09:39:34 normalization > There are 2 unused configuration paths:
2023-12-26 09:39:34 normalization > - models.airbyte_utils.generated.airbyte_incremental
2023-12-26 09:39:34 normalization > - models.airbyte_utils.generated.airbyte_views
2023-12-26 09:39:34 normalization > 09:39:34 Found 4 models, 0 tests, 0 snapshots, 0 analyses, 753 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
2023-12-26 09:39:34 normalization > 09:39:34
2023-12-26 09:39:34 normalization > 09:39:34 Concurrency: 1 threads (target='prod')
2023-12-26 09:39:34 normalization > 09:39:34
2023-12-26 09:39:36 normalization > 09:39:36 1 of 1 START sql table model airbyte_spapi_bcna.amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL ............................ [RUN]
2023-12-26 09:39:37 normalization > 09:39:37 1 of 1 OK created sql table model airbyte_spapi_bcna.amz_spapi_bc_na2_GET__BY_ORDER_DATE_GENERAL ....................... [OK in 1.49s]
2023-12-26 09:39:37 normalization > 09:39:37
2023-12-26 09:39:37 normalization > 09:39:37 Finished running 1 table model in 0 hours 0 minutes and 3.32 seconds (3.32s).
2023-12-26 09:39:37 normalization > 09:39:37
2023-12-26 09:39:37 normalization > 09:39:37 Completed successfully
2023-12-26 09:39:37 normalization > 09:39:37
2023-12-26 09:39:37 normalization > 09:39:37 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2023-12-26 09:39:39 platform > Terminating normalization process...
2023-12-26 09:39:39 normalization > Unable to connect ClickHouse Connect C to Numpy API [No module named 'numpy'], falling back to pure Python
2023-12-26 09:39:39 platform > Normalization process successfully terminated.
2023-12-26 09:39:39 platform > Normalization executed in 22 seconds for job 135.
2023-12-26 09:39:39 platform > Normalization summary: io.airbyte.config.NormalizationSummary@25988aa9[startTime=1703583557342,endTime=1703583579485,failures=[],additionalProperties={}]
2023-12-26 09:39:39 platform >
2023-12-26 09:39:39 platform > ----- END DEFAULT NORMALIZATION -----
2023-12-26 09:39:39 platform >
Contribute
- [X] Yes, I want to contribute
Made quick change to the title and added labels.
For other streams has order-id field, are you able to find the values?
yes . stream GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL has field amazon-order-id in raw json. and amazon-order-id in rows has value . raw json field contains the following keys:
"amazon-order-id" "merchant-order-id" "shipment-id" "shipment-item-id" "amazon-order-item-id" "merchant-order-item-id" "purchase-date" "payments-date" "shipment-date" "reporting-date" "buyer-email" "buyer-name" "buyer-phone-number" "sku" "product-name" "quantity-shipped" "currency" "item-price" "item-tax" "shipping-price" "shipping-tax" "gift-wrap-price" "gift-wrap-tax" "ship-service-level" "recipient-name" "ship-address-1" "ship-address-2" "ship-address-3" "ship-city" "ship-state" "ship-postal-code" "ship-country" "ship-phone-number" "bill-address-1" "bill-address-2" "bill-address-3" "bill-city" "bill-state" "bill-postal-code" "bill-country" "item-promotion-discount" "ship-promotion-discount" "carrier" "tracking-number" "estimated-arrival-date" "fulfillment-center-id" "fulfillment-channel" "sales-channel" "dataEndTime"
Hello @ojhtyy,
The schema for the stream GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL has changed in version 3.0.0 (please see PR) and it doesn't have order-id field anymore (please see the list of fields in use). Instead, there is amazon-order-id field now.
With this in mind, could you please refresh the source schema, reset this stream and try to sync your data again? Please let me know if it helped. Thank you!
Closing the ticket for now as no response from the customer. Please reopen if needed.