airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Published 20 hours ago • verified publisher icon airbytehq

Amazon SP: Fulfilled Shipments Report is always empty

Open rainyhour opened this issue 3 years ago • 3 comments

Enviroment

  • Airbyte version: v0.30.22-alpha
  • Source Connector and version: Amazon SP v0.2.1

Current Behavior

Fulfilled Shipments Report is always empty. (amazon reportType: GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL, connector stream class name: FulfilledShipmentsReports)

I guess connector does not provide dataStartTime parameter when creating a report. dataStartTime defines the start date of data to include into report. It defaults to now if not provided. (Amazon docs). Could not find any uses of this parameter in connector source code as well.

Worker log shows dataStartTime is the same as report createdTime (full logs attached below): 2021-10-27 14:35:13 WARN () DefaultAirbyteStreamFactory(internalLog):89 - There are no report document related in stream GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL. Report body {'reportType': 'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL', 'processingEndTime': '2021-10-27T14:34:29+00:00', 'processingStatus': 'CANCELLED', 'marketplaceIds': ['A1PA6795UKMFR9'], 'reportId': '71298018927', 'dataEndTime': '2021-10-27T14:34:13+00:00', 'createdTime': '2021-10-27T14:34:13+00:00', 'processingStartTime': '2021-10-27T14:34:18+00:00', 'dataStartTime': '2021-10-27T14:34:13+00:00'}

I guess other reports will also have the same issue.

Expected Behavior

Report should not be empty if orders are present. Connector should employ replication start date or some other reasonable date as start date

Logs

LOG 
2021-10-27 14:34:10 INFO () WorkerRun(call):47 - Executing worker wrapper. Airbyte version: 0.30.20-alpha
2021-10-27 14:34:11 INFO () TemporalAttemptExecution(get):94 - Executing worker wrapper. Airbyte version: 0.30.20-alpha
2021-10-27 14:34:11 WARN () Databases(createPostgresDatabaseWithRetry):38 - Waiting for database to become available...
2021-10-27 14:34:11 INFO () JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2021-10-27 14:34:11 INFO () Databases(createPostgresDatabaseWithRetry):55 - Database available!
2021-10-27 14:34:11 INFO () DefaultReplicationWorker(run):82 - start sync worker. job id: 144 attempt id: 0
2021-10-27 14:34:11 INFO () DefaultReplicationWorker(run):91 - configured sync modes: {null.GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL=full_refresh - append}
2021-10-27 14:34:11 INFO () DefaultAirbyteDestination(start):58 - Running destination...
2021-10-27 14:34:11 INFO () LineGobbler(voidCall):65 - Checking if airbyte/destination-gcs:0.1.2 exists...
2021-10-27 14:34:11 INFO () LineGobbler(voidCall):65 - airbyte/destination-gcs:0.1.2 was found locally.
2021-10-27 14:34:11 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/144/0 --network host --log-driver none airbyte/destination-gcs:0.1.2 write --config destination_config.json --catalog destination_catalog.json
2021-10-27 14:34:11 INFO () LineGobbler(voidCall):65 - Checking if airbyte/source-amazon-seller-partner:0.2.1 exists...
2021-10-27 14:34:11 INFO () LineGobbler(voidCall):65 - airbyte/source-amazon-seller-partner:0.2.1 was found locally.
2021-10-27 14:34:11 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/144/0 --network host --log-driver none airbyte/source-amazon-seller-partner:0.2.1 read --config source_config.json --catalog source_catalog.json --state input_state.json
2021-10-27 14:34:11 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):226 - Destination output thread started.
2021-10-27 14:34:11 INFO () DefaultReplicationWorker(run):119 - Waiting for source thread to join.
2021-10-27 14:34:11 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):190 - Replication thread started.
2021-10-27 14:34:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:13 [32mINFO[m i.a.i.b.IntegrationRunner(run):96 - {} - Running integration: io.airbyte.integrations.destination.gcs.GcsDestination
2021-10-27 14:34:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:13 [32mINFO[m i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2021-10-27 14:34:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:13 [32mINFO[m i.a.i.b.IntegrationRunner(run):100 - {} - Command: WRITE
2021-10-27 14:34:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:13 [32mINFO[m i.a.i.b.IntegrationRunner(run):101 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2021-10-27 14:34:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:13 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-10-27 14:34:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:13 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-10-27 14:34:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:13 [32mINFO[m i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):42 - {} - S3 format config: {"flattening":"Root level flattening","format_type":"CSV","part_size_mb":5}
2021-10-27 14:34:14 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:14 [32mINFO[m i.a.i.d.g.c.GcsCsvWriter(<init>):74 - {} - Full GCS path for stream 'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL': airbyte/data_sync/amazon/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL/2021_10_27_1635345254259_0.csv
2021-10-27 14:34:14 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:14 [32mINFO[m i.a.i.d.s.u.S3StreamTransferManagerHelper(getDefault):75 - {} - PartSize arg is set to 5 MB
2021-10-27 14:34:14 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:34:14 [32mINFO[m a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - {} - Initiated multipart upload to airbyte/data_sync/amazon/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL/2021_10_27_1635345254259_0.csv with full ID ABPnzm4BpkjS_0UlzHUbL3Cciv_oMW4JfTbBXZ7aTm8XcG06djoUhFxpmQ7sGj4F8ZdP3eKz
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(internalLog):90 - Starting syncing SourceAmazonSellerPartner
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(internalLog):90 - Syncing stream: GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL 
2021-10-27 14:35:13 WARN () DefaultAirbyteStreamFactory(internalLog):89 - There are no report document related in stream `GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL`. Report body {'reportType': 'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL', 'processingEndTime': '2021-10-27T14:34:29+00:00', 'processingStatus': 'CANCELLED', 'marketplaceIds': ['A1PA6795UKMFR9'], 'reportId': '71298018927', 'dataEndTime': '2021-10-27T14:34:13+00:00', 'createdTime': '2021-10-27T14:34:13+00:00', 'processingStartTime': '2021-10-27T14:34:18+00:00', 'dataStartTime': '2021-10-27T14:34:13+00:00'}
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(internalLog):90 - Read 0 records from GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL stream
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(internalLog):90 - Finished syncing SourceAmazonSellerPartner
2021-10-27 14:35:13 INFO () DefaultReplicationWorker(run):121 - Source thread complete.
2021-10-27 14:35:13 INFO () DefaultReplicationWorker(run):122 - Waiting for destination thread to join.
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:13 [32mINFO[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded.
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:13 [32mINFO[m i.a.i.d.g.w.BaseGcsWriter(close):129 - {} - Uploading remaining data for stream 'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL'.
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:13 [32mINFO[m a.m.s.MultiPartOutputStream(close):158 - {} - Called close() on [MultipartOutputStream for parts 1 - 10000]
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:13 [32mINFO[m a.m.s.MultiPartOutputStream(close):158 - {} - Called close() on [MultipartOutputStream for parts 1 - 10000]
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:13 [33mWARN[m a.m.s.MultiPartOutputStream(close):160 - {} - [MultipartOutputStream for parts 1 - 10000] is already closed
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:13 [32mINFO[m a.m.s.StreamTransferManager(complete):367 - {} - [Manager uploading to airbyte/data_sync/amazon/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL/2021_10_27_1635345254259_0.csv with id ABPnzm4Bp...F8ZdP3eKz]: Uploading leftover stream [Part number 1 containing 0.00 MB]
2021-10-27 14:35:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:13 [32mINFO[m a.m.s.StreamTransferManager(uploadStreamPart):558 - {} - [Manager uploading to airbyte/data_sync/amazon/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL/2021_10_27_1635345254259_0.csv with id ABPnzm4Bp...F8ZdP3eKz]: Finished uploading [Part number 1 containing 0.00 MB]
2021-10-27 14:35:14 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:14 [32mINFO[m a.m.s.StreamTransferManager(complete):395 - {} - [Manager uploading to airbyte/data_sync/amazon/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL/2021_10_27_1635345254259_0.csv with id ABPnzm4Bp...F8ZdP3eKz]: Completed
2021-10-27 14:35:14 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:14 [32mINFO[m i.a.i.d.g.w.BaseGcsWriter(close):131 - {} - Upload completed for stream 'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL'.
2021-10-27 14:35:14 INFO () JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $: null found, object expected
2021-10-27 14:35:14 ERROR () DefaultAirbyteStreamFactory(lambda$create$1):63 - Validation failed: null
2021-10-27 14:35:14 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-27 14:35:14 [32mINFO[m i.a.i.b.IntegrationRunner(run):153 - {} - Completed integration: io.airbyte.integrations.destination.gcs.GcsDestination
2021-10-27 14:35:14 INFO () DefaultReplicationWorker(run):124 - Destination thread complete.
2021-10-27 14:35:14 INFO () DefaultReplicationWorker(run):152 - sync summary: io.airbyte.config.ReplicationAttemptSummary@33b356cf[status=completed,recordsSynced=0,bytesSynced=0,startTime=1635345251093,endTime=1635345314153]
2021-10-27 14:35:14 INFO () DefaultReplicationWorker(run):161 - Source did not output any state messages
2021-10-27 14:35:14 WARN () DefaultReplicationWorker(run):169 - State capture: No new state, falling back on input state: io.airbyte.config.State@3fe83136[state={}]
2021-10-27 14:35:14 INFO () TemporalAttemptExecution(get):115 - Stopping cancellation check scheduling...
2021-10-27 14:35:14 INFO () SyncWorkflow$ReplicationActivityImpl(replicate):178 - sync summary: io.airbyte.config.StandardSyncOutput@337100b7[standardSyncSummary=io.airbyte.config.StandardSyncSummary@65a4154d[status=completed,recordsSynced=0,bytesSynced=0,startTime=1635345251093,endTime=1635345314153],state=io.airbyte.config.State@3fe83136[state={}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@3f53797e[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@cf092f6[stream=io.airbyte.protocol.models.AirbyteStream@373f5f67[name=GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL,jsonSchema={"type":"object","title":"Amazon Fulfilled Data General","$schema":"http://json-schema.org/draft-07/schema#","properties":{"sku":{"type":["null","string"]},"carrier":{"type":["null","string"]},"currency":{"type":["null","string"]},"item-tax":{"type":["null","string"]},"bill-city":{"type":["null","string"]},"ship-city":{"type":["null","string"]},"bill-state":{"type":["null","string"]},"buyer-name":{"type":["null","string"]},"item-price":{"type":["null","string"]},"ship-state":{"type":["null","string"]},"buyer-email":{"type":["null","string"]},"shipment-id":{"type":["null","string"]},"bill-country":{"type":["null","string"]},"product-name":{"type":["null","string"]},"ship-country":{"type":["null","string"]},"shipping-tax":{"type":["null","string"]},"gift-wrap-tax":{"type":["null","string"]},"payments-date":{"type":["null","string"]},"purchase-date":{"type":["null","string"]},"sales-channel":{"type":["null","string"]},"shipment-date":{"type":["null","string"]},"bill-address-1":{"type":["null","string"]},"bill-address-2":{"type":["null","string"]},"bill-address-3":{"type":["null","string"]},"recipient-name":{"type":["null","string"]},"reporting-date":{"type":["null","string"]},"ship-address-1":{"type":["null","string"]},"ship-address-2":{"type":["null","string"]},"ship-address-3":{"type":["null","string"]},"shipping-price":{"type":["null","string"]},"amazon-order-id":{"type":["null","string"]},"gift-wrap-price":{"type":["null","string"]},"tracking-number":{"type":["null","string"]},"bill-postal-code":{"type":["null","string"]},"quantity-shipped":{"type":["null","string"]},"ship-postal-code":{"type":["null","string"]},"shipment-item-id":{"type":["null","string"]},"merchant-order-id":{"type":["null","string"]},"ship-phone-number":{"type":["null","string"]},"buyer-phone-number":{"type":["null","string"]},"ship-service-level":{"type":["null","string"]},"fulfillment-channel":{"type":["null","string"]},"amazon-order-item-id":{"type":["null","string"]},"fulfillment-center-id":{"type":["null","string"]},"estimated-arrival-date":{"type":["null","string"]},"merchant-order-item-id":{"type":["null","string"]},"item-promotion-discount":{"type":["null","string"]},"ship-promotion-discount":{"type":["null","string"]}},"description":"Amazon Fulfilled Data General Reports"},supportedSyncModes=[full_refresh],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}]]

Are you willing to submit a PR?

No

rainyhour avatar Oct 27 '21 14:10 rainyhour

@rainyhour thanks for reporting the issue, pulled it into our sprint

sherifnada avatar Oct 29 '21 07:10 sherifnada

removing from sprint until SP sandbox is fixed

sherifnada avatar Dec 10 '21 05:12 sherifnada

👋 I'm interested in leveraging the Amazon Seller connector for my org. This Fulfilled Shipments report is really important for ecommerce companies because it contains a hashed email address that analysts can use to determine LTV, Churn, etc for the Amazon sales channel.

Can you confirm if this issue has been resolved? @harshithmullapudi recommended I reach out here instead of Slack.

MisterClean avatar Jul 20 '22 22:07 MisterClean

Fixed in https://github.com/airbytehq/airbyte/pull/16430

lazebnyi avatar Feb 07 '23 14:02 lazebnyi