Source Stripe: does not support Incremental Sync Mode for Invoice Line Items

[satyanash]

Connector Name

source-stripe

Connector Version

3.17.4

What step the error happened?

During the sync

Revelant information

There are two streams in the Stripe Source:

• invoices
• invoice_line_items which is a "sub-stream" of invoices.

The invoice_line_items does not support the "Incremental | Append" sync mode, even though its parent stream invoices supports it. Any of the two remaining options cause a Full Refresh on the invoices stream to happen on every sync, even if that parent stream is set to "Incremental | Append" mode. I have attached the relevant log lines which indicate that Airbyte does two separate syncs, and fetching a super-set of the data when syncing the invoice_line_items from the parent stream. This also defeats the purpose of having "Incremental | Append" on the parent stream, since the sub-stream causes a "Full Refresh ..." of the parent stream anyway and takes a much longer time.

Relevant log output

{
  "streamStats" : [ {
    "streamName" : "invoices",
    "stats" : {
      "bytesCommitted" : 147562,
      "bytesEmitted" : 147562,
      "recordsEmitted" : 32,
      "recordsCommitted" : 32
    }
  }, {
    "streamName" : "invoice_line_items",
    "stats" : {
      "bytesCommitted" : 194884577,
      "bytesEmitted" : 194884577,
      "recordsEmitted" : 129096,
      "recordsCommitted" : 129096
    }
  } ]
}

Contribute

• [ ] Yes, I want to contribute

[davydov-d]

Incremental sync mode added to InvoiceLineItems in version 4.0.0

[maxi297]

It seems like this issue was closed while the InvoiceLineItems stream had never been incremental. Hence, I'll re-open the issue

[enkeboll]

I'd like to call out that SubscriptionItems is also not incremental, as of version v5.3.9, despite having a created timestamp that could conceivably be used (or tied to its Subscription parent)

SubscriptionItems and InvoiceLineItems represented 99% of our billed rows per sync, so we had to separate them into separate connections to sync on different schedules than the incremental tables.

[FilahAnas]

Hello, Here is a PR to resolve this issue: 43711 @lideke

[williamkaper]

I put up a PR that adds incremental support to both invoice line items and subscription items: https://github.com/airbytehq/airbyte/pull/45860