🎉 Source Mixpanel low code migration

[midavadim]

trafficstars

https://github.com/airbytehq/airbyte-internal-issues/issues/6915

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		May 7, 2024 8:15pm

[midavadim]

Due to API rate limits limitation (1 req per minute), the full run of the regression tool took 20 hrs and 670000 records were read.

So "read" went well:

stream	control record count	target record count	Δ
cohorts	26	26	0
engage	7001	7001	0
export	652787	652502	-285
funnels	952	2030	1078
annotations	13	13	0
revenue	371	370	-1
cohort_members	0	1584	1584

Part of tests failed:

Results (78538.96s (21:48:58)): 3 failed - test_read.py:272 TestDataIntegrity.test_record_count_without_state ______ expected - test_read.py:333 TestDataIntegrity.test_all_pks_are_produced_in_target_version_without_state ______ failed due to "Object of type AirbyteMessage is not JSON serializable" - test_read.py:383 TestDataIntegrity.test_record_schema_match_without_state ______ it is expected because engage and export streams have dynamic schemas 1 error: ______ duckdb.duckdb.NotImplementedException: Not implemented Error: Duplicate name "utm_medium" in struct auto-detected in JSON, try ignore_errors=true

[natikgadzhi]

In https://github.com/airbytehq/airbyte/issues/37833, someone wants to use composite PKs for Members stream, and that change, if it makes sense, would be breaking. @midavadim, should we roll it into this PR so we avoid two breaking changes on a connector in a row?

Will this PR be a breaking change on it's own, or a clean migration?

[RobinHerzog]

In #37833, someone wants to use composite PKs for Members stream, and that change, if it makes sense, would be breaking. @midavadim, should we roll it into this PR so we avoid two breaking changes on a connector in a row?

Will this PR be a breaking change on it's own, or a clean migration?

Does Airbyte has usage tracking over streams ? I think I saw segment in the logs

I understand it's a breaking change but it's weird no-one every reported it. Currently it's not possible to use this specific stream as it gives wrong answer.

[natikgadzhi]

Usage tracking: yes, for Cloud, but I'm not entirely sure about per-stream data in OSS installations. We'll need to take a look.

You'd be surprised. ;)

[midavadim]

there are a few unanswered questions.

main concerns:

• there's a lot of custom components. Are they all necessary?
• There's a strange incremental_sync component. how is it expected to work?

1. I have re-checked that all components are needed. I created related issues to indicate problems with low code: https://github.com/airbytehq/airbyte-internal-issues/issues/7602 https://github.com/airbytehq/airbyte-internal-issues/issues/7601 https://github.com/airbytehq/airbyte-internal-issues/issues/7603
2. I removed one odd incremental_sync component

[midavadim]

@natikgadzhi @RobinHerzog lets prohandle key for cohost members stream in separate PR