Add SubscriptionUsage stream to Orb Source

[pmossman]

What

This is a WIP. Fair warning - this is my first real connector development so I'm definitely looking for feedback! I'm also not a python developer by any means so please give feedback on the actual python code here as well 😅

The SubscriptionUsage API Endpoint is not currently available as a stream in the Orb Source. This PR is adding a new stream for this endpoint.

I highly recommend reading the docs page for this endpoint, as this endpoint works very differently compared to other Orb API endpoints: https://docs.withorb.com/docs/orb-docs/api-reference/operations/get-a-subscription-usage

As this is my first real connector development that I've done, I don't have a great gauge for how hairy and complex the logic here is. It feels complicated, but perhaps this is par for the course when it comes to building out incremental streams against endpoints that don't offer straight-forward pagination?

I still need to add unit tests and integration tests, but I want to put up a PR draft to start collecting feedback now that I have something that seems to sort of work when I run it in Airbyte OSS 😅

Thanks for taking a look!

How

There are a few key behaviors of this endpoint that inform my implementation approach for this stream:

• This endpoint is NOT paginated in a traditional sense - instead, usage quantities for a subscription are returned over a time window that is specified by the caller.
  • If I want to incrementally fetch subscription usage since yesterday, for example, I need to pass a timeframe_start param with yesterday's date, and timeframe_end parameter with the current date. Furthermore, if I want to fetch usage broken down for each day in my time window, I need to specify granularity: day, otherwise I'll receive a single quantity representing usage over the entire time window.
  • I noticed that our Slack source behaves in a similar way, so I modeled my approach off of this section of the Slack source
  • I decided to hardcode granularity: day inside the stream because breaking down usage into single-day records makes reasoning around state much easier, and I think lends itself to an incremental stream nicely.
• It is very useful to break down subscription usage by a particular trait, like an Airbyte job_id for example. Luckily, the usage endpoint supports a group_by parameter which will return usage grouped by the event key you pass it.
  • A caveat here is that to use a group_by param, the caller must also specify a particular billing_metric_id. To work around this constraint, I implemented stream slicing to detect when a group_by is present, and yield a slice for every billing_metric_id present in the subscription's plan in this case.

Also I made a small change to the subscriptions stream while I was working on this - I added the external_customer_id to subscription records because I know this will help our analytics team out, it isn't used or needed to implement the new subscription_usage stream.

Recommended reading order

1. source.py
2. the rest

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4028696366 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4028696366 :bug: https://gradle.com/s/hmnwvmnzprkco

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
================== 5 failed, 26 passed, 31 warnings in 40.61s ==================

[pmossman]

/test connector=source-orb ref=master

[pmossman]

/test connector=source-orb

[pmossman]

/test connector=source-orb

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4057339653 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4057339653 :bug: https://gradle.com/s/e7vrwo76ufijs

Build Failed

Test summary info:

Could not find result summary

[pmossman]

/test connector=source-orb

[pmossman]

/test connector=source-orb

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4058891960 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4058891960 :bug: https://gradle.com/s/x3kbqo53jbmzk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
============ 1 failed, 36 passed, 37 warnings in 345.53s (0:05:45) =============

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4059278924 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4059278924 :bug: https://gradle.com/s/fujndrtu6hpk2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
============ 1 failed, 36 passed, 37 warnings in 357.19s (0:05:57) =============

[pmossman]

^ The integration test failed because we don't have any subscription usage data in the test account that the tests call. I am following up with Orb to see if they can add some test data to the account. We have an API key for the account, but I don't see login credentials in lastpass, so I'm assuming Orb has access on their end and can help us add some fake data

[girarda]

@YowanR do you know where we can get the credentials for the orb test account? can whoever has them upload them to lastpass?

[pmossman]

@YowanR @girarda I got in touch with a dev at Orb who has account access, and he will send me credentials soon that I will add to LastPass

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4077003891 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4077003891 :bug: https://gradle.com/s/7loyn6r67trrs

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
============ 1 failed, 36 passed, 37 warnings in 632.82s (0:10:32) =============

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4079447200 :white_check_mark: source-orb https://github.com/airbytehq/airbyte/actions/runs/4079447200 Python tests coverage:

Name                     Stmts   Miss  Cover
--------------------------------------------
source_orb/__init__.py       2      0   100%
source_orb/source.py       323     40    88%
--------------------------------------------
TOTAL                      325     40    88%
	 Name                                                    Stmts   Miss  Cover   Missing
	 -------------------------------------------------------------------------------------
	 connector_acceptance_test/base.py                          12      4    67%   16-19
	 connector_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 connector_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 connector_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 connector_acceptance_test/tests/test_core.py              476    117    75%   53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
	 connector_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 connector_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 connector_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 connector_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 connector_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 connector_acceptance_test/utils/json_schema_helper.py     114     13    89%   31-32, 39, 42, 66-69, 97, 121, 203-205
	 -------------------------------------------------------------------------------------
	 TOTAL                                                    1690    341    80%

Build Passed

Test summary info:

All Passed

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4088220905 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4088220905 :bug: https://gradle.com/s/o3waxljkrxtb6

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_streams.py::test_subscription_usage_schema[foo-key] - ...
	 FAILED unit_tests/test_streams.py::test_subscription_usage_schema[None] - Typ...
	 [31m================== [31m[1m2 failed[0m, [32m60 passed[0m, [33m132 warnings[0m[31m in 0.83s[0m[31m ==================[0m

[pmossman]

@girarda I made start_date a required field and set the new version to 1.0.0 to reflect the breaking change. Let me know if there's anything else I should address! I had to set backward_compatibility_tests_config.disable_for_version: "0.1.4" in the acceptance test config to get the tests to pass locally, hopefully that's the right way to go about making a breaking change!

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4088383752 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4088383752 :bug: https://gradle.com/s/t2aik4nl2aicy

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_config_match_spec[inputs0] - Failed: Conf...
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:107: Backward compatibility tests are disabled for version 0.1.4.
======= 3 failed, 20 passed, 1 skipped, 37 warnings, 13 errors in 34.21s =======

> Task :airbyte-integrations:connectors:source-orb:connectorAcceptanceTest FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
48 actionable tasks: 23 executed, 25 up-to-date

Publishing build scan...
https://gradle.com/s/t2aik4nl2aicy

The remote build cache was disabled during the build due to errors.
S3 cache writes: 1, elapsed: 630ms, sent to cache: 446 B

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4088533799 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4088533799 :bug: https://gradle.com/s/hngoztttoqww6

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_config_match_spec[inputs0] - Failed: Conf...
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:107: Backward compatibility tests are disabled for version 0.1.4.
======= 3 failed, 20 passed, 1 skipped, 37 warnings, 13 errors in 34.36s =======

> Task :airbyte-integrations:connectors:source-orb:connectorAcceptanceTest FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
48 actionable tasks: 23 executed, 25 up-to-date

Publishing build scan...
https://gradle.com/s/hngoztttoqww6

The remote build cache was disabled during the build due to errors.
S3 cache writes: 1, elapsed: 707ms, sent to cache: 451 B

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4089058038 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4089058038 :bug: https://gradle.com/s/5qsqohpobb5fy

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_config_match_spec[inputs0] - Failed: Conf...
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:107: Backward compatibility tests are disabled for version 0.1.4.
======= 3 failed, 20 passed, 1 skipped, 37 warnings, 13 errors in 33.23s =======

> Task :airbyte-integrations:connectors:source-orb:connectorAcceptanceTest FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
48 actionable tasks: 23 executed, 25 up-to-date

Publishing build scan...
https://gradle.com/s/5qsqohpobb5fy

The remote build cache was disabled during the build due to errors.
S3 cache writes: 1, elapsed: 1617ms, sent to cache: 450 B

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4089171977 :x: source-orb https://github.com/airbytehq/airbyte/actions/runs/4089171977 :bug:

[pmossman]

/test connector=source-orb

:clock2: source-orb https://github.com/airbytehq/airbyte/actions/runs/4089191734 :white_check_mark: source-orb https://github.com/airbytehq/airbyte/actions/runs/4089191734 Python tests coverage:

Name                     Stmts   Miss  Cover
--------------------------------------------
source_orb/__init__.py       2      0   100%
source_orb/source.py       323     40    88%
--------------------------------------------
TOTAL                      325     40    88%
	 Name                                                    Stmts   Miss  Cover   Missing
	 -------------------------------------------------------------------------------------
	 connector_acceptance_test/base.py                          12      4    67%   16-19
	 connector_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 connector_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 connector_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 connector_acceptance_test/tests/test_core.py              476    117    75%   53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
	 connector_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 connector_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 connector_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 connector_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 connector_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 connector_acceptance_test/utils/json_schema_helper.py     114     13    89%   31-32, 39, 42, 66-69, 97, 121, 203-205
	 -------------------------------------------------------------------------------------
	 TOTAL                                                    1690    341    80%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:107: Backward compatibility tests are disabled for version 0.1.4.
============ 36 passed, 1 skipped, 37 warnings in 284.21s (0:04:44) ============

[pmossman]

@girarda sorry for how noisy this PR has been, I finally figured out the GSM secret changes I needed to make to get the tests all passing now that start_date is required. Should be good for review now!

[pmossman]

/publish connector=source-orb

:clock2: Publishing the following connectors:
source-orb
https://github.com/airbytehq/airbyte/actions/runs/4116419734

Connector	Did it publish?	Were definitions generated?
source-orb	:x:	:x:

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[pmossman]

/publish connector=connectors/source-orb

:clock2: Publishing the following connectors:
connectors/source-orb
https://github.com/airbytehq/airbyte/actions/runs/4116561412

Connector	Did it publish?	Were definitions generated?
connectors/source-orb	:white_check_mark:	:white_check_mark:

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[github-actions[bot]]

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	24.53%	:x: