Source Okta: add permission stream under a custom role

[YiyangLi]

What

Add permissions to okta source connectors https://developer.okta.com/docs/reference/api/roles/#list-permissions Each org is allowed to create at most 100 custom roles (added in this PR https://github.com/airbytehq/airbyte/pull/14610), and each custom roles can add at most 32 permission. The API always returns all permission given a role id, so it's a full refresh

How

Describe the solution

Recommended reading order

1. source.python

🚨 User Impact 🚨

No impact, new API stream

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• [x] Community member? Grant edit access to maintainers (instructions)
• [ ] Secrets in the connector's spec are annotated with airbyte_secret
• [x] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• [ ] Code reviews completed
• [ ] Documentation updated
  • [ ] Connector's README.md
  • [ ] Connector's bootstrap.md. See description and examples
  • [x] docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • [ ] docs/integrations/README.md
  • [ ] airbyte-integrations/builds.md
• [ ] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• [ ] Create a non-forked branch based on this PR and test the below items on it
• [ ] Build is successful
• [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
• [ ] /test connector=connectors/<name> command is passing
• [ ] New Connector version released on Dockerhub by running the /publish command described here
• [ ] After the connector is published, connector added to connector index as described here
• [ ] Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• [ ] Grant edit access to maintainers (instructions)
• [ ] Secrets in the connector's spec are annotated with airbyte_secret
• [ ] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• [ ] Code reviews completed
• [ ] Documentation updated
  • [ ] Connector's README.md
  • [ ] Connector's bootstrap.md. See description and examples
  • [ ] Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• [ ] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• [ ] Create a non-forked branch based on this PR and test the below items on it
• [ ] Build is successful
• [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
• [ ] /test connector=connectors/<name> command is passing
• [ ] New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• [ ] Issue acceptance criteria met
• [ ] PR name follows PR naming conventions
• [ ] If adding a new generator, add it to the list of scaffold modules being tested
• [ ] The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• [ ] Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

collecting ... 
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_config_match_spec[inputs0] ✓                                                                                                                       4% ▌         
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓                                                                                                                          8% ▉         
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_docker_env[inputs0] ✓                                                                                                                             12% █▍        
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oneof_usage[inputs0] ✓                                                                                                                            17% █▋        
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓                                                                                                                               21% ██▏       
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓                                                                                                                               25% ██▌       
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓                                                                                                                             29% ██▉       
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓                                                                                                             33% ███▍      
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓                                                                                                   38% ███▊      
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓                                                                                                                  42% ████▎     
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓                                                                                                                            46% ████▋     
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓                                                                                                                            50% █████     
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓                                                                                                                          54% █████▌    
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓                                                                                                   58% █████▉    
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓                                                                                                      62% ██████▍   
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-allOf] ✓                                                                                             67% ██████▋   
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-not] ✓                                                                                               71% ███████▏  
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_primary_keys_exist_in_schema[inputs0] ✓                                                                                                      75% ███████▌  
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓                                                                                                                              79% ███████▉  {"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\n{\"type\": \"TRACE\", \"trace\": {\"type\": \"ERROR\", \"emitted_at\": 1657855159900.5042, \"error\": {\"message\": \"Something went wrong in the connector. See the logs for more details.\", \"internal_message\": \"2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\", \"stack_trace\": \"Traceback (most recent call last):\\n  File \\\"/airbyte/integration_code/main.py\\\", line 13, in <module>\\n    launch(source, sys.argv[1:])\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 123, in launch\\n    for message in source_entrypoint.run(parsed_args):\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 111, in run\\n    config_catalog = self.source.read_catalog(parsed_args.catalog)\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\\\", line 54, in read_catalog\\n    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\\n  File \\\"/usr/local/lib/python3.9/site-packages/pydantic/main.py\\\", line 521, in parse_obj\\n    return cls(**obj)\\n  File \\\"/usr/local/lib/python3.9/site-packages/pydantic/main.py\\\", line 341, in __init__\\n    raise validation_error\\npydantic.error_wrappers.ValidationError: 2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\\n\", \"failure_type\": \"system_error\"}}}\n"}}

 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_airbyte_trace_message_on_failure[inputs0] ✓                                                                                                  83% ████████▍ 
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓                                                                                                        88% ████████▊ 
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓                                                                                                     92% █████████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_read_sequential_slices[inputs0] ✓                                                                                                   96% █████████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] ✓                                                                                      100% ██████████
{"type": "LOG", "log": {"level": "INFO", "message": "/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta - SAT run - 59689040345ec87fd6f68a3e3ac60907e37f9eb9 - PASSED"}}

============================================================================================================================= warnings summary =============================================================================================================================
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 17 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py: 3 warnings
  /Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta/.venv/lib/python3.9/site-packages/docker/utils/utils.py:52: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s1 = StrictVersion(v1)

airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 17 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py: 3 warnings
  /Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta/.venv/lib/python3.9/site-packages/docker/utils/utils.py:53: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s2 = StrictVersion(v2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (153.22s):

[YiyangLi]

@alafanechere Hi, since you helped review the PR for custom roles, and permissions is related to customer roles, can you help review this PR? Thanks.

[sajarin]

/test connector=connectors/source-okta

:clock2: connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2749013664 :x: connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2749013664 :bug: https://gradle.com/s/izpfvzlnzze2c

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
=================== 1 failed, 28 passed in 93.67s (0:01:33) ====================

[YiyangLi]

@marcosmarxm thanks for reviewing, I also add some unit tests.

[YiyangLi]

/test connector=connectors/source-okta

[sajarin]

/test connector=connectors/source-okta

:clock2: connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2775486433 :x: connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2775486433 :bug:

[YiyangLi]

/test connector=connectors/source-okta

[sajarin]

/test connector=connectors/source-okta

:clock2: connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2791197502 :white_check_mark: connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2791197502 Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        81      6    93%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  77     17    78%
source_acceptance_test/tests/test_core.py              328    121    63%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  995    261    74%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_okta/__init__.py       2      0   100%
source_okta/source.py       194      3    98%
---------------------------------------------
TOTAL                       196      3    98%

Build Passed

Test summary info:

All Passed

[YiyangLi]

rebased again

[sajarin]

/publish connector=connectors/source-okta

:clock2: Publishing the following connectors:
connectors/source-okta
https://github.com/airbytehq/airbyte/actions/runs/2791510158

Connector	Did it publish?	Were definitions generated?
connectors/source-okta	:x:	:x:

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[sajarin]

/publish connector=connectors/source-okta

[marcosmarxm]

/publish connector=connectors/source-okta

:clock2: Publishing the following connectors:
connectors/source-okta
https://github.com/airbytehq/airbyte/actions/runs/2791989958

Connector	Did it publish?	Were definitions generated?
connectors/source-okta	:white_check_mark:	:white_check_mark:

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️