🎉 New source: Primetric

[andrzejdackiewicz]

What

• Added a new source-primetric connector. Now we will be able to use Primetric as a data source.
• Changes should not affect frontend

How

*Added a new source connector

Recommended reading order

1. y.python

🚨 User Impact 🚨

These changes should not break any existing features of Airbyte.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• [ ] Community member? Grant edit access to maintainers (instructions)
• [ ] Secrets in the connector's spec are annotated with airbyte_secret
• [ ] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• [ ] Code reviews completed
• [ ] Documentation updated
  • [ ] Connector's README.md
  • [ ] Connector's bootstrap.md. See description and examples
  • [ ] docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • [ ] docs/integrations/README.md
  • [ ] airbyte-integrations/builds.md
• [ ] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• [ ] Create a non-forked branch based on this PR and test the below items on it
• [ ] Build is successful
• [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
• [ ] /test connector=connectors/<name> command is passing
• [ ] New Connector version released on Dockerhub by running the /publish command described here
• [ ] After the connector is published, connector added to connector index as described here
• [ ] Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• [ ] Grant edit access to maintainers (instructions)
• [ ] Secrets in the connector's spec are annotated with airbyte_secret
• [ ] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• [ ] Code reviews completed
• [ ] Documentation updated
  • [ ] Connector's README.md
  • [ ] Connector's bootstrap.md. See description and examples
  • [ ] Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• [ ] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• [ ] Create a non-forked branch based on this PR and test the below items on it
• [ ] Build is successful
• [ ] If new credentials are required for use in CI, add them to GSM. Instructions.
• [ ] /test connector=connectors/<name> command is passing
• [ ] New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• [ ] Issue acceptance criteria met
• [ ] PR name follows PR naming conventions
• [ ] If adding a new generator, add it to the list of scaffold modules being tested
• [ ] The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• [ ] Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

[CLAassistant]

All committers have signed the CLA.

[andrzejdackiewicz]

Hello @Nmaxime Sorry I have a problem with acceptance tests. I was trying to find the reason but perhaps you know and can help me? integration_tests_errors.txt Seems there is a problem reading the connector config file.

[Nmaxime]

Hello @Nmaxime Sorry I have a problem with acceptance tests. I was trying to find the reason but perhaps you know and can help me? integration_tests_errors.txt Seems there is a problem reading the connector config file.

Here is the catch! We did not find the airbyte/source-primetric:latest image for this connector. This probably means this version has not yet been published to an accessible docker registry like DockerHub. It might be a typo while tagging the docker image (airbyte/source-primetric:latest)

[andrzejdackiewicz]

@Nmaxime This is how my docker images look like now: Both tags are referring to the same image. So i think we should be good. I am still getting errors. Everywhere in code i refer to :dev image. integration_tests_errors.txt Can You take a look?

[andrzejdackiewicz]

@Nmaxime Hello, I am having a problem with some of the acceptance tests. I do not quite understand what I am doing wrong. Can You help me? I was trying different things for some time. Today I have a bit less time for looking at it, I will try to search some more in around 8h from now. Is there a chance that you check if acceptance tests pass for you / can we have a call where we check what to do together? Is it fine if I write directly to you on Slack?

[sajarin]

Hey @andrzejdackiewicz thanks for the PR. We're currently a bit blocked since we don't have a sandbox account for this new source yet on our side. I believe we're in the process of obtaining credentials but we don't need to wait for them before merging this PR since it is a new PR.

We do need to see proof that the acceptance tests are successfully passing on your end.

Could you run the following commands in the airbyte/airbyte-integrations/connectors/source-primetric directory?

docker build . -t airbyte/source-primetric:dev
python -m pytest -p integration_tests.acceptance

Post the output of the above commands. Let me know if you have any questions! We're so close to closing this PR out :)

[andrzejdackiewicz]

Hello @sajarin @Nmaxime I am still having an issue with passing some of the acceptance tests. integration_tests_errors.txt docker.errors.ContainerError: Command 'discover --config /data/tap_config.json' in image '<Image: 'airbyte/source-primetric:dev'>' returned non-zero exit status 1: Seems like some config can not be found. I do not know what that tap_config.json is. I did not find anything about it in the docs. I showed also how the acceptance-test-config.yml looks like and what is the location of my secrets file.

I am able to perform: python3 main.py check --config secrets/config.json python3 main.py spec python3 main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json Everything is fine, i am able to read data with read. The same for using docker.

Can you help me understanding what is the issue. I spent some time trying to find a solution but did not find much.

[andrzejdackiewicz]

Hello!!! @Nmaxime @sajarin I found a bit of time to look for a solution to my problem. Finally I was able to run acceptance tests and was no longer running into a problem with /data/tap_config.json

Also I found that one of the Streams (Capacities) was removed from the API. Had to remove it from my code. Also because I could only write Full_refresh synchronization and I have quite a bit of data on my Primetric account the api would throw a backoff error because of too many requests in short time. Because of that I had to remove the more populated streams from my configured_catalog.json in acceptance testing so that the error does not occur. Otherwise Airbyte testing would treat it as a serious error and the test would fail. Is it ok to have not a full list of streams in acceptance configured_catalog ? Also One of the types of data had to be replaced from number to big number. Hope these changes are ok with you :pray:

Here is a screenshot with acceptance-tests passing.

I also checked unit tests:

And I rerun my export on my Airbyte instance. Everything looks good :smile_cat:

[andrzejdackiewicz]

@Nmaxime I made a change for backoff_time as you suggested and the acceptance tests succeeded for whole configured_catalog.json with all streams :smile_cat: I needed to adjust timeout_seconds in acceptance-test-config to have enough time for full refresh and basic reading. I updated the unit test since we have backoff_time and now everything looks fine! Thanks for help! I think we are good to go :smiley_cat:

[andrzejdackiewicz]

Hurray! Thanks @Nmaxime @sajarin for help with PR. Should I now update the branch with changes from master and merge into master or further steps with adding this to master and deploying to airbyte will be done by somebody else at some time? When can I expect these changes to appear in official Airbyte :smiley_cat:

[sajarin]

/publish connector=connectors/source-primetric run-tests=false

:clock2: Publishing the following connectors:
connectors/source-primetric
https://github.com/airbytehq/airbyte/actions/runs/3091242879

Connector	Did it publish?	Were definitions generated?
connectors/source-primetric	:x:	:x:

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[Nmaxime]

@andrzejdackiewicz , I forgot the formatting. You need to run ./gradlew :airbyte-integrations:connectors:source-primetric:airbytePythonFormat to format the code (doc).

The error is raised by Flake : https://github.com/airbytehq/airbyte/actions/runs/3091242879/jobs/5001111540#step:14:783

We're almost done 🙂.

[andrzejdackiewicz]

@Nmaxime Thanks! I must have forgotten to do that, sorry. I just added formatting changes. Now we should be good :smiley:

[Nmaxime]

/publish connector=connectors/source-primetric run-tests=false

[sajarin]

/publish connector=connectors/source-primetric run-tests=false

:clock2: Publishing the following connectors:
connectors/source-primetric
https://github.com/airbytehq/airbyte/actions/runs/3098654755

Connector	Did it publish?	Were definitions generated?
connectors/source-primetric	:x:	:x:

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[sajarin]

@andrzejdackiewicz, the unit tests are failing. Unit tests should not rely on secrets, you should construct a mock for the config and test that the connection fails. For the other tests that are failing, consider looking at recently merged connectors and see how they handled the unit tests. We're almost there!

[andrzejdackiewicz]

@Nmaxime @sajarin changed unit tests to use mocking. All unit tests pass now without using secret file.

[sajarin]

/publish connector=connectors/source-primetric run-tests=false

:clock2: Publishing the following connectors:
connectors/source-primetric
https://github.com/airbytehq/airbyte/actions/runs/3100724329

Connector	Did it publish?	Were definitions generated?
connectors/source-primetric	:white_check_mark:	:white_check_mark:

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[sajarin]

Thanks for the PR @andrzejdackiewicz and thanks for the review @Nmaxime!

[andrzejdackiewicz]

@sajarin @Nmaxime Thank You for help with the PR! We finally made it :champagne: Thanks for being patient with me :laughing:

[rach-r]

Finally I was able to run acceptance tests and was no longer running into a problem with /data/tap_config.json

Hello @andrzejdackiewicz, I am having the exact same issue for a connector that I'm currently building. Do you mind sharing what was your workaround? 🙏

[andrzejdackiewicz]

Hi @rach-r ! To be honest even after solving the problem I did not know what was causing it. But instead of running the acceptance test from command line I tried to do the same from my Intellij. For some unknown to me reason that worked fine :shrug: Perhaps it will work for You as well? Let me know if it helps :smiley: