Consider catalog file when populating a schema message

[agrandotech]

Description of change

In the current implementation, the catalog.json being passed is not considered for the creation of the schema message. The schema always matches the schema in the schema json files within the repository. This leads to the following issues:

• Fields that are not selected will still stay in the schema
• Fields provided by the catalog file are not present in the schema at all

We encountered this issue when we synced to a BigQuery target. The table created did not match the data in the records.

Manual QA steps

• Run the tap in sync mode without providing a catalog.json file -> it should use the default schema and populate this in the schema message
• Run the tap in sync mode with a catalog.json file where you set certain fields to "selected": false -> those fields should not show up in the schema message
• Run the tap in sync mode with a catalog.json file that contains fields that are not part of the default schema (e.g. custom jira fields) -> those fields should show up in the schema message

Risks

• Existing implementations might show different behavior in case they provided a catalog.json file where certain fields are unselected

Rollback steps

• revert this branch

[singer-bot]

Hi @agrandotech, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

[singer-bot]

You did it @agrandotech!

Thank you for signing the Singer Contribution License Agreement.

[edgarrmondragon]

I guess it's an old PR, but would the maintainers interested in getting this merged?