Harshal Sheth

docs(ingest): add airflow docs that use the `PythonVirtualenvOperator`

2

New docs look like this: ## Checklist - [ ] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly [Commit Message Format](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md#commit-message-format)) - [ ] Links to related issues (if applicable)...

feat(ingest): run profiler in more cardinality cases

2

- Run full profiling on ints, regardless of cardinality - ~Run min/max profiling on strings~ See https://datahubspace.slack.com/archives/CUMUWQU66/p1668487482130239?thread_ts=1667551488.446339&cid=CUMUWQU66 ## Checklist - [ ] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly...

ci(ingest): test with python 3.11

2

Currently blocked on https://github.com/googleapis/python-bigquery-sqlalchemy/issues/500. ## Checklist - [ ] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly [Commit Message Format](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md#commit-message-format)) - [ ] Links to related issues (if applicable) -...

Redshift IAM support

1

We should be able take the user's IAM credentials and call `redshift:GetClusterCredentials` to get the credentials dynamically. As per https://stackoverflow.com/questions/44243169/connect-to-redshift-using-python-using-iam-role, the AWS-native redshift_connector driver has support for this built in,...

fix(ingest): issue deprecation warning correctly

2

## Checklist - [ ] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly [Commit Message Format](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md#commit-message-format)) - [ ] Links to related issues (if applicable) - [ ] Tests for...

Changes stacked on top of https://github.com/datahub-project/datahub/pull/9867 - Adds a process-based timeout decorator - Calls `traceback.clear_frames` to maybe fix a memory leak in exception reporting ## Checklist - [ ] The...

## Checklist - [ ] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly [Commit Message Format](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md#commit-message-format)) - [ ] Links to related issues (if applicable) - [ ] Tests for...

## Checklist - [ ] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly [Commit Message Format](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md#commit-message-format)) - [ ] Links to related issues (if applicable) - [ ] Tests for...

Changes stacked on top of https://github.com/datahub-project/datahub/pull/9860 ## Checklist - [ ] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly [Commit Message Format](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md#commit-message-format)) - [ ] Links to related issues (if...

## Checklist - [x] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly [Commit Message Format](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md#commit-message-format)) - [ ] Links to related issues (if applicable) - [ ] Tests for the...