CDK Example for AWS Glue Workflow including Glue Jobs, Triggers, Crawlers, securtiy group and Database

[PuneetBabbar]

trafficstars

:rocket: Feature Request

General Information

• [X] :wave: I want to implement this feature request

• [ ] :warning: This feature might incur a breaking change

Description

This example will be like another example in the REPO. This new feature request will add another example and will cover everything around AWS Glue, containing AWS Glue workflow, Glue Jobs, Triggers, Crawlers, security group and Database.

This will enhance and add new feature. As currently there is no code example available online and also no AWS documentation with code example to demonstrate on how can can use CDK to setup an ecosystem around Glue.

Proposed Solution

The idea is to build an example that will setup an ETL pipeline using tool set available in Glue ecosystem, and implement or construct it via complete CDK code.

I was thinking to open source data and build an example for a Data ETL job, something similar to blog like https://aws.amazon.com/blogs/devops/provision-codepipeline-glue-workflows/ or something similar. But the idea is to build an ETL pipeline via workflow. And have steps constructed via CDK

1. Glue Crawler to catalog S3 data.
2. Glue Jobs (Spark) to process and transform the catalog data
3. Glue Trigger for calling the above Crawler and Jobs
4. Glue Workflow to orchestrate the above components.

I don't have the diagram for the workflow, but can create and will be able to add to the example for better understanding.

Environment

• CDK Version: 1.102.0:
• Example : GLUE Workflow
• Example Version: N/A
• OS: Ubuntu
• language: All

Other information

[PuneetBabbar]

@NGL321 can this be assigned to me ? So can start working on this. thanks

[NGL321]

We assign internally to the team for who is supposed to shepherd the process. But please feel free to take this on yourself!

Quick heads-up that we are backed up on reviewing contributions to this repo due to some considerations about how to grow/develop it, but if you contribute a PR we will review it as able!

For my part, I think a well-written Glue example would be very useful.

😸 😷

[maddy2u]

@PuneetBabbar - Are you coming up with this? Would be very useful

[kaiz-io]

Glue alpha construct would be the answer. Provides examples in the docs.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.