examples
examples copied to clipboard
Published
20 hours ago •
decodableco
decodableco
This repository will contain examples of use cases that utilize Decodable streaming solution
Decodable Examples
This repository contains examples of use cases that utilize Decodable streaming solution as well as demos for related open-source projects such as Apache Flink, Debezium, and Postgres.
Contents
| Example | Description |
|---|---|
| Flink Learn | Apache Flink tutorials and webinar |
| AsyncAPI | Publishing Data Products with AsyncAPI |
| Opinionated Data Pipelines | Building data pipelines with schema on write streams. |
| Postman | Building data pipelines with Postman. |
| Change Streams | Using change streams to build materialized views in Postgres |
| XML Processing | Parse XML and transform to JSON |
| OSQuery Routing | Route OSQuery logs with SQL |
| Masking | Ways to mask data |
| Apache Pinot | Transforming osquery logs to Apache Pinot and Superset |
| Apache Druid | This example sends covid 19 data to Decodable using it's REST API. The data is then cleansed using Decodable SQL and send the data to a Kafka sink. |
| Rockset | We will be utilizing a cloud MQTT broker and AWS Kinesis to capture and stream data. Decodable will be responsible for preparing and aggregating the data prior to reaching the real-time analytical database (Rockset) |
| Tinybird | We write data to Tinybird and build a simple real time web application. |
| Apache Kafka | Installing Apache Kafka on EC2 and writing to S3 with Decodable |
| Apache Kafka mTLS | We install Apache Kafka on EC2 and configure it with mTLS and configure Decodable to read from it |
| Snowflake + Snowpipe | We setup a snowpipe at the end of a Decodable S3 sink. |
| Confluent | Clickstream from Confluent Cloud joined with CDC user data from Postgres |
| Snowflake + Snowpipe + Merge | Leveraging Snowpipe, we send CDC data from Postgres to be processed in Snowflake using an Append Only Stream in Snowflake to merge CDC data in a Snowflake table. Essentially mirroring the table in Postgres in a Snowflake table. |
| Reading S3 Events in a Lambda Function | We configure an S3 bucket with a Lambda notification to send data to Kinesis to be processed in Decodable |
| MSSQL CDC | We enable a MSSQL in Docker with CDC. We then stand up a Debezium server to read from MSSQL and write the change events into AWS Kinesis |
| Oracle CDC | We configure a Oracle AWS RDS with LogMiner. We then stand up a Debezium server to read change events into AWS Kinesis |
| DynamoDb CDC | We configure a DynamoDB to send change data to Kinesis. Then we read those changes into Decodable for transformation or replication. |
| Logical Decoding Message Examples | We show how to retrieve logical decoding messages from the Postgres WAL |
| GitHub Webhooks | We show how to process GitHub Webhook events using the Decodable REST source connector |
| PyFlink | We run a basic PyFlink job on Kubernetes |
| Kafka / Flink / Iceberg | Integrating Apache Kafka with Apache Iceberg through Apache Flink. As presented at Kafka Summit London 2024 |
| Kafka / Flink / Iceberg (with Decodable) | Streaming from Apache Kafka to Apache Iceberg with Decodable |
| Flink SQL Troubleshooting | A set of Docker Compose environments for demonstrating various Flink SQL troubleshooting scenarios (see related blog) |
| Array Aggregation | Using the array_agg() UDF for denormalizing data in a pipeline from MySQL to OpenSearch |
| Kafka with ngrok | Docker Compose for running Apache Kafka locally, accessible from the internet using ngrok |
| PyFlink on Decodable | Running a PyFlink job as a Custom Pipeline on Decodable |
License
This code base is available under the Apache License, version 2.
Metadata
50
Stars
24
Forks
Watchers
Owner
decodableco
Metadata
This repository will contain examples of use cases that utilize Decodable streaming solution