amazon-managed-service-for-apache-flink-examples
amazon-managed-service-for-apache-flink-examples copied to clipboard
Iceberg sql json
Add Iceberg SQL Examples with Glue and S3 Tables Catalogs
Purpose of the change
Add two new examples demonstrating how to write streaming data to Iceberg tables using SQL operations with different catalog implementations:
- Using AWS Glue Data Catalog
- Using Amazon S3 Tables
Both examples showcase implementation of streaming stock price data to Iceberg tables using Flink SQL, providing users with patterns for both Glue and S3 Tables integration.
Verifying this change
Local Testing
Tested both examples locally in IntelliJ:
- Configure
flink-application-properties-dev.jsonwith appropriate S3 bucket/ARN - Run the main application class
- Verified data generation and table creation in both Glue Data Catalog and S3 Tables
- Confirmed data writing through SQL queries
Example output from Glue example:
mysql> SELECT * FROM iceberg.stock_data LIMIT 5;
+--------+---------+-------------------------+
| price | ticker | eventtime |
+--------+---------+-------------------------+
| 45.23 | AAPL | 2025-02-07 11:00:15.0 |
| 89.67 | AMZN | 2025-02-07 11:00:15.2 |
| 67.89 | MSFT | 2025-02-07 11:00:15.4 |
| 34.12 | INTC | 2025-02-07 11:00:15.6 |
| 78.90 | TBV | 2025-02-07 11:00:15.8 |
+--------+---------+-------------------------+
Significant changes
- [x] Completely new example
- [ ] Updated an existing example to newer Flink version or dependencies versions
- [ ] Improved an existing example
- [ ] Modified the runtime configuration of an existing example
- [ ] Modified the expected input or output of an existing example
New features:
- Flink SQL operations with Iceberg
- Integration with AWS Glue Data Catalog
- Integration with Amazon S3 Tables
Dependencies:
- Flink 1.19.0
- Iceberg 1.6.1
- Java 11
- AWS SDK v2
- S3 Tables Catalog 0.1.3