amazon-managed-service-for-apache-flink-examples icon indicating copy to clipboard operation
amazon-managed-service-for-apache-flink-examples copied to clipboard

Iceberg sql json

Open jeremyber-aws opened this issue 8 months ago • 0 comments

Add Iceberg SQL Examples with Glue and S3 Tables Catalogs

Purpose of the change

Add two new examples demonstrating how to write streaming data to Iceberg tables using SQL operations with different catalog implementations:

  1. Using AWS Glue Data Catalog
  2. Using Amazon S3 Tables

Both examples showcase implementation of streaming stock price data to Iceberg tables using Flink SQL, providing users with patterns for both Glue and S3 Tables integration.

Verifying this change

Local Testing

Tested both examples locally in IntelliJ:

  1. Configure flink-application-properties-dev.json with appropriate S3 bucket/ARN
  2. Run the main application class
  3. Verified data generation and table creation in both Glue Data Catalog and S3 Tables
  4. Confirmed data writing through SQL queries

Example output from Glue example:

mysql> SELECT * FROM iceberg.stock_data LIMIT 5;
+--------+---------+-------------------------+
| price  | ticker  | eventtime              |
+--------+---------+-------------------------+
| 45.23  | AAPL    | 2025-02-07 11:00:15.0 |
| 89.67  | AMZN    | 2025-02-07 11:00:15.2 |
| 67.89  | MSFT    | 2025-02-07 11:00:15.4 |
| 34.12  | INTC    | 2025-02-07 11:00:15.6 |
| 78.90  | TBV     | 2025-02-07 11:00:15.8 |
+--------+---------+-------------------------+

Significant changes

  • [x] Completely new example
  • [ ] Updated an existing example to newer Flink version or dependencies versions
  • [ ] Improved an existing example
  • [ ] Modified the runtime configuration of an existing example
  • [ ] Modified the expected input or output of an existing example

New features:

  • Flink SQL operations with Iceberg
  • Integration with AWS Glue Data Catalog
  • Integration with Amazon S3 Tables

Dependencies:

  • Flink 1.19.0
  • Iceberg 1.6.1
  • Java 11
  • AWS SDK v2
  • S3 Tables Catalog 0.1.3

jeremyber-aws avatar Feb 07 '25 17:02 jeremyber-aws