Glue catalog create table doesn't work with schema with required columns

[lliangyu-lin]

Apache Iceberg version

main (development)

Please describe the bug 🐞

Current Glue catalog create table implementation simply use Glue Create Table API to create the table in Glue data catalog and use the metadata file created by Glue.

However, Glue Create Table API doesn't have a way to check or specify if a column is required in a schema, hence, all columns in the table metadata file created by Glue is an optional column.

Checking iceberg-python and iceberg-rust, both clients have implementations that creates the metadata file using FileIO and then creates the table using that metadata file created.

• https://github.com/apache/iceberg-rust/blob/main/crates/catalog/glue/src/catalog.rs#L396-L399
• https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/glue.py#L451