iceberg

iceberg copied to clipboard

How to set Spark conf to use Parquet and Iceberg tables using glue catalog without catalog name(spark_catalog)?

5

### Query engine **Using** ``` AWS EMR 6.10.0 Spark 3.3.1 Iceberg 1.1.0-amzn-0 AWS Glue catalog ``` **Spark-defaults** ``` 'spark.sql.extensions': 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions', 'spark.sql.catalog.spark_catalog': 'org.apache.iceberg.spark.SparkCatalog', 'spark.sql.catalog.spark_catalog.catalog-impl': 'org.apache.iceberg.aws.glue.GlueCatalog', 'spark.sql.catalog.spark_catalog.warehouse': 's3://{MY_BUCKET}', 'spark.sql.catalog.spark_catalog.io-impl': 'org.apache.iceberg.aws.s3.S3FileIO', 'spark.sql.catalog.spark_catalog.lock-impl': 'org.apache.iceberg.aws.dynamodb.DynamoDbLockManager',...

ryu3065

[DeleteManifest] Making file validation optional

9

### Feature Request / Improvement ## Background This issue comes up when we are trying to fix a corrupted table where one manifest file (with active reference to data files)...

haizhou-zhao

Iceberg Spark streaming skips rows of data

6

### Apache Iceberg version 1.5.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 When using spark readStream the option `stream-from-timestamp` is used to position the read...

cccs-jc

Core: Fix create v1 table on REST Catalog

1

This PR fix the problems of `Cannot downgrade v2 table to v1` when creating v1 table on REST Catalog. Fix issue: #8756

hantangwangd

Hive: Add View support for HIVE catalog

8

This changes include: 1. Introduction of common metadata interface(BaseMetadata) for table and view. 2. Refactor for HiveTableOperation to have common code for table and view commits. Ref: https://github.com/apache/iceberg/pull/9682 , https://github.com/apache/iceberg/pull/9461...

nk1506

Hive: Common namespace validation for HiveCatalog

1

Ref: https://github.com/apache/iceberg/pull/9852#discussion_r1516227703

nk1506

REST: assume issued token type is access token

3

The REST client wrongly assumes that the `issued_token_type` field is present in all OAuth responses, but that isn't true: e.g. in the `client_credentials` flow, this field is undefined. See RFC...

adutra

REST: disallow overriding "credential" in table sessions

1

See #10256 for context. As requested, this change disallows overriding the `credential` property in table sessions, by introducing an allow-list of auth-related properties that can be overridden in such situations....

adutra

The responsibility of the Trigger Manager is to start the Maintenance Tasks based on the incoming Table Change messages and prevent overlapping Maintenance Task runs. The event time of the...

pvary

Flink: Pre-create fieldGetters to avoid constructing them for each row

1

I found that RowDataUtil.clone consumes a significant amount of CPU time by analyzing the flame graph. It constructs a FieldGetter object for each field of every row being processed. This...

fengjiajie