koombal

well, i tried explicitly to set it to false on the client side (Spark) but to the best of my understanding it is returned and determined by the REST catalog...

it's important to note that the same app is running on **Azure** with: `ADLSFileIO` without any issues. The issue is only occurring in **AWS** with `S3FileIo `

@nastra it's the same error for vended credentials in S3. no issue in Azure. could be related to S3FileIo?

i will just fix my comment before - in Azure we didn't try doing catalog authentication, we were doing against azure blob storage

@c-thiel @nastra we are running 1. spark 3.5_2.12 (java). 2. Lakekeeper catalog 3. K8S 4. configuring hadoop with s3a and creating the checkpoints successfully using hadoop 5. Iceberg 1.8.1 -->...

@nastra ``` builder.config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog") .config("spark.sql.catalog.iceberg.type", "rest") .config("spark.sql.catalog.iceberg.uri", icebergUri) .config("spark.sql.catalog.iceberg.warehouse", icebergWarehouseName); builder.config("spark.sql.adaptive.enabled", "true").config("spark.sql.adaptive.skewJoin.enabled", "true"); builder.config("spark.sql.timestampType", "TIMESTAMP_NTZ"); builder.config("spark.sql.catalog.iceberg.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") .config( "spark.sql.catalog.iceberg.client.region", s3ConfigurationProperties .getGeneralConfigurationProperties() .getRegion()) .config("spark.sql.catalog.iceberg.s3.endpoint", s3ConfigurationProperties.getSparkUrl()) .config("spark.sql.catalog.iceberg.s3.path-style-access", "true"); builder.config("spark.hadoop.fs.s3a.access.key", s3ConfigurationProperties.getAccessKey()); builder.config("spark.hadoop.fs.s3a.secret.key", s3ConfigurationProperties.getSecretKey());...

@nastra s3a. the checkpoint is created successfully before. ` log.info("Ensuring checkpoint directory {}", ckptDir); FileSystem fs = FileSystem.get(new URI(ckptDir), spark.sparkContext().hadoopConfiguration()); Path dir = new Path(ckptDir); if (!fs.exists(dir)) { log.info("Checkpoint directory...