Field comments are not written for timestamp field

[itaise]

We are trying to write field comments using spark, and comments are not written for timestamp (without tz) fields. For timestamp with timezone fields comments are written.

Here is a minimal example that reproduces the issue:

from pyspark.sql import SparkSession

spark = SparkSession.builder.master('local[1]').appName('example') \
    .config("spark.sql.iceberg.use-timestamp-without-timezone-in-new-tables", "true") \
    .getOrCreate()

field_metadata = {'comment': '{"is_test": true}'}

df = spark.sql("select CAST(1000 AS TIMESTAMP)")

df = df.select([df[col_name].alias('some_field', metadata=field_metadata) for col_name in df.columns])

spark.sql(f"use iprod")  # catalog
spark.sql(f"CREATE SCHEMA IF NOT EXISTS iprod.test_schema")

df.write.mode("overwrite").format("iceberg").saveAsTable("iprod.test_schema.timestamp_example")

When the config (use-timestamp-without...) is set to true, the data type is Timestamp, but without the field comments. When it is set to False - The timestamp is with time zone and with field comments.

(Spark version is 3.1.2)

Thanks a lot!

[itaise]

@rdblue mentioned that field comments are supported for timestamps. Might be an issue with this config?

[itaise]

@rdblue we will be very grateful to get your input on this. thank you!

[itaise]

@rdblue can you or anyone else advise regarding that? thank you!

[rdblue]

@itaise, it looks like you're saving the table as Parquet, not Iceberg. That could be a problem.

[itaise]

Thanks you @rdblue, I tried with "iceberg" instead of parquet, and got the same results. We would like to write the field comments to "timestamp without tz" field, any other suggestion?

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[ariksa1]

Hi, there is any update regarding the issue?

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions[bot]]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'