Ravi D

I am facing similar issue when trying to write to a delta table previously written to by a Glue PySpark job. I chose to use Glue for one time full...

@ion-elgreco can you please clarify, I am using Glue 4.0 which supports Spark 3.3. From what I see pyspark.sql.functions.timestamp_micros is a new feature in Spark 3.5. Also, I am not...

@ion-elgreco The original value from the source is a string `2024-08-06T16:34:16.000574Z` I did a full load of that data using spark to s3 in delta format: `df = df.withColumn(col, fn.to_timestamp(fn.col(col)))`...

@ion-elgreco OK. sounds like spark 3.5 is required, which doesn't exist in Glue (yet).

Thanks for the pointer @ion-elgreco the following config worked: `SparkSession.config("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")`