Scott Sandre comments

Results 160 comments of


                                            Scott Sandre

[Infra] Separate out delta-spark python tests

@tdas > we don't run python tests for spark master at the moment

2024-08-15 infra test

delta-io/delta#3508

[Bug] Output parsed stats for delta lake tables

@sclmn are you saying that when `delta.checkpoint.writeStatsAsStruct` is true, delta-spark is not writing out the `stats_parsed` field in the delta checkpoint? That seems like a bug. Thanks for pointing this...

[Bug] Output parsed stats for delta lake tables

@prakharjain09 can you take a look?

[BUG] Kernel cannot read partition column of type `ISO8601 formatted timestamp adjusted to UTC`

Hi @peeyushgupta1 -- sorry for the delayed response! Yup we have plans for this to be picked up starting roughly next week!

[Spark] Fix metadata cleanup by retaining files required for log reconstruction

@dhruvarya-db and @felipepessoto -- is this considered a bug or correctness fix? Thats' the only reason we would want to backport it to the 4.0 branch

[PROTOCOL RFC] Catalog-managed Tables

@michelleon -- That's a great suggestion. Will implement that. Thanks!

Not null invariant incorrectly fails on non-nullable field inside a nullable struct

Hi @Kimahriman, thanks for bringing this to our attention. We will look into this.

[Infra] [WIP] Add test report listener to delta-spark and bucket test suites by estimated runtime

Another observation of mine: not sure if our hashing is even/fair enough. The max difference between any two threads in a given shard is 68% (i.e. one thread has 68%...

[Infra] [WIP] Add test report listener to delta-spark and bucket test suites by estimated runtime

![image](https://github.com/user-attachments/assets/427001a0-d81e-41c9-8f36-6c4e0ef98551) 24 minute disparity between first and last