4.0.2 Release Notes

[yiguolei]

New Feature

AI & Search

• Inverted index supports custom analyzers, including Pinyin tokenizer and Pinyin filter (#57097)
• Added support for multi-position PhraseQuery in inverted index search functions (#57588)
• Added Ann index only-scan capability (#57243)

Function

• Added the sem aggregate function (#57545)
• Supported the factorial simple SQL function derived from Hive (#57144)
• Added support for zero-width assertions in some regular expression functions (#57643)
• Enabled GROUP BY and DISTINCT operations for JSON type (#57679)
• Added the add_time/sub_time time functions (#56200)
• Added the deduplicate_map function (#58403)

Materialized View (MTMV)

• Materialized views can still participate in transparent query rewrite when data changes occur in their non-partitioned base tables (#56745)
• Supported creating MTMV based on views (#56423)
• MTMV refresh supports multiple PCT tables (#58140)
• Supported window function rewrite when materialized views contain window functions (#55066)

Data Lake

• Implemented the Iceberg rewrite_data_files action to support table optimization and compaction (#56413, #56638)
• Supported VARBINARY type mapping for Hive/Iceberg/Paimon/JDBC tables (#57821, #58215)
• Supported Partition Evolution DDL for Iceberg tables (#57972)

Optimizations

• Optimized the performance of the FROM_UNIXTIME function (#57423)
• Removed the castTo conversion in PartitionKey comparison to improve partition processing efficiency (#57518)
• Optimized the performance of Parquet reader when decoding RLE_DICTIONARY encoding (#57208)
• Reduced the memory footprint of the Column class in Catalog (#57401)
• Accumulated multiple small batches before Ann index training to improve training efficiency (#57623)

Bug Fixes

Query

• Fixed the issue where the utc_time function returned incorrect results when the input was null (#57716)
• Fixed the exception thrown when UNION ALL is combined with TVF (#57889)
• Fixed the problem that the WHERE clause contained non-key columns when creating a materialized view on a unique key table (#57915)
• Fixed window functions: enabled constant expression evaluation for the offset parameter of LAG/LEAD (#58200)
• Fixed aggregate functions: abnormal push-down of aggregate operations before projection on nullable columns; count push-down aggregation issue on non-null columns (#58234)
• Fixed time functions: the second/microsecond functions did not handle time literals; time_to_sec reported errors due to garbage values when processing null values (#56659, #58410)
• Fixed AI functions: unknown error occurred when _exec_plan_fragment_impl called AI functions (#58521)
• Fixed geo module: memory leak in the geo module (#58004)
• Fixed information_schema: timezone format incompatibility when using offset timezone (#58412)

Materialized View and Schema Change

• Fixed the failure of rewrite when materialized views contain group sets and filters above scan (#57343)
• Fixed the coredump issue caused by reading non-overlapping segments from a single rowset during heavy schema change (#57191)

Storage-Compute Separation

• Fixed the issue of broadcast remote read in TopN queries (#58044)
• Fixed the accumulation of tablet deletion tasks in the cloud environment (#58131)
• Fixed the problem of long service startup time during the first boot in the cloud environment (#58152)

Data Lake

• Fixed Iceberg: enabled dynamic partition pruning only for identity partitions (#58033)
• Fixed the permission authentication issue when loading Iceberg partitions (#57988)
• Fixed the partition path scheme mismatch when inserting into Hive partitioned tables on object storage (#57973)
• Fixed the issue where Hive cache was not refreshed when inconsistent (#58074)
• Fixed Paimon Catalog: OSS access failure when using DLS endpoint (#58099)
• Fixed Iceberg: FE did not refresh logs after ALTER TABLE ... EXECUTE; enabled dynamic partition pruning only for identity partitions; added auth regression tests for Iceberg system tables (#58355, #58033, #58298)
• Fixed Hive: StackOverflowError caused by insert overwrite on S3-compatible storage; Hive cache not refreshed when inconsistent; partition path scheme mismatch (#58504, #58074, #57973)
• Fixed Paimon: OSS access failure under DLS endpoint; supported user-defined S3 config prefixes and unified to HDFS S3A protocol (#58099, #57116)

[plastikman]

Will this release support https://github.com/apache/doris/pull/57219 This PR was tagged to be done (it looks like) in 4.0.1 but its not supported because it was not built with the correct flags.