ZENOTME

Tracking: Support partition write

2

> BTW, do we prepare to support partition in 0.1 veriosn? It seems the whole API can build without partition. >> Yes, it's necessary for practical use. As discussed, support...

discussion: use order_float instead of f32,f64

7

To support partition write, we need something like `HashMap`. So this means that we need to impl Eq for AnyValue. To do this, we need to replace f32, f64 to...

split in_memory module like on_disk

3

I find that in_memory.rs is so long, do we consider to split into manifest_file, manifest_list like on_disk? cc @liurenjie1024 @Xuanwo

solve #501 This is an initial PR for #501, I hope it can be a starting point to discuss what's going on if we simplify PrimitiveLiteral. For now, something still...

Implement nan_value_counts && distinct_counts metrics in parquet writer

3

For parquet writer, we still miss following field in DataFile. - nan_value_count - distinct_counts

Enhancement: refine the reader interface

13

Hi, I find that in some cases our reader interface seems redundant for me. E.g. ``` let table_scan = table .scan() .with_batch_size(Some(self.batch_size)) .select(self.schema.names()) .predict(predict) .build() .map_err(BatchError::Iceberg)?; let file_scan_stream = table_scan.plan_files();...

Difference in performance between the default and explicit presentation specifiers in format string compilation

1

Format Specification can used to constrain the type safety but I find that when format using Format Specification, there is a huge decline in performance. Is there way to avoid...

FMT_INLINE works differently than using `__attribute__((always_inline))` directly. ## Reproduce cmake: cmake -DCMAKE_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Release .. gcc: c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 ``` int main() { auto res = fmt::format(FMT_COMPILE("{} {} {}"), 1,...

Add apply interface in transaction

9

> This will work for now but might get problematic later on. Just a heads up. > An important concept for Iceberg is to stack snapshots in a single commit....

Should we use to_avro instead of `RawLiteral` indirection layer

7

BTW, why to we need the `RawLiteral` indirection layer, instead of having sth like ``` impl Literal { fn to_avro(&self) -> avro::Value {...} } ``` _Originally posted by @xxchan in...