Danny Chan

It is as expected because even if the incremental checkpointing is enabled, Flink triggers a full checkpointing every N delta checkpinting, it is not relevent with compaction actually.

That inc/full checkpointing is managed by Flink, in hudi, we do have an option 'index.ttl' to control the liveness of the index items but it is not suggested because that...

It's the mapping from hoodie record key to location, for a location it is comprised by a partition path and file group id.

you can choose bucket index, bucket index does not support updates among multiple partitions and the bucket number can not scale well if it not consistent hashing.

We have support for consistent hashing index which can scales the bucket number automically.

It should work if the payload are not merged by the writer, otherwise the writer just takes the onus of merging.

Did you try to add filter condition with the partition fields?

> but I want a config that can tell source that only reads the partition that in my configs so I do not need to use filter That does not...

Currently, you should stop the streaming job and execute the alter table cmd with spark then restart the job.

No automatic schema evolution for streaming writer now, the limitation is from the Flink engine, the Flink table API already assumes constant schema for all the records there, so for...