PIP-221: Make TableView support read the non-persistent topic

[Demogorgon314]

Motivation

Currently, the table view only supports persistent topics with read compacted.

However, some data don't require persistent storage, like load data in PIP-192 #16691.

We need to support non-persistent topic read for table view and introduce the TTL to delete the old data. The TTL is based on the message publish time.

Goal

• Support non-persistent topics for table view
• Add TTL for persistent and non-persistent topic data in the table view.

API Changes

public interface TableViewBuilder<T> {
// ...

    /**
     * Set key to expire after written TTL. 
     * If not set, the key will never expire unless received the tombstone(The value is null) message.
     *
     * @param ttl The key time to live.
     * @param unit The ttl unit.
     * @return The {@link TableViewBuilder} builder instance
     */
    TableViewBuilder<T> ttl(int ttl, TimeUnit unit);
}

Implementation

The data can be stored in a caffeine cache. We can easily use expireAfter API to add TTL support.

Alternatives

N/A

Anything else?

No response

[codelipenghui]

@Demogorgon314 It's a little confusing if the TTL can only apply to the table view that is based on the non-persistent topic. Or can it also apply to the persistent topic?

And the TTL is based on the message publish time or the time arrive the consumer.

[Demogorgon314]

@codelipenghui I have changed the PIP describe base on your comment. Please take a look again! Thanks.

[codelipenghui]

@Demogorgon314 LGTM

[eolivelli]

I don't think that TTL is related to supporting non-persistent topics.

Maybe the TTL feature is worth a PIP, but supporting non-persistent topics is like a small enhancement (or completing the TableView) feature to me and it doesn't need a PIP, as it doesn't need API changes

[Demogorgon314]

@eolivelli Yes, I can change this PIP title to Make TableView support TTL.

[heesung-sohn]

Discussion Email Thread: https://lists.apache.org/thread/jq8dr965vqox52f5n6h9jt2npnwkxrfn

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[heesung-sohn]

@Demogorgon314 Should we close this PIP if we decide not to work on this?