Compacted topic ... how to get the most recent message?

[HeikoMueller]

I am struggeling with reading from a compacted topic. How would I set up a consumer or a reader that receives the most recent message (published before starting up the client) on a compacted topic? I am only able to receive either all messages or none. I tried settting up the consumer with these options:

{ topic: "persistent://public/default/topic", subscription: 'sub-1', subscriptionType: 'Exclusive', subscriptionInitialPosition: 'Earliest', readCompacted: true, } { topic: "persistent://public/default/topic", subscription: 'sub-1', subscriptionType: 'Exclusive', subscriptionInitialPosition: 'Latest', readCompacted: true, }

...and for the reader: { topic: 'persistent://public/default/topic', startMessageId: Pulsar.MessageId.latest(), readCompacted: true } { topic: 'persistent://public/default/topic', startMessageId: Pulsar.MessageId.eariest(), readCompacted: true }

Am I missing something here?

[merlimat]

Compaction runs in background on a schedule, based on size and time. Perhaps the data was not compacted yet?

[HeikoMueller]

Actually I think everything shall be in place. What I did:

I published some messages to the topic like this: const message = { key: "key-1", data: Buffer.from("hello world x") } producer.send(message)

Then I manually triggered compaction with pulsar-admin topics compact persistent://public/default/topic

Also I reduced the compaction threshold: pulsar-admin topics set-compaction-threshold persistent://public/default/topic --threshold 16

I can see that the topic gets cleared every x messages. At this point, I do not get any initial value back. Then as the message count increases, I see all the new messages come in at re-initalization until the topic reaches the threshold and is cleared again due to compaction process, which again returns null at the initialization of the reader or subscription.

Can you tell if I use the right setup and message format here?