Observable message loss is undocumented

[wfhartford]

trafficstars

I have a jet batch job which produces a large number of results (several millions). The sink in this job is an observable, which is used to collect and deliver the results. In the course of job execution, I get a large number of warnings logged along the lines of

Message loss of 1000 messages detected in listener 'afe1cde4-4725-4c74-99ba-8f8eb1309ac1/__jet.observables.4f713bed-ea30-4095-a295-f2c75afc26de'

A brief look through the observable code has lead me to believe that there is no back-pressure mechanism, so this situation is expected when a large number messages are produced to an observable very quickly.

While this unreliable delivery system is a reasonable design decision, it does not appear to be documented in any way, I believe it should be.

[frant-hartm]

Hi @wfhartford , thanks for reporting this.

There is a note in the documentation for the observable sink:

Jet internally uses Hazelcast's Ringbuffer to create a temporary buffer to write the results into and these are then fetched by the client:

It's worth noting that Ringbuffer may lose events, if they are being produced at a higher-rate than the clients can consume it. There will be a warning logged in such cases. You can also configure the capacity using the setCapacity() method on the Observable.

https://jet-start.sh/docs/api/sources-sinks#observables

We can add this info to javadoc as well, is there any other place where you would expect to see this info?

[wfhartford]

Ahh, thanks, I missed that part of the docs. I tend to stick to Javadocs as I'm coding and use the docs you linked to more as a general guide to point me in the right direction. I would have expected to see a note about losing events in the Observable interface javadoc, probably in the second or third paragraph.