cccs-jc

@code-asher would like to get more details on enabling jupyter notebooks. Thanks.

make sure to set `stream-from-timestamp` to the current time. Then start a streaming job (let's call it the writer) that writes every minute. The other streaming job consumes that Iceberg...

@singhpk234 any progress on this issue ?

The issue is that when a streamy query resumes (either it was killed, died, gracefully stopped) it does not resume where it left off but rather resumes based on the...

@singhpk234 best way to reproduce the issue is to run a streaming job which sets `stream-from-timestamp` to the current time. The job will read the latest data. You then stop...

@singhpk234 are you able to reproduce the issue ?

in your test case, re-create the readStream from scratch. This will set the .option(SparkReadOptions.STREAM_FROM_TIMESTAMP, System.currentTimeMillis()) to the current time. I think right now the readStream uses the "old" timestamp. so...

@singhpk234 does my previous explanation make sense ?

@singhpk234 This is still an issue which we have to work around and make sure to communicate to all our users who use Iceberg streaming spark jobs. Any chance you...

Seems like it would. I'm not a reviewer but I do want to the fix :-)