flink-scala-api icon indicating copy to clipboard operation
flink-scala-api copied to clipboard

Savepoint incompatibility between 1.1.4 and 1.1.5

Open IgnasD opened this issue 8 months ago • 15 comments

After upgrading this library from 1.1.4 to 1.1.5 we've noticed that savepoints made while using older version became incompatible with the job containing newer version.

After digging in, I've traced the cause to the PR https://github.com/flink-extended/flink-scala-api/pull/98. Within this PR the serialized format for the case classes was changed to include arity in the blob. Now the deserializer expects arity as the first 4 bytes, however in the pre-1.1.5 format those first 4 bytes contained some field(s) payload. This results in incorrect deserialization.

Are breaking changes to be expected within patch level releases of this library?

To be honest, I'm not really sure how to get out of this. Since we're dealing with streamed input, naive try { newFormat } catch { oldFormat } won't do the trick, at least at the level the issue was introduced in.

IgnasD avatar Jun 03 '24 13:06 IgnasD