flink-scala-api
flink-scala-api copied to clipboard
Savepoint incompatibility between 1.1.4 and 1.1.5
After upgrading this library from 1.1.4 to 1.1.5 we've noticed that savepoints made while using older version became incompatible with the job containing newer version.
After digging in, I've traced the cause to the PR https://github.com/flink-extended/flink-scala-api/pull/98. Within this PR the serialized format for the case classes was changed to include arity
in the blob. Now the deserializer expects arity
as the first 4 bytes, however in the pre-1.1.5 format those first 4 bytes contained some field(s) payload. This results in incorrect deserialization.
Are breaking changes to be expected within patch level releases of this library?
To be honest, I'm not really sure how to get out of this. Since we're dealing with streamed input, naive try { newFormat } catch { oldFormat }
won't do the trick, at least at the level the issue was introduced in.