paimon icon indicating copy to clipboard operation
paimon copied to clipboard

[Bug] Sequence group may cause record in wrong state.

Open leaves12138 opened this issue 8 months ago • 1 comments

Search before asking

  • [x] I searched in the issues and found nothing similar.

Paimon version

If my records come disorderly, -D record may have been compact out of table. Then +I (which sequence number is smaller then -D record) may survive.

                StreamWriteBuilder builder = getTable().newStreamWriteBuilder();
                StreamTableWrite write = builder.newWrite();
                write.withIOManager(new IOManagerImpl("~/temp/"));
                StreamTableCommit commit = builder.newCommit();

                // +I[2025-04-01, 1, type1, 29, 5, name2, 29, 5, name2, null, 9223372036854775802]
                GenericRow genericRow = GenericRow.of(
                        1L,
                        2L,
                        2L);

                GenericRow genericRow2 = GenericRow.ofKind(RowKind.DELETE,
                        1L,
                        null,
                        Long.MAX_VALUE - 1);

                GenericRow genericRow3 = GenericRow.of(
                        1L,
                        1L,
                        1L);


                write.write(genericRow);
                commit.commit(1L, write.prepareCommit(true, 1L));

                write.write(genericRow2);
                commit.commit(2L, write.prepareCommit(true, 2L));
                readData();

                write.write(genericRow3);
                commit.commit(3L, write.prepareCommit(true, 3L));

Record should be 1,null,Long.MAX_VALUE-1

Compute Engine

Flink

Minimal reproduce step

above

What doesn't meet your expectations?

--

Anything else?

No response

Are you willing to submit a PR?

  • [ ] I'm willing to submit a PR!

leaves12138 avatar Apr 02 '25 03:04 leaves12138

@leaves12138 In your case, if the sequence number is defined as the seqComparator in the sequence group, the output record should be 1,null,Long.MAX_VALUE-1

xiangyuf avatar Apr 13 '25 15:04 xiangyuf