[FLINK-37851][state] Migrate state processor API from source API v1 to source API v2

[gaborgsomogyi]

What is the purpose of the change

The state processor API is now depending on data source v1 API and in the PR I've migrated it to data source v2.

Brief change log

Migrated state processor API from DSV1 to DSV2.

Verifying this change

Existing and newly added unit tests.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• f441cc9fb704f1ad164ab9e3b43dcdf8ba767c98 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[gaborgsomogyi]

cc @Zakelly high scale tests are ongoing + the actual version not yet contains the rich trick what we've spoken about. First I would like to reach a rock stable state and then make the enrichment. All in all this will be code base hopefully with small modifications...

[gaborgsomogyi]

I've just tested it on load and both streaming and batch execution mode works fine with the same performance. Now enrichment is the missing piece.

[gaborgsomogyi]

Rebased to master and added the enrichment so it's final from my point of view.

[gaborgsomogyi]

cc @gyfora

[gaborgsomogyi]

Seems like unrelated test failure:

May 31 22:49:33 22:49:33.667 [ERROR]   Updating frozen violations is disabled (enable by configuration freeze.store.default.allowStoreUpdate=true)

[gaborgsomogyi]

Ok, so finally green run after I've fixed FLINK-37884 and made the same re-generation here as well. In the last commit this new code just eliminated 2 old violations which I think is fine. All in all this is ready for review.

[gaborgsomogyi]

I just came out with our new simplified version. The code is not yet cleaned up, some renaming and class move is maybe needed, just want to see whether it works. Just to double check the fundamentals, @gyfora you meant something like this?

[gaborgsomogyi]

Now cleaned up things and fixed the bugs.

[gaborgsomogyi]

Just rebased to the latest master to be fresh.

[gaborgsomogyi]

I've tested it under load on cluster and works fine.

[gaborgsomogyi]

Thanks for the efforts! Waiting on couple of days before merge to let others to share opinions. Intended to merge mid next week if no further comments arise.

[gaborgsomogyi]

After some further testing we've found cases where it's not running correctly so bugfix needed.

[gaborgsomogyi]

The fix has been added since it works fine.

[gaborgsomogyi]

Branch cut has been done so green light to merge it. Before merge rebased to the latest master just to be on safe side.