Use single sequence for task and job ids of different schema versions

[klopfdreh]

trafficstars

Problem description: We currently have ~250k job executions and when we migrate to Spring Boot 3 and Spring Batch 5 the ids are starting from one. Because of this we have problems finding the right page to look at:

Also the CTR and task applications are running in different schema versions, so different sequences are used and the ids are divergent.

Solution description: Use a single sequence for boot 2 and boot 3 schemas so that the ids are increasing no matter what kind of task application has started.

Description of alternatives: Implement a filter to narrow down the view to only show Boot2 / Boot3 Task and Job executions

Additional context: N/A

[klopfdreh]

This issue is the only one which prevents us from upgrading to Spring Boot 3 applications, as we don't have the time to navigate through all pages to see the task / job execution outcome.

If it is not an option to change the schema of the database as it is the same for Spring Batch 5 / Spring Batch 4 - I would suggest to implement the alternative solution to narrow down the view.

[cppwfs]

We add search criteria that will implement the alternative solution you suggested for the 2.11.5 release.
Thanks for opening the issue.

[klopfdreh]

Hey @cppwfs - thanks a lot for the information - if I can contribute in any way please let me know! 👍

[klopfdreh]

@cppwfs - do you want me to contribute the ui part of this feature?

[corneil]

Filter with EXIT_CODE!=0 will probably help find issues while ordering by END_TIME,desc

[corneil]

Here's a quick solution to find the first 100 failures:

curl -s "$DATAFLOW_IP/tasks/thinexecutions?size=100&sort=END_TIME,desc" \
  | jq '._embedded.taskExecutionThinResourceList | .[] | select(.exitCode!=0) | {executionId,startTime,endTime,taskName,taskExecutionStatus,schemeTarget}' \
  | jq -s '.'

[klopfdreh]

Hey @corneil - thanks a lot for help! 👍

We already implemented some custom endpoints to 2.11.5 to filter with targetSchema and made the SCDF UI adjustments in our code base. This way you can order each column without restrictions.

If it would help to provide it for 3.x - let us know. 👍

[corneil]

That would probably be useful since the views are going away and we could probably do with a STATUS column instead of deriving the value in resource assembler, making sorting and filtering a challenge.