starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] Improve routine load too many error rows message

Open wyb opened this issue 1 year ago • 3 comments

Why I'm doing:

mysql> show routine load from kafka_test\G
*************************** 1. row ***************************
                  Id: 282486
                Name: label0
          CreateTime: 2024-05-11 15:26:20
           PauseTime: 2024-05-11 15:27:15
             EndTime: NULL
              DbName: kafka_test
           TableName: lineorder2
               State: PAUSED
      DataSourceType: KAFKA
      CurrentTaskNum: 0
       JobProperties: {"partitions":"*","rowDelimiter":"\t","partial_update":"false","columnToColumnExpr":"*","maxBatchIntervalS":"10","partial_update_mode":"null","whereExpr":"*","timezone":"Asia/Shanghai","format":"csv","columnSeparator":"'|'","log_rejected_record_num":"0","taskTimeoutSecond":"60","json_root":"","maxFilterRatio":"1.0","strict_mode":"false","js
onpaths":"","taskConsumeSecond":"15","desireTaskConcurrentNum":"5","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"1","maxBatchRows":"200000"}
DataSourceProperties: {"topic":"wyb_test","currentKafkaPartitions":"0","brokerList":"127.0.0.1:9093"}
    CustomProperties: {"group.id":"label0_012bae3f-19f4-452f-bb32-8133ecc9f2c8"}
           Statistic: {"receivedBytes":2953715,"errorRows":29657,"committedTaskNum":0,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":1,"totalRows":29657,"unselectedRows":0,"receivedBytesRate":196000,"taskExecuteTimeMs":15059}                                                                                                                                                     Progress: {"0":"81418"}
   TimestampProgress: {"0":"1715412435492"}
ReasonOfStateChanged: ErrorReason{errCode = 102, msg='current error rows is more than max error num'}
        ErrorLogUrls: http://127.0.0.1:8049/api/_load_error_log?file=error_log_6f7b01c3144a4b98_ab0e42d4b3329205
         TrackingSQL: select tracking_log from information_schema.load_tracking_logs where job_id=282486
            OtherMsg: 
LatestSourcePosition: {"0":"62499"}
1 row in set (0.00 sec)

What I'm doing:

mysql> show routine load from kafka_test\G
*************************** 1. row ***************************
                  Id: 283192
                Name: label0
          CreateTime: 2024-05-11 16:12:36
           PauseTime: 2024-05-11 16:24:56
             EndTime: NULL
              DbName: kafka_test
           TableName: lineorder2
               State: PAUSED
      DataSourceType: KAFKA
      CurrentTaskNum: 0
       JobProperties: {"partitions":"*","rowDelimiter":"\t","partial_update":"false","columnToColumnExpr":"*","maxBatchIntervalS":"10","partial_update_mode":"null","whereExpr":"*","timezone":"Asia/Shanghai","format":"csv","columnSeparator":"'|'","log_rejected_record_num":"0","taskTimeoutSecond":"60","json_root":"","maxFilterRatio":"1.0","strict_mode":"false","jsonpaths":"","taskConsumeSecond":"15","desireTaskConcurrentNum":"5","maxErrorNum":"10000","strip_outer_array":"false","currentTaskConcurrentNum":"1","maxBatchRows":"200000"}
DataSourceProperties: {"topic":"wyb_test","currentKafkaPartitions":"0","brokerList":"127.0.0.1:9093"}
    CustomProperties: {"group.id":"label0_c3e8da1a-8cc9-4df9-85c7-eb7b72e5020c"}
           Statistic: {"receivedBytes":2422562,"errorRows":24345,"committedTaskNum":0,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":6,"totalRows":24345,"unselectedRows":0,"receivedBytesRate":307000,"taskExecuteTimeMs":7881}
            Progress: {"0":"130452"}
   TimestampProgress: {"0":"1715415895552"}
ReasonOfStateChanged: ErrorReason{errCode = 102, msg='Current error rows: 13755 is more than max error num: 10000. Check the 'TrackingSQL' field for detailed information'}
        ErrorLogUrls: http://127.0.0.1:8049/api/_load_error_log?file=error_log_f3a6d9fc207f4bcd_a592dd995cb80bce
         TrackingSQL: select tracking_log from information_schema.load_tracking_logs where job_id=283192
            OtherMsg:
LatestSourcePosition: {"0":"124109"}
1 row in set (0.01 sec)

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [x] Yes, this PR will result in a change in behavior.
  • [ ] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [x] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [x] 3.3
    • [x] 3.2
    • [x] 3.1
    • [ ] 3.0
    • [ ] 2.5

wyb avatar May 11 '24 08:05 wyb

[FE Incremental Coverage Report]

:white_check_mark: pass : 4 / 4 (100.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/load/routineload/RoutineLoadJob.java 3 3 100.00% []
:large_blue_circle: com/starrocks/common/ErrorCode.java 1 1 100.00% []

github-actions[bot] avatar May 13 '24 04:05 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar May 13 '24 04:05 github-actions[bot]

@Mergifyio backport branch-3.3

github-actions[bot] avatar May 16 '24 06:05 github-actions[bot]

@Mergifyio backport branch-3.2

github-actions[bot] avatar May 16 '24 06:05 github-actions[bot]

@Mergifyio backport branch-3.1

github-actions[bot] avatar May 16 '24 06:05 github-actions[bot]

backport branch-3.3

✅ Backports have been created

mergify[bot] avatar May 16 '24 06:05 mergify[bot]

backport branch-3.2

✅ Backports have been created

mergify[bot] avatar May 16 '24 06:05 mergify[bot]

backport branch-3.1

✅ Backports have been created

mergify[bot] avatar May 16 '24 06:05 mergify[bot]