Fix the scheduler may retry an impossible task for long time

[yah01]

/kind bug related #23604

[mergify[bot]]

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[codecov[bot]]

Codecov Report

Merging #23649 (fd4c964) into 2.2.0 (cb82783) will decrease coverage by 0.02%. The diff coverage is 43.75%.

@@            Coverage Diff             @@
##            2.2.0   #23649      +/-   ##
==========================================
- Coverage   80.62%   80.61%   -0.02%     
==========================================
  Files         717      717              
  Lines      101499   101514      +15     
==========================================
  Hits        81837    81837              
- Misses      16451    16466      +15     
  Partials     3211     3211              

Impacted Files	Coverage Δ
internal/querycoordv2/task/executor.go	68.03% <10.00%> (-1.64%)	:arrow_down:
internal/querycoordv2/job/job.go	84.11% <100.00%> (+0.23%)	:arrow_up:

... and 16 files with indirect coverage changes

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, yah01

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [congqixia]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment