Improve logging for starting jobs and test cleanup

[AlanCoding]

SUMMARY

This is a mop-up of any remaining issues I saw while trying to do performance testing related to dispatcherd.

We had a case where scale-ups could result in missed messages, making these messages really important. Although the lost messages were recovered, it was still a pretty bad bug. It should be fixed now, but the logs are also valuable to keep.

This also adds some re-scheduling logic, which is going off of a gut intuition more than anything else. This did not ever cause a delay in job starting, and in practice it was always quite fast.

ISSUE TYPE

• Bug, Docs Fix or other nominal change

COMPONENT NAME

• API

---

[!NOTE] Dispatch waiting jobs in batches (up to 25) with timing/job-id logs and self-rescheduling for bursts; update bulk job launch test to override job cap.

• Tasks/Dispatching:
  • Batch process waiting jobs in dispatch_waiting_jobs (slice [:25]).
  • Log dispatched job IDs and elapsed time; debug log when none found.
  • Re-dispatch dispatch_waiting_jobs when multiple jobs are queued to handle bursts.
• Tests:
  • In tests/functional/test_bulk.py, wrap bulk job launch with override_settings(BULK_JOB_MAX_LAUNCH=...) to allow requested job count during query assertions.

^{Written by Cursor Bugbot for commit e6ffadb5d34a98d42ecaddcd9d9b24ab7a265a5c. This will update automatically on new commits. Configure here.}