[BUG] Timeout error may be thrown when the worker has already finished.

[lnfjpt]

As title

[github-actions[bot]]

/cc @lnfjpt, this issus/pr has had no activity for for a long time, could you folks help to review the status ? To suppress further notifications,

• for issues,
  • if it is waiting for further response from the reporter/author, please help to add the label requires-further-info,
  • if you have already started working on it, please add the label work-in-progress to the issue,
  • if this issue requires further designing discussion and not in current plan, or won't be fixed, please add the label requires-further-discussion or wontfix to the issue,
• for pull requests,
  • if you are still working on it and it is not ready for reviewing, please convert this pull request as draft PR,
  • if you have decided to hold this development on, please add the requires-further-discussion label to the pull request. Thanks!