UI sometimes gives "waiting" status while job is running

[AlanCoding]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

Launching the Demo Job Template with current devel, I can relatively frequently observe (say 1 time in 4) a job staying in the "waiting" state in the UI while it is actually running. As the job finishes, it then flips from waiting to successful.

So the UI is not processing the "running" status message (which is sent by the API), or some other data incorrectly takes precedence over that.

AWX version

devel

Select the relevant components

• [X] UI
• [ ] API
• [ ] Docs
• [ ] Collection
• [ ] CLI
• [ ] Other

Installation method

docker development environment

Modifications

no

Ansible version

N/A

Operating system

N/A

Web browser

Chrome

Steps to reproduce

Launch the Demo Job Template several times. It may take a handful of tries to get this behavior. I tend to use the relaunch button.

Expected results

I expect that the job status will be running while I am getting events coming in.

Actual results

Stays in the waiting status until it completes.

Additional information

I check the websocket network traffic and see that all status transitions are sent for the job in question - pending, waiting, running, and successful.

Interestingly, I do see that the UI makes multiple API requests for the job here.

In the cases where the status gets stuck in waiting, I do see an API request returning the waiting status. It is entirely possible that the API reports this waiting status after the "running" websocket message has come in, due to comparative slowness in request processing.

I found this issue when researching for https://github.com/ansible/awx/pull/12573, where, bizarrely, completely removing the websocket message for the "waiting" message did not avoid this bug. That gives further evidence to back the idea that the UI gives a stale status based on an API request, which is extremely likely to predate the websocket data.