Fix port in worker-address in vllm_worker for #2467

[yaqiangsun]

When just change --port , it has no error but it does not work well when run sevural models in one machine.

And this does not change the behavior when explicitly specify the -worker-address.

Why are these changes needed?

When running more than one model with one controller in one machine, changing --port is a natural behavior. This a more frequent case should be supported.

Related issue number

Open #2467"

Checks

• [x] I've run format.sh to lint the changes in this PR.
• [x] I've included any doc changes needed.
• [x] I've made sure the relevant tests are passing (if applicable).

[yaqiangsun]

@merrymercy can help to review this?