gloo icon indicating copy to clipboard operation
gloo copied to clipboard

Allow ports to be reused in gloo

Open H-Huang opened this issue 1 year ago • 9 comments

Summary: ProcessGroupGloo and gloo seem to be opening and closing sockets without allowing the port to be reused. We see this issue pop up in larger training jobs "Address already in use" and we assume it to be because all the ephemeral ports are exhausted.

This diff allows ports to be reused, we see a reduced number of ports being in TIME_WAIT state.

context: https://fb.workplace.com/groups/319878845696681/permalink/5988899781205532/

another issue: https://fb.workplace.com/groups/319878845696681/permalink/958768178474408/

Differential Revision: D44029927

H-Huang avatar Mar 27 '23 15:03 H-Huang

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 27 '23 15:03 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 27 '23 17:03 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 27 '23 17:03 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 29 '23 18:03 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 29 '23 18:03 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 29 '23 20:03 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 29 '23 20:03 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 30 '23 18:03 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D44029927

facebook-github-bot avatar Mar 30 '23 18:03 facebook-github-bot