[FEATURE]: Can we support automatically searching the idle local network port

[ErenChan]

Describe the feature

I often met the problem "RuntimeError: The server socket has failed to listen on any local network address." Usually, I manually run the script several times to get my program running.

 /root/py37/lib/python3.7/site-packages/torch/distributed/rendezvous.py:201   │
│ in _tcp_rendezvous_handler                                                                       │
│                                                                                                  │
│   198 │   world_size = int(query_dict["world_size"])                                             │
│   199 │   assert result.hostname is not None                                                     │
│   200 │                                                                                          │
│ ❱ 201 │   store = _create_c10d_store(result.hostname, result.port, rank, world_size, timeout)    │
│   202 │                                                                                          │
│   203 │   yield (store, rank, world_size)                                                        │
│   204                                                                                            │
│                                                                                                  │
│ /root/py37/lib/python3.7/site-packages/torch/distributed/rendezvous.py:178   │
│ in _create_c10d_store                                                                            │
│                                                                                                  │
│   175 │   else:                                                                                  │
│   176 │   │   start_daemon = rank == 0                                                           │
│   177 │   │   return TCPStore(                                                                   │
│ ❱ 178 │   │   │   hostname, port, world_size, start_daemon, timeout, multi_tenant=True           │
│   179 │   │   )                                                                                  │
│   180                                                                                            │
│   181                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: The server socket has failed to listen on any local network address. The server socket has failed to bind to [::]:10000 (errno: 98 - Address already in use). The server socket has 
failed to bind to 0.0.0.0:10000 (errno: 98 - Address already in use).
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 113885 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 113884) of binary: /root/py37/bin/python
Traceback (most recent call last):
  File "/root/py37/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==1.13.1', 'console_scripts', 'torchrun')())
  File "/root/py37/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/root/py37/lib/python3.7/site-packages/torch/distributed/run.py", line 762, in main
    run(args)
  File "/root/py37/lib/python3.7/site-packages/torch/distributed/run.py", line 756, in run
    )(*cmd_args)
  File "/root/py37/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/py37/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 248, in launch_agent
    failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
./train_gpt_bd.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-02-21_23:01:20
  host      : 10-78-115-12.job-0bb63ef7f0948e9f-trainer-0.8bb8a74a-9fe9-54a8-be50-66a2ff2c3dab.svc.cluster.local
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 113884)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================