[PGNCCL] Enable non-blocking API mode by default

[kwen2501]

Stack from ghstack (oldest at bottom):

• -> #137544
• #138384
• #138374
• #137855

Resolves RFC https://github.com/pytorch/pytorch/issues/137007.

Changelist:

• Set default value of nccl_use_nonblocking to true (previous: false).

cc @XilunWu @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o