vmstorage: time cost on VMInsertServer 's shutdown will be longer than -vminsertConnsShutdownDuration

[JayiceZ]

Describe the bug

Currently, to improve re-routing performance during restart, vmstorage will close the connections with vminsert one by one. After closing the connections to one vminsert-host, it will wait for a interval(calculated by storage.vminsertConnsShutdownDuration) before closing the connections to next vminsert-host.

These logics were implemented in here.

But after closing the last connection, it will still wait for the interval, causing the total time cost will always be longer than storage.vminsertConnsShutdownDuration. The extra time overhead will be:

storage.vminsertConnsShutdownDuration / (vminsert_hosts_number -1)

Becasue the actual shutdown-time will always be longer than expectation, so if vmstorage is managed by systemd, k8s or sth, the actual shutdown-time may exceed the shutdown timeout configured in these manage-service, then vmstorage instance may be killed.

To Reproduce

1. use more than 1 vminsert host to connect vmstorage. (if there's only one vminsert host, vmstorage will shutdown without waiting).
2. shut down the vmstorage process.
3. See the log here, it shows the time cost will always be more than storage.vminsertConnsShutdownDuration

Version

Latest version, after https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9487