Not-shard-aware port fallback policy should be optional

[vladzcloudius]

What version of Scylla or Cassandra are you using?

2022.2.6

What version of Gocql are you using?

HEAD: e38b2bcca0a61280272806d1e2bd2b23c9749773

Description Scylla shard-aware driver has a not very practical fallback connection policy which core is here: https://github.com/scylladb/gocql/blob/master/scylla.go#L407

In gist it is going to fall back to a "storm" connection policy where the driver is going to open and close TCP sockets till all shards are populated if a connection to a shard-aware port (by default 19042) fails for any reason.

But if a connection fails for any reason except for actual misconfiguration this policy is only going to make things worse. For example, if a connection to 19042 timed out because a corresponding shard is overloaded sending even more sockets in a storm-like manner is only going to add up to the overload.

I suggest to make such a fallback optional (via some configurable) so that people who know how to configure their systems and networking can disable it.

Ref https://github.com/scylladb/gocql/issues/122

[vladzcloudius]

cc @isburmistrov

[dkropachev]

True, but we need to consider many cases, like tablets, busy shard(connection) handling, lwt handling, other type of partitioners. Other part of the issue that could be handled at the same place is host policy/prioritization/filtering based on dc/rack.

We might want to come up with something that unifies all the variants/configurations under one umbrella join together HostSelectionPolicy and ConnPicker.