kvserver: proxy requests that return a NotLeaseholderError

[andrewbaptist]

This handles Partial partitions from the DistSender layer.

Here is an example of how this test is set up

n3 /
n1---n2

n1 is the client and n2 is the leaseholder.

The connection between n1 and n2 is broken n3 /
n1 n2

For the following request this sequence happens:

n1 -> n2 -> fails as expected - (network cut) n1 -> n1 -> fails with not leaseholder don't proxy since self-client n1 -> n3 -> fails as expected, but proxy to its local dist_sender n3 -> n1 -> fail as expected. leaseholder not known and can't proxy n3 -> n2 -> success

We could optimize out the first leaseholder lookup on n3 by pre-populating its distsender cache before the proxy request, but that requires a bit of a layering violation and isn't necessary for correctness.

Epic: none

Release note: None

[blathers-crl[bot]]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[andrewbaptist]

This is mostly working now. There are a few issues still open related to it:

1. Observability. We need metrics that show when proxying is happening. (In progress)
2. Changefeeds: These will currently block during a partition and are not easily observable.
3. DistSQL: Its unclear if this is a problem. There isn't great testing related to this right now so at a minimum we should add that.

[andrewbaptist]

bors r=erikgrinaker

TFTR!

[craig[bot]]

Build succeeded:

• Bazel Essential CI (Cockroach)
• check_generated_code
• docker_image_amd64
• examples_orms
• macos_amd64_build
• macos_arm64_build
• windows_build