proxy: cache certain non-retriable console errors for a short time

[conradludgate]

Problem

If there's a quota error, it makes sense to cache it for a short window of time. Many clients do not handle database connection errors gracefully, so just spam retry 🤡

Summary of changes

Updates the node_info cache to support storing console errors. Store console errors if they cannot be retried (using our own heuristic. should only trigger for quota exceeded errors).

Checklist before requesting a review

• [x] I have performed a self-review of my code.
• [ ] If it is a core feature, I have added thorough tests.
• [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• [ ] Do not forget to reformat commit message to not include the above checklist

[github-actions[bot]]

3006 tests run: 2891 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (3)

Postgres 15

• test_storage_controller_smoke: release
• test_tenant_delete_smoke: release

Postgres 14

• test_subscriber_restart: release

Code coverage* (full report)

• functions: 32.7% (6934 of 21229 functions)
• lines: 50.0% (54322 of 108609 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
bddacc1cfe607d1d8538a60e3e5aee836c170f88 at 2024-07-03T15:43:42.686Z :recycle:}