pp: add basic auth to /brokers

[NyaliaLui]

Cover letter

Moved PR #6338 to this PR because the former had 150+ comments that became difficult to sift through.

At Redpanda there is an effort to improve security by support HTTP Basic Authentication to our RESTful services (e.g., Pandaproxy and Schema Registry). The Pandaproxy will need to account for multiple authenticated connections on top of validating user credentials from the Basic Auth header.

This PR adds a function wrapper to all Pandaproxy route handlers where basic authentication support is within the wrapper. Inside the wrapper is where credentials are taken from the auth header and then validated against the shared credential store.

This PR adds support for multiple authenticated connections by creating a Kafka client cache. The cache is designed as a sharded service, it has a LRU replacement policy, and all clients are destroyed after 30s of liveness. The cache is added to the brokers route only in this patch.

The new configuration options to enable basic auth are attached to the listener in pandaproxy_api:

authentication_method: object
  authentication_type: string
    Optional values: null, http_basic, mTLS
    Default value: null

Examples for the new authn config:

pandaproxy:
  pandaproxy_api:
  - address: localhost
    port: 8082
    authentication_method:
      authentication_type: none

pandaproxy:
  pandaproxy_api:
  - address: localhost
    port: 8082
    authentication_method:
      authentication_type: mtls

pandaproxy:
  pandaproxy_api:
  - address: localhost
    port: 8082
    authentication_method:
      authentication_type: http_basic

After this PR, the leftover tasks are:

• Add support for multiple authenticated connections to the remaining Pandaproxy routes /topic and /consumers
• Add Basic Auth to Schema Registry routes /config, /mode, /schemas, /subjects, /compatibility
• Add ducktape tests for mTLS and mTLS + Basic Auth
• Create upgrade, scale, and feature tests in ducktape
• Manual tests on cloud infra

Changes from force-push 36aad02:

• Use redact_secrets::no
• Add a gate to all instances of the client cache
• Pass the handler to the shard that holds the client instead of passing the client across shards

Backport Required

• [x] not a bug fix
• [ ] issue does not exist in previous branches
• [ ] papercut/not impactful enough to backport
• [ ] v22.2.x
• [ ] v22.1.x
• [ ] v21.11.x

UX changes

• none

Release notes

• Adds HTTP Basic Auth to /brokers on the Pandaproxy

[NyaliaLui]

Ducktape failures are related to this PR.

[dotnwat]

Moved PR https://github.com/redpanda-data/redpanda/pull/6338 to this PR because the former had 150+ comments that became difficult to sift through.

@NyaliaLui does this mean that this PR is ready for review?

[NyaliaLui]

Moved PR #6338 to this PR because the former had 150+ comments that became difficult to sift through.

@NyaliaLui does this mean that this PR is ready for review?

@dotnwat Everything except the ducktape tests are ready for review now. I still need to fix those.

[NyaliaLui]

~~Test failures are related to this PR. Working on them now~~

Edit: Fixed by creating a shard_local_cfg() for pandaproxy::rest::configuration

[NyaliaLui]

I'll fix conflicts (rebase on dev) after this round of reviews

[BenPope]

I'll fix conflicts (rebase on dev) after this round of reviews

Feel free to cherry-pick x <sha> the conflicting change in the meanwhile if you'd like tests to run. Looks like it's one of the most recent 3: https://github.com/redpanda-data/redpanda/commits/dev/src/v/redpanda/application.cc

[dotnwat]

/ci-repeat 5

[dotnwat]

/ci-repeat 10

[NyaliaLui]

BLOCKER: PartitionBalancerTest.test_unavailable_nodes https://github.com/redpanda-data/redpanda/issues/6614

EDIT: Spoke with @piyushredpanda about this. Since the failure is unrelated to this PR we can continue with final reviews