[SPARK-38862][Core] - allow integration of a secure gateway to SparkSubmissionServer alongside spark.authenticate props

[JackBuggins]

What changes were proposed in this pull request?

Adds a new property spark.master.rest.auth.mode which accepts 2 values:

• None: No auth mechanism is at play, this causes spark to fail (as it does today) when using properties spark.authenticate
• SecureGateway: User is telling us they are covering all authentication/authorisation to the rest submission server resource.
• Unknown values default to None providing backward compatibility to config in versions 3x.
• Choose not to expose this on the UI - it doesn't feel need-to-know, let me know if reviewers prefer it is displayed.

Why are the changes needed?

Right now, when enabling the rest submission server, there are validations (sensible ones) that prevent the user from running the master server if configuration if set for spark.authenticate contains a key. It's completely possible a user wants to do use spark.authenticate for other facets of spark, while providing their own gateway to the submission server. We need a way to specify in configuration that the responsibility is not with spark in this deployment.

There are reasons why I feel providing authentication within the rest submission server itself would be unnecessary/limiting/undesirable. I debated with myself on the Jira ticket about piggybacking spark.authenticate props but my overall conclusion is to not do that. In consistency with the rest of the security guidance, we instead enable users to think about this properly themselves.

It is well established to delegate front-door authentication responsibilities to a gateway/proxy service, which can then act as a secure gateway/proxy to the spark master. This could be done trivially using tools like Nginx to provide authentication flexibly, and in a way that is decoupled from any spark distribution supporting this setting. Auth is an area where people often have their own strict requirements too - it opens a can of requirement worms doing something.

To be fully flexible enough to meet everyones requirements via spark source alone we'd likely end up with bloat, not used by most as best case outcome. The additional concern about doing this handling is ource is maintenance of authentication. it could be a sensitive/risk area of the solution that simply does not need to be introduced for now.

Does this PR introduce any user-facing change?

No breaking change, but new configuration options that can be utilised. By default we select the mode None which provides no change. However, new options are doc'd/public scope so that a user can easily toggle the secure gateway mode on. If an unsupported mode is provided, we default to None opposed to throwing a critical error/handling this via require.

How was this patch tested?

Extends on master suite and adds a new suite for somebody to easily extend this in future, providing direct and indirect coverage of the new methods/logic. In addition, tested manually.

Was this patch authored or co-authored using generative AI tooling?

No

[ScrapCodes]

@JackBuggins Thanks for working on it. I have not tested it, but the changes look straight forward.

Title suggestion: [SPARK-38862][deploy] - Add SecureGateway support to spark standalone cluster deploy mode.

[ScrapCodes]

IBM is planning to deprecate Secure gateway: https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-dep-overview . Any comment @JackBuggins ?

[JackBuggins]

Thanks so much for the initial review @ScrapCodes, it's highly appreciated.

IBM is planning to deprecate Secure gateway: https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-dep-overview . Any comment @JackBuggins ?

The SecureGateway option would not be tied to any specific technology, this is just a generic term I've used to try and describe it's function - I'm open to changing that term if it's not clear.

I want to drop in any middleware that runs in the public->local network scope, which can be used as the gateway to/from the public network. For example, even something commonly used such as NGINX could be used to do this.

They cannot without this change because configuration of spark.authenticate properties with the rest submission server being enabled will result in errors to protect the user. This PR aims to allow the admin to tell spark they will take care of the gaps themselves with their "Secure Gateway". What the user actually chooses to do at that gateway is up to them.

Users already need gateways of sorts for egress with any type of submission today when you need to fetch a jar from a remote server that has auth requirement, I have to upgrade that connection myself. This change is similar, but in the opposite direction in that I'm handling the downgrade of connections in a way that matches up with my security policy - basically saying "trust me, I've got this" to the other existing auth components in spark.

[github-actions[bot]]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!