Check create topic permission on topic creation using pulsar proto clients

[KannarFr]

Fixes #17406

Motivation

Enable CREATE_TOPIC permission check on topic auto creation.

Modifications

Introduce AuthAction.create_topic for the PulsarAuthorizationProvider

Verifying this change

• [ ] Make sure that the change passes the CI checks.

This change updated tests and can be verified as follows:

• updated test and give create_topic permission on the namespace

Does this pull request potentially affect one of the following parts:

• it affects permissions as currently, CREATE_TOPIC wasn't checked on the default authZ provider.

• [x] doc-required

[github-actions[bot]]

@KannarFr Please provide a correct documentation label for your PR. Instructions see Pulsar Documentation Label Guide.

[KannarFr]

@gaoran10 the failing tests are due to CI flakyness, nope?

[KannarFr]

@michaeljmarshall are you ok with the way I've done it in handleProducer?

[github-actions[bot]]

@KannarFr Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[KannarFr]

@michaeljmarshall there is something I missed?

[github-actions[bot]]

The pr had no activity for 30 days, mark with Stale label.

[KannarFr]

@poorbarcode, @michaeljmarshall looks not available, can you ping someone else? It would be cool to merge it upstream asap :D.

[KannarFr]

After reviewing this, here are a collection of thoughts on Pulsar's Authorization.

At it's core, I think the problem described by #17406 is that allowAutoTopicCreation is a configuration about permission/authorization.

Yes?

In my view, the allowAutoTopicCreation=true says "a role with permission to produce/consume to a topic also has permission to create that topic".

The HTTP admin API does not comply with this sentence. The CREATE_TOPIC operation is defined and used by HTTP admin API authz checks. I agree that this is introducing breaking changes in the permissions system and this is a problem, but there is authZ plugin provider providing this operation check and does not verify it during producer/consumer. So, I have no idea what the best answer is, but we can't stay and need to find a solution or make a decision here.

This change proposes that allowAutoTopicCreation=true and produce/consumer permissions are insufficient, and that a role must also have explicit permission to create a topic.

Well, this is the default behavior of every configurations keys in pulsar. Like namespace policies or tiered storage configurations, there is a default value that can be overridden by custom namespace policies.

[michaeljmarshall]

The HTTP admin API does not comply with this sentence.

Fair point. I hadn't considered the pulsar and http endpoints together when writing that generalization.

I agree that this is introducing breaking changes in the permissions system and this is a problem, but there is authZ plugin provider providing this operation check and does not verify it during producer/consumer.

Is there another way to achieve this? Perhaps we can introduce an additional check that calls the authorization service and then we'll implement the default PulsarAuthorizationProvider in such a way that there are no breaking changes.

I think it might make sense to discuss the underlying inconsistency on the pulsar mailing list.

[KannarFr]

Perhaps we can introduce an additional check that calls the authorization service and then we'll implement the default PulsarAuthorizationProvider in such a way that there are no breaking change

This LGTM. Can you drive this discussion on the ML if you think it's required? Shall I do it?

[michaeljmarshall]

Perhaps we can introduce an additional check that calls the authorization service and then we'll implement the default PulsarAuthorizationProvider in such a way that there are no breaking change

This LGTM. Can you drive this discussion on the ML if you think it's required? Shall I do it?

I won't be able to drive this discussion. Let me know if you have any questions though.

[github-actions[bot]]

The pr had no activity for 30 days, mark with Stale label.

[KannarFr]

I sent

Hi,

CREATE_TOPIC authorization check is not performed when trying to PRODUCE/CONSUME a topic, it has been referenced: https://github.com/apache/pulsar/issues/17406.

I opened a PR to fix it https://github.com/apache/pulsar/pull/17411, but Michael reported issues about backward compatibility (which is totally correct). Adding support of CREATE_TOPIC authorization as-is will break current authorization system. I noticed that HTTP Admin API verifies the CREATE_TOPIC right when creating topic, so we have inconsistencies between pulsar binary protocol and the HTTP admin API on this.

Also, the AuthorizationProvider is an interface exposing the CREATE_TOPIC feature for authZ plugins. But it is inconsistent too.

Michael suggested to fix this interface to support the CREATE_TOPIC check and adapt the pulsar's DefaultAuthzProvider to continue as-is.

I'd like to know what do you think?

Thanks,

Kannar

On the ML the 20/04/2023 and still have no answers @michaeljmarshall @nodece.

[Nowadays]

Is there any progression on this issue ?

[KannarFr]

Unfortunately, no. I don't know why this is stuck @michaeljmarshall?

[KannarFr]

@gaoran10 @MarvinCai

[KannarFr]

@michaeljmarshall @gaoran10 @MarvinCai, it makes sense to rebase it..?

[Nowadays]

Is there any way to move forward with this PR ?

publishing/consuming and creating/deleting a topic are clearly different actions, and thus should require different authorization.

Why would there be an api to create/delete topics if that was not the case ?

[KannarFr]

Any bumps?