Michael Marshall

> This PR remain as the second PR to optimize the implementation of permission. I am not familiar with how to get consensus, so I would wait for update this...

I just ran into the following error while running `testBacklogConsumerCacheReads` on my MacOS on a branch that includes #16996. Looks like there might be a separate issue, too. I'm going...

Reopening this ticket because I am consistently getting this test to fail on my local machine. I also just go this variant: ``` 2022-08-12T16:51:21,672 - ERROR - [broker-topic-workers-OrderedExecutor-7-0:Commands@1859] - [PersistentSubscription{topic=persistent://my-property/my-ns/cache-read,...

Seems like this might be related to https://github.com/apache/pulsar/pull/10924.

https://github.com/apache/pulsar/pull/10924 is not related. Just before calling the `parseFrom` in the stack trace, we call `skipBrokerEntryMetadataIfExist`. https://github.com/apache/pulsar/blob/926834ef8b2b57d3964aa7e9773e6245bcee861c/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/Commands.java#L436-L446

It's clear to me now that the issue is a data race with the `entries` array. After https://github.com/apache/pulsar/pull/16603, entries are published to another thread. I believe they are "safely published"...

I wrote https://github.com/michaeljmarshall/pulsar/commit/8f48cf2ebc0bc5b8004be807611271aea29d8c5f thinking that it would solve the issue. However, I am still getting the same exceptions, so there must be something I am missing. I'll continue on this...

It looks like the test is failing sometimes because the consumer is getting duplicate messages. This flakiness may be an actual bug. I'm able to reproduce this flaky test failure...

Both #16812 and #16609 look related since they modified the `readMoreEntries` method used heavily in this test.

#16968 also looks related.