apache
[Bug] can't write to ZK
Search before asking
- [X] I searched in the issues and found nothing similar.
Version
3.2.0
Minimal reproduce step
/
What did you expect to see?
/
What did you see instead?
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: 2024-02-06T10:02:23,199+0000 [pulsar-load-manager-1-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error writing broker data on metadata store
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: java.util.concurrent.CompletionException: java.lang.IllegalStateException: Lock was not in valid state: Released
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:413) ~[?:?]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2118) ~[?:?]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:1107) ~[org.apache.pulsar-pulsar-broker-3.2.0.jar:3.2.0]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:1092) ~[org.apache.pulsar-pulsar-broker-3.2.0.jar:3.2.0]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:108) ~[org.apache.pulsar-pulsar-broker-3.2.0.jar:3.2.0]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:39) ~[org.apache.pulsar-pulsar-broker-3.2.0.jar:3.2.0]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.105.Final.jar:4.1.105.Final]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: at java.lang.Thread.run(Thread.java:840) ~[?:?]
Feb 06 10:02:23 yo-pulsar-broker-c3-n4 pulsar[2794497]: Caused by: java.lang.IllegalStateException: Lock was not in valid state: Released
Anything else?
No response
Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
@KannarFr Thanks for reporting the issue! what is the frequency of these errors?
Do you see "Unloading namespace-bundles..." in the logs before this exception?
The version I mentioned in the issue isn't correct. I'm running master (a 3.3.0-SNAPSHOT from Jan 31) on the brokers/proxies because I've had many problems with our production cluster for weeks. We are about to deploy the 3.2.0 on every component to correctly define what we have because we have mixed versions of components due to these fixes applications.
One error every 5s until I restarted the broker.
The first error occurred in the following context https://gist.githubusercontent.com/KannarFr/931033f7053eac3468f0424f141e65e1/raw/2222b7cc06250b12321b203e2c33046a5ef53808/gistfile1.txt.
The issue has disappeared since we applied 3.2.0 to each component. Closing.