pulsar
pulsar copied to clipboard
[fix][broker] fix OutOfDirectMemoryError in standalone with zk
Motivation
https://github.com/apache/pulsar/pull/18217 fixed the OutOfDirectMemoryError of startBookieWithMetadataStore. But if you use zookeeper as metadata store, OutOfDirectMemoryError will occur again:
> PULSAR_STANDALONE_USE_ZOOKEEPER=1 ./bin/pulsar standalone --num-bookies 3
2022-11-04T16:45:04,059+0800 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage - Creating single directory db ledger storage on data/standalone/bookkeeper1/current
2022-11-04T16:45:04,060+0800 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 536870912 byte(s) of direct memory (used: 3758161920, max: 4294967296)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:806) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:735) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:186) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledByteBufAllocator.java:181) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:91) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at io.netty.buffer.Unpooled.directBuffer(Unpooled.java:148) ~[io.netty-netty-buffer-4.1.77.Final.jar:4.1.77.Final]
at org.apache.bookkeeper.bookie.storage.ldb.ReadCache.<init>(ReadCache.java:79) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
at org.apache.bookkeeper.bookie.storage.ldb.ReadCache.<init>(ReadCache.java:67) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:174) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:196) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:156) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
at org.apache.bookkeeper.bookie.BookieResources.createLedgerStorage(BookieResources.java:110) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:423) ~[org.apache.bookkeeper-bookkeeper-server-4.15.1.jar:4.15.1]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startBK(LocalBookkeeperEnsemble.java:458) ~[org.apache.pulsar-pulsar-broker-2.11.0.jar:2.11.0]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:325) ~[org.apache.pulsar-pulsar-broker-2.11.0.jar:2.11.0]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:436) ~[org.apache.pulsar-pulsar-broker-2.11.0.jar:2.11.0]
at org.apache.pulsar.PulsarStandalone.startBookieWithZookeeper(PulsarStandalone.java:472) ~[org.apache.pulsar-pulsar-broker-2.11.0.jar:2.11.0]
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:299) ~[org.apache.pulsar-pulsar-broker-2.11.0.jar:2.11.0]
at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:141) ~[org.apache.pulsar-pulsar-broker-2.11.0.jar:2.11.0]
So this PR fixes it in the same way
Documentation
- [ ]
doc - [ ]
doc-required - [x]
doc-not-needed - [ ]
doc-complete
Matching PR in forked repository
PR in forked repository: https://github.com/labuladong/pulsar/pull/7