Pulsar cluster occasionally experiences consumption timeout

[ktpktr0]

Search before asking

• [X] I searched in the issues and found nothing similar.

Version

os: CentOS Linux release 7.9.2009 (Core) pulsar: 2.10.5

Minimal reproduce step

Occasionally appearing, unable to reproduce

What did you expect to see?

Consumption completed in a short period of time

What did you see instead?

Occasionally, if the consumption exceeds the limit, it will take at least ten minutes or longer to complete the consumption

Anything else?

Pulsar Bookie Configuration

bookiePort=3181
journalDirectory=data/bookkeeper/journal
minUsableSizeForIndexFileCreation=1073741824
advertisedAddress=192.168.0.93
allowLoopback=false
bookieDeathWatchInterval=1000
flushInterval=60000
useHostNameAsBookieID=false
gcWaitTime=900000
gcOverreplicatedLedgerWaitTime=86400000
numAddWorkerThreads=0
numReadWorkerThreads=8
numHighPriorityWorkerThreads=8
maxPendingReadRequestsPerThread=2500
maxPendingAddRequestsPerThread=10000
readWorkerThreadsThrottlingEnabled=true
enableBusyWait=false
auditorPeriodicBookieCheckInterval=86400
rereplicationEntryBatchSize=100
openLedgerRereplicationGracePeriod=30000
autoRecoveryDaemonEnabled=true
lostBookieRecoveryDelay=0
useV2WireProtocol=true
reppDnsResolverClass=org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping
serverTcpNoDelay=true
nettyMaxFrameSizeBytes=5253120
journalFormatVersionToWrite=5
journalMaxSizeMB=2048
journalMaxBackups=5
journalPreAllocSizeMB=16
journalWriteBufferSizeKB=64
journalRemoveFromPageCache=true
journalWriteData=true
journalSyncData=true
journalAdaptiveGroupWrites=true
journalMaxGroupWaitMSec=1
journalBufferedWritesThreshold=524288
numJournalCallbackThreads=8
journalAlignmentSize=4096
journalFlushWhenQueueEmpty=false
ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
ledgerDirectories=data/bookkeeper/ledgers
auditorPeriodicCheckInterval=604800
openFileLimit=0
fileInfoFormatVersionToWrite=0
pageLimit=0
zkLedgersRootPath=/ledgers
logSizeLimit=1073741824
entryLogFilePreallocationEnabled=true
flushEntrylogBytes=268435456
readBufferSizeBytes=4096
writeBufferSizeBytes=65536
compactionRate=1000
minorCompactionThreshold=0.2
minorCompactionInterval=3600
compactionMaxOutstandingRequests=100000
majorCompactionThreshold=0.5
majorCompactionInterval=86400
isThrottleByBytes=false
compactionRateByEntries=1000
compactionRateByBytes=1000000
statsProviderClass=org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
prometheusStatsHttpPort=8100
readOnlyModeEnabled=true
diskUsageThreshold=0.95
diskCheckInterval=10000
metadataServiceUri=
zkServers=192.168.0.118:2181,192.168.0.93:2181,192.168.0.225:2181
zkTimeout=30000
zkEnableSecurity=false
httpServerEnabled=false
httpServerPort=8000
httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer
dbStorage_writeCacheMaxSizeMb=
dbStorage_readAheadCacheMaxSizeMb=
dbStorage_readAheadCacheBatchSize=1000
dbStorage_rocksDB_blockCacheSize=
dbStorage_rocksDB_writeBufferSizeMB=64
dbStorage_rocksDB_sstSizeInMB=64
dbStorage_rocksDB_blockSize=65536
dbStorage_rocksDB_bloomFilterBitsPerKey=10
dbStorage_rocksDB_numLevels=-1
dbStorage_rocksDB_numFilesInLevel0=4
dbStorage_rocksDB_maxSizeInLevel1MB=256

Pulsar broker configuration

metadataStoreUrl=192.168.0.118:2181,192.168.0.93:2181,192.168.0.225:2181
configurationMetadataStoreUrl=192.168.0.118:2181,192.168.0.93:2181,192.168.0.225:2181
brokerServicePort=6650
brokerServicePortTls=
webServicePort=8080
webServicePortTls=
webServiceTlsProtocols=
webServiceTlsCiphers=
bindAddress=0.0.0.0
bindAddresses=
advertisedAddress=192.168.0.93
haProxyProtocolEnabled=false
numAcceptorThreads=
numIOThreads=
numOrderedExecutorThreads=8
numHttpServerThreads=
numExecutorThreadPoolSize=
numCacheExecutorThreadPoolSize=10
enableBusyWait=false
maxConcurrentHttpRequests=1024
isRunningStandalone=
clusterName=pulsar-cluster
maxTenants=0
failureDomainsEnabled=false
metadataStoreSessionTimeoutMillis=30000
metadataStoreOperationTimeoutSeconds=30
metadataStoreCacheExpirySeconds=300
brokerShutdownTimeoutMs=60000
skipBrokerShutdownOnOOM=false
backlogQuotaCheckEnabled=true
backlogQuotaCheckIntervalInSeconds=60
backlogQuotaDefaultLimitBytes=-1
backlogQuotaDefaultLimitSecond=-1
backlogQuotaDefaultRetentionPolicy=producer_request_hold
ttlDurationDefaultInSeconds=0
allowAutoTopicCreation=true
allowAutoTopicCreationType=non-partitioned
allowAutoSubscriptionCreation=true
defaultNumPartitions=1
brokerDeleteInactiveTopicsEnabled=true
brokerDeleteInactiveTopicsFrequencySeconds=60
brokerDeleteInactiveTopicsMode=delete_when_no_subscriptions
brokerDeleteInactivePartitionedTopicMetadataEnabled=false
brokerDeleteInactiveTopicsMaxInactiveDurationSeconds=
forceDeleteTenantAllowed=false
forceDeleteNamespaceAllowed=false
maxPendingPublishRequestsPerConnection=1000
messageExpiryCheckIntervalInMinutes=5
activeConsumerFailoverDelayTimeMillis=1000
subscriptionExpirationTimeMinutes=0
subscriptionRedeliveryTrackerEnabled=true
subscriptionExpiryCheckIntervalInMinutes=5
subscriptionTypesEnabled=Exclusive,Shared,Failover,Key_Shared
subscriptionKeySharedUseConsistentHashing=true
subscriptionKeySharedConsistentHashingReplicaPoints=100
brokerDeduplicationEnabled=false
brokerDeduplicationMaxNumberOfProducers=10000
brokerDeduplicationSnapshotFrequencyInSeconds=120
brokerDeduplicationSnapshotIntervalSeconds=120
brokerDeduplicationEntriesInterval=1000
brokerDeduplicationProducerInactivityTimeoutMinutes=360
defaultNumberOfNamespaceBundles=4
maxNamespacesPerTenant=0
maxTopicsPerNamespace=0
brokerMaxConnections=0
brokerMaxConnectionsPerIp=0
isAllowAutoUpdateSchemaEnabled=true
clientLibraryVersionCheckEnabled=false
statusFilePath=
preferLaterVersions=false
maxUnackedMessagesPerConsumer=50000
maxUnackedMessagesPerSubscription=200000
maxUnackedMessagesPerBroker=0
maxUnackedMessagesPerSubscriptionOnBrokerBlocked=0.16
unblockStuckSubscriptionEnabled=false
topicPublisherThrottlingTickTimeMillis=10
preciseTopicPublishRateLimiterEnable=false
brokerPublisherThrottlingTickTimeMillis=50
brokerPublisherThrottlingMaxMessageRate=0
brokerPublisherThrottlingMaxByteRate=0
maxPublishRatePerTopicInMessages=0
maxPublishRatePerTopicInBytes=0
subscribeThrottlingRatePerConsumer=0
subscribeRatePeriodPerConsumerInSecond=30
dispatchThrottlingRateInMsg=0
dispatchThrottlingRateInByte=0
dispatchThrottlingRatePerTopicInMsg=0
dispatchThrottlingRatePerTopicInByte=0
dispatchThrottlingOnBatchMessageEnabled=false
dispatchThrottlingRatePerSubscriptionInMsg=0
dispatchThrottlingRatePerSubscriptionInByte=0
dispatchThrottlingRatePerReplicatorInMsg=0
dispatchThrottlingRatePerReplicatorInByte=0
dispatchThrottlingRateRelativeToPublishRate=false
dispatchThrottlingOnNonBacklogConsumerEnabled=true
dispatcherMaxReadBatchSize=100
dispatcherMaxReadSizeBytes=5242880
dispatcherMinReadBatchSize=1
dispatcherMaxRoundRobinBatchSize=20
dispatcherReadFailureBackoffInitialTimeInMs=15000
dispatcherReadFailureBackoffMaxTimeInMs=60000
dispatcherReadFailureBackoffMandatoryStopTimeInMs=0
preciseDispatcherFlowControl=false
entryFilterNames=
entryFiltersDirectory=
maxConcurrentLookupRequest=50000
maxConcurrentTopicLoadRequest=5000
maxConcurrentNonPersistentMessagePerConnection=1000
numWorkerThreadsForNonPersistentTopic=
enablePersistentTopics=true
enableNonPersistentTopics=true
enableRunBookieTogether=false
enableRunBookieAutoRecoveryTogether=false
maxProducersPerTopic=0
maxSameAddressProducersPerTopic=0
encryptionRequireOnProducer=false
maxConsumersPerTopic=0
maxSameAddressConsumersPerTopic=0
maxSubscriptionsPerTopic=0
maxConsumersPerSubscription=0
maxMessageSize=5242880
brokerServiceCompactionMonitorIntervalInSeconds=60
brokerServiceCompactionThresholdInBytes=0
brokerServiceCompactionPhaseOneLoopTimeInSeconds=30
delayedDeliveryEnabled=true
delayedDeliveryTickTimeMillis=1000
isDelayedDeliveryDeliverAtTimeStrict=false
delayedDeliveryFixedDelayDetectionLookahead=50000
acknowledgmentAtBatchIndexLevelEnabled=false
enableReplicatedSubscriptions=true
replicatedSubscriptionsSnapshotFrequencyMillis=1000
replicatedSubscriptionsSnapshotTimeoutSeconds=30
replicatedSubscriptionsSnapshotMaxCachedPerSubscription=10
maxMessagePublishBufferSizeInMB=
retentionCheckIntervalInSeconds=120
maxMessageSizeCheckIntervalInSeconds=60
maxNumPartitionsPerPartitionedTopic=0
zookeeperSessionExpiredPolicy=reconnect
systemTopicEnabled=false
systemTopicSchemaCompatibilityStrategy=ALWAYS_COMPATIBLE
topicLevelPoliciesEnabled=false
topicFencingTimeoutSeconds=0
proxyRoles=
authenticateOriginalAuthData=false
tlsCertRefreshCheckDurationSec=300
tlsCertificateFilePath=
tlsKeyFilePath=
tlsTrustCertsFilePath=
tlsAllowInsecureConnection=false
tlsHostnameVerificationEnabled=false
tlsProtocols=
tlsCiphers=
tlsRequireTrustedClientCertOnConnect=false
tlsProvider=
webServiceTlsProvider=Conscrypt
tlsEnabledWithKeyStore=false
tlsKeyStoreType=JKS
tlsKeyStore=
tlsKeyStorePassword=
tlsTrustStoreType=JKS
tlsTrustStore=
tlsTrustStorePassword=
brokerClientTlsEnabledWithKeyStore=false
brokerClientSslProvider=
brokerClientTlsTrustStoreType=JKS
brokerClientTlsTrustStore=
brokerClientTlsTrustStorePassword=
brokerClientTlsCiphers=
brokerClientTlsProtocols=
metadataStoreBatchingEnabled=true
metadataStoreBatchingMaxDelayMillis=5
metadataStoreBatchingMaxOperations=1000
metadataStoreBatchingMaxSizeKb=128
authenticationEnabled=false
authenticationProviders=
authenticationRefreshCheckSeconds=60
authorizationEnabled=false
authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
authorizationAllowWildcardsMatching=false
superUserRoles=
brokerClientTlsEnabled=false
brokerClientAuthenticationPlugin=
brokerClientAuthenticationParameters=
brokerClientTrustCertsFilePath=
athenzDomainNames=
anonymousUserRole=
basicAuthConf=
tokenSecretKey=
tokenPublicKey=
tokenAuthClaim=
tokenAudienceClaim=
tokenAudience=
saslJaasClientAllowedIds=.*pulsar.*
saslJaasServerSectionName=PulsarBroker
saslJaasServerRoleTokenSignerSecretPath=
httpMaxRequestSize=-1
disableHttpDebugMethods=false
httpRequestsLimitEnabled=false
httpRequestsMaxPerSecond=100.0
bookkeeperMetadataServiceUri=
bookkeeperClientAuthenticationPlugin=
bookkeeperClientAuthenticationParametersName=
bookkeeperClientAuthenticationParameters=
bookkeeperClientTimeoutInSeconds=30
bookkeeperClientNumWorkerThreads=
bookkeeperClientSpeculativeReadTimeoutInMillis=0
bookkeeperNumberOfChannelsPerBookie=16
bookkeeperUseV2WireProtocol=true
bookkeeperClientHealthCheckEnabled=true
bookkeeperClientHealthCheckIntervalSeconds=60
bookkeeperClientHealthCheckErrorThresholdPerInterval=5
bookkeeperClientHealthCheckQuarantineTimeInSeconds=1800
bookkeeperClientQuarantineRatio=1.0
bookkeeperClientGetBookieInfoIntervalSeconds=86400
bookkeeperClientGetBookieInfoRetryIntervalSeconds=60
bookkeeperClientRackawarePolicyEnabled=true
bookkeeperClientRegionawarePolicyEnabled=false
bookkeeperClientMinNumRacksPerWriteQuorum=2
bookkeeperClientEnforceMinNumRacksPerWriteQuorum=false
bookkeeperClientReorderReadSequenceEnabled=false
bookkeeperClientIsolationGroups=
bookkeeperClientSecondaryIsolationGroups=
bookkeeperClientMinAvailableBookiesInIsolationGroups=
bookkeeperEnableStickyReads=true
bookkeeperTLSProviderFactoryClass=org.apache.bookkeeper.tls.TLSContextFactory
bookkeeperTLSClientAuthentication=false
bookkeeperTLSKeyFileType=PEM
bookkeeperTLSTrustCertTypes=PEM
bookkeeperTLSKeyStorePasswordPath=
bookkeeperTLSTrustStorePasswordPath=
bookkeeperTLSKeyFilePath=
bookkeeperTLSCertificateFilePath=
bookkeeperTLSTrustCertsFilePath=
bookkeeperTlsCertFilesRefreshDurationSeconds=300
bookkeeper_tlsHostnameVerificationEnabled=false
bookkeeperDiskWeightBasedPlacementEnabled=false
bookkeeperExplicitLacIntervalInMills=0
managedLedgerDefaultEnsembleSize=2
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2
managedLedgerCursorPositionFlushSeconds=60
managedLedgerStatsPeriodSeconds=60
managedLedgerDigestType=CRC32C
managedLedgerNumWorkerThreads=
managedLedgerNumSchedulerThreads=
managedLedgerCacheSizeMB=
managedLedgerCacheCopyEntries=false
managedLedgerCacheEvictionWatermark=0.9
managedLedgerCacheEvictionIntervalMs=10
managedLedgerCacheEvictionTimeThresholdMillis=1000
managedLedgerCursorBackloggedThreshold=1000
managedLedgerDefaultMarkDeleteRateLimit=1.0
managedLedgerMaxEntriesPerLedger=50000
managedLedgerMinLedgerRolloverTimeMinutes=10
managedLedgerMaxLedgerRolloverTimeMinutes=240
managedLedgerInactiveLedgerRolloverTimeSeconds=0
managedLedgerMaxSizePerLedgerMbytes=2048
managedLedgerOffloadDeletionLagMs=14400000
managedLedgerOffloadAutoTriggerSizeThresholdBytes=-1
managedLedgerCursorMaxEntriesPerLedger=50000
managedLedgerCursorRolloverTimeInSeconds=14400
managedLedgerMaxUnackedRangesToPersist=10000
managedLedgerMaxUnackedRangesToPersistInZooKeeper=1000
autoSkipNonRecoverableData=false
lazyCursorRecovery=false
managedLedgerMetadataOperationsTimeoutSeconds=60
managedLedgerReadEntryTimeoutSeconds=0
managedLedgerAddEntryTimeoutSeconds=0
managedLedgerPrometheusStatsLatencyRolloverSeconds=60
managedLedgerTraceTaskExecution=true
managedLedgerNewEntriesCheckDelayInMillis=10
loadBalancerEnabled=true
loadBalancerReportUpdateThresholdPercentage=10
loadBalancerReportUpdateMinIntervalMillis=5000
loadBalancerReportUpdateMaxIntervalMinutes=15
loadBalancerHostUsageCheckIntervalMinutes=1
loadBalancerSheddingEnabled=true
loadBalancerSheddingIntervalMinutes=1
loadBalancerSheddingGracePeriodMinutes=30
loadBalancerBrokerMaxTopics=50000
loadBalancerBrokerOverloadedThresholdPercentage=85
loadBalancerResourceQuotaUpdateIntervalMinutes=15
loadBalancerAutoBundleSplitEnabled=true
loadBalancerAutoUnloadSplitBundlesEnabled=true
loadBalancerNamespaceBundleMaxTopics=1000
loadBalancerNamespaceBundleMaxSessions=1000
loadBalancerNamespaceBundleMaxMsgRate=30000
loadBalancerNamespaceBundleMaxBandwidthMbytes=100
loadBalancerNamespaceMaximumBundles=128
loadBalancerOverrideBrokerNicSpeedGbps=1000
loadManagerClassName=org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl
supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide
defaultNamespaceBundleSplitAlgorithm=range_equally_divide
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerBrokerThresholdShedderPercentage=10
loadBalancerMsgRateDifferenceShedderThreshold=50
loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold=4
loadBalancerHistoryResourcePercentage=0.9
loadBalancerBandwithInResourceWeight=1.0
loadBalancerBandwithOutResourceWeight=1.0
loadBalancerCPUResourceWeight=1.0
loadBalancerMemoryResourceWeight=1.0
loadBalancerDirectMemoryResourceWeight=1.0
loadBalancerBundleUnloadMinThroughputThreshold=10
namespaceBundleUnloadingTimeoutMs=60000
replicationMetricsEnabled=true
replicationConnectionsPerBroker=16
replicationProducerQueueSize=1000
replicatorPrefix=pulsar.repl
replicationPolicyCheckDurationSeconds=600
defaultRetentionTimeInMinutes=0
defaultRetentionSizeInMB=0
keepAliveIntervalSeconds=30
bootstrapNamespaces=
webSocketServiceEnabled=false
webSocketNumIoThreads=
webSocketConnectionsPerBroker=
webSocketSessionIdleTimeoutMillis=300000
webSocketMaxTextFrameSize=1048576
exposeTopicLevelMetricsInPrometheus=true
exposeConsumerLevelMetricsInPrometheus=false
exposeProducerLevelMetricsInPrometheus=false
exposeManagedLedgerMetricsInPrometheus=true
exposeManagedCursorMetricsInPrometheus=false
metricsServletTimeoutMs=30000
exposeBundlesMetricsInPrometheus=false
functionsWorkerEnabled=false
exposePublisherStats=true
statsUpdateFrequencyInSecs=60
statsUpdateInitialDelayInSecs=60
exposePreciseBacklogInPrometheus=false
splitTopicAndPartitionLabelInPrometheus=false
aggregatePublisherStatsByProducerName=false
schemaRegistryStorageClassName=org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorageFactory
isSchemaValidationEnforced=false
schemaCompatibilityStrategy=FULL
offloadersDirectory=./offloaders
managedLedgerOffloadDriver=
managedLedgerOffloadMaxThreads=2
managedLedgerOffloadPrefetchRounds=1
managedLedgerUnackedRangesOpenCacheSetEnabled=true
s3ManagedLedgerOffloadRegion=
s3ManagedLedgerOffloadBucket=
s3ManagedLedgerOffloadServiceEndpoint=
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864
s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576
gcsManagedLedgerOffloadRegion=
gcsManagedLedgerOffloadBucket=
gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864
gcsManagedLedgerOffloadReadBufferSizeInBytes=1048576
gcsManagedLedgerOffloadServiceAccountKeyFile=
managedLedgerOffloadBucket=
fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
fileSystemURI=
transactionCoordinatorEnabled=true
transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
transactionBufferSnapshotMaxTransactionCount=1000
transactionBufferSnapshotMinTimeInMillis=5000
transactionBufferClientMaxConcurrentRequests=1000
transactionPendingAckLogIndexMinLag=500
enablePackagesManagement=false
packagesManagementStorageProvider=org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider
packagesReplicas=1
packagesManagementLedgerRootPath=/ledgers
zookeeperServers=
globalZookeeperServers=
configurationStoreServers=
zooKeeperSessionTimeoutMillis=-1
zooKeeperOperationTimeoutSeconds=-1
zooKeeperCacheExpirySeconds=-1
replicationTlsEnabled=false
brokerServicePurgeInactiveFrequencyInSeconds=60
backlogQuotaDefaultLimitGB=-1
tlsEnabled=false
subscriptionKeySharedEnable=true
managedLedgerCacheEvictionFrequency=0

log file

pulsar-broker-ecs-69711187-002.novalocal.log-01-04-2024-1.log pulsar-broker-ecs-69711187-002.novalocal.log-01-10-2024-1.log

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!