pulsar-helm-chart Ledgers not being offloaded to AWS S3 during testing

Describe the bug Using 4.1.0 version of this helm chart. Everything appears correct on the broker's env:

s3ManagedLedgerOffloadServiceEndpoint=https://s3.eu-west-1.amazonaws.com
AWS_DEFAULT_REGION=eu-west-1
AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::111111111111:role/eks-pulsar
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
managedLedgerOffloadDriver=aws-s3
AWS_STS_REGIONAL_ENDPOINTS=regional
s3ManagedLedgerOffloadRegion=eu-west-1
offloadersDirectory=/pulsar/offloaders
PULSAR_PREFIX_managedLedgerOffloadThresholdInBytes=10
loggerLevelUpdates=org.apache.bookkeeper.mledger.offload=DEBUG,org.jclouds=DEBUG
PULSAR_PREFIX_managedLedgerOffloadDeletionLagInMillis=0
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=mybucketishere

To Reproduce Steps to reproduce the behavior: Use these broker settings (warning: settings are set to offload fast for testing)

broker:
  podSecurityContext:   # << testing may not be needed. please advise
    enabled: true
    fsGroup: 10000
  service_account:
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/eks-pulsar
  storageOffload:
    driver: aws-s3
    bucket: "had-no-choice-but-to-hardcode-no-extraenvs-worked-in-these-keys"
    region: "had-no-choice-but-to-hardcode-no-extraenvs-worked-in-these-key"
    managedLedgerOffloadAutoTriggerSizeThresholdBytes: "10"
    managedLedgerOffloadDeletionLagMs: "0"
  configData:
    acknowledgmentAtBatchIndexLevelEnabled: "true"
    brokerDeleteInactiveTopicsEnabled: "false"
    offloadersDirectory: "/pulsar/offloaders". # << may not be needed
    managedLedgerMinLedgerRolloverTimeMinutes: "5" 
    managedLedgerMaxLedgerSize: "256M"
    s3ManagedLedgerOffloadServiceEndpoint: "https://s3.eu-west-1.amazonaws.com"
    loggerLevelUpdates: "org.apache.bookkeeper.mledger.offload=DEBUG,org.jclouds=DEBUG"

Also, extraenvvars for broker did not work for me for anything in the broker key, but I may create another ticket for that as it's much less blocking.

Expected behavior It should offload within 5 minutes when a ledger is closed to my knowledge. I tried running commands to roll it over manually and it never rolled over. I tried consuming and it didn't make a difference.

Logs

Offload triggered for persistent://public/default/offload-test-3 for messages before 793:0:-1

2025-08-01T04:24:39,760+0000 [pulsar-web-40-5] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/offload-test-3] Starting offload operation at messageId 793:0:-1

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/offload-test-3] Start ledgersOffload. ledgers=[793] totalSize=842208

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/offload-test-3] No ledgers to offload

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/offload-test-3] Completed successfully offload operation at messageId 793:0:-1

2025-08-01T04:24:39,762+0000 [pulsar-web-40-5] INFO  org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [01/Aug/2025:04:24:39 +0000] "PUT /admin/v2/persistent/public/default/offload-test-3/offload HTTP/1.1" 204 0 "-" "Pulsar-Java-v4.0.5" 4

Desktop (please complete the following information):

OS: [e.g. iOS]

Additional context Would appreciate any help here. I can also definitely contribute to the env vars. IMHO extraenvvars for the entire chart would make it very malleable and repeatable for people with many clusters.

Aug 01 '25 05:08 oren-cohen

The topic stats and internal stats would be useful. One possible reason could be that the messages aren't retained.

In Pulsar, messages get discarded if there isn't any subscription or the message retention isn't configured. Have you checked that already?

Aug 01 '25 07:08 lhotari

It looks like the topic is empty. As if it offloaded but nothing send to s3..so...

I have a subscription and I am constantly flooding it with messages now and still no s3 transfers.

Topic stats:

{ "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesInCounter" : 0, "msgInCounter" : 0, "systemTopicBytesInCounter" : 0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "bytesOutInternalCounter" : 0, "averageMsgSize" : 0.0, "msgChunkPublished" : false, "storageSize" : 0, "backlogSize" : 0, "backlogQuotaLimitSize" : -1, "backlogQuotaLimitTime" : -1, "oldestBacklogMessageAgeSeconds" : -1, "publishRateLimitedTimes" : 0, "earliestMsgPublishTimeInBacklogs" : 0, "offloadedStorageSize" : 0, "lastOffloadLedgerId" : 0, "lastOffloadSuccessTimeStamp" : 0, "lastOffloadFailureTimeStamp" : 0, "ongoingTxnCount" : 0, "abortedTxnCount" : 0, "committedTxnCount" : 0, "publishers" : [ ], "waitingPublishers" : 0, "subscriptions" : { "wallah" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "msgRateRedeliver" : 0.0, "messageAckRate" : 0.0, "chunkedMessageRate" : 0.0, "msgBacklog" : 0, "backlogSize" : 0, "earliestMsgPublishTimeInBacklog" : 0, "msgBacklogNoDelayed" : 0, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "msgInReplay" : 0, "unackedMessages" : 0, "type" : "Exclusive", "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 1754024182067, "lastConsumedTimestamp" : 0, "lastAckedTimestamp" : 0, "lastMarkDeleteAdvancedTimestamp" : 0, "consumers" : [ ], "isDurable" : true, "isReplicated" : false, "allowOutOfOrderDelivery" : false, "consumersAfterMarkDeletePosition" : { }, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "subscriptionProperties" : { }, "filterProcessedMsgCount" : 0, "filterAcceptedMsgCount" : 0, "filterRejectedMsgCount" : 0, "filterRescheduledMsgCount" : 0, "dispatchThrottledMsgEventsBySubscriptionLimit" : 0, "dispatchThrottledBytesEventsBySubscriptionLimit" : 0, "dispatchThrottledMsgEventsByTopicLimit" : 0, "dispatchThrottledBytesEventsByTopicLimit" : 0, "dispatchThrottledMsgEventsByBrokerLimit" : 0, "dispatchThrottledBytesEventsByBrokerLimit" : 0, "durable" : true, "replicated" : false } }, "replication" : { }, "deduplicationStatus" : "Disabled", "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "compaction" : { "lastCompactionRemovedEventCount" : 0, "lastCompactionSucceedTimestamp" : 0, "lastCompactionFailedTimestamp" : 0, "lastCompactionDurationTimeInMills" : 0 }, "ownerBroker" : "pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:8080" }

Internal Stats:

{ "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesInCounter" : 0, "msgInCounter" : 0, "systemTopicBytesInCounter" : 0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "bytesOutInternalCounter" : 0, "averageMsgSize" : 0.0, "msgChunkPublished" : false, "storageSize" : 0, "backlogSize" : 0, "backlogQuotaLimitSize" : -1, "backlogQuotaLimitTime" : -1, "oldestBacklogMessageAgeSeconds" : -1, "publishRateLimitedTimes" : 0, "earliestMsgPublishTimeInBacklogs" : 0, "offloadedStorageSize" : 0, "lastOffloadLedgerId" : 0, "lastOffloadSuccessTimeStamp" : 0, "lastOffloadFailureTimeStamp" : 0, "ongoingTxnCount" : 0, "abortedTxnCount" : 0, "committedTxnCount" : 0, "publishers" : [ ], "waitingPublishers" : 0, "subscriptions" : { "wallah" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "msgRateRedeliver" : 0.0, "messageAckRate" : 0.0, "chunkedMessageRate" : 0.0, "msgBacklog" : 0, "backlogSize" : 0, "earliestMsgPublishTimeInBacklog" : 0, "msgBacklogNoDelayed" : 0, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "msgInReplay" : 0, "unackedMessages" : 0, "type" : "Exclusive", "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 1754024182067, "lastConsumedTimestamp" : 0, "lastAckedTimestamp" : 0, "lastMarkDeleteAdvancedTimestamp" : 0, "consumers" : [ ], "isDurable" : true, "isReplicated" : false, "allowOutOfOrderDelivery" : false, "consumersAfterMarkDeletePosition" : { }, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "subscriptionProperties" : { }, "filterProcessedMsgCount" : 0, "filterAcceptedMsgCount" : 0, "filterRejectedMsgCount" : 0, "filterRescheduledMsgCount" : 0, "dispatchThrottledMsgEventsBySubscriptionLimit" : 0, "dispatchThrottledBytesEventsBySubscriptionLimit" : 0, "dispatchThrottledMsgEventsByTopicLimit" : 0, "dispatchThrottledBytesEventsByTopicLimit" : 0, "dispatchThrottledMsgEventsByBrokerLimit" : 0, "dispatchThrottledBytesEventsByBrokerLimit" : 0, "durable" : true, "replicated" : false } }, "replication" : { }, "deduplicationStatus" : "Disabled", "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "compaction" : { "lastCompactionRemovedEventCount" : 0, "lastCompactionSucceedTimestamp" : 0, "lastCompactionFailedTimestamp" : 0, "lastCompactionDurationTimeInMills" : 0 }, "ownerBroker" : "pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:8080" }

And during flooding, stats:

pulsar-admin --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params "token:${PULSAR_ADMIN_TOKEN}" topics stats persistent://public/default/offload-test-5 { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 68.99998524895315, "msgThroughputOut" : 4284.515750708608, "bytesInCounter" : 3021708, "msgInCounter" : 49000, "systemTopicBytesInCounter" : 0, "bytesOutCounter" : 2932073, "msgOutCounter" : 47504, "bytesOutInternalCounter" : 0, "averageMsgSize" : 0.0, "msgChunkPublished" : false, "storageSize" : 3021708, "backlogSize" : 3021708, "backlogQuotaLimitSize" : -1, "backlogQuotaLimitTime" : -1, "oldestBacklogMessageAgeSeconds" : 1515, "oldestBacklogMessageSubscriptionName" : "wallah", "publishRateLimitedTimes" : 0, "earliestMsgPublishTimeInBacklogs" : 0, "offloadedStorageSize" : 0, "lastOffloadLedgerId" : 0, "lastOffloadSuccessTimeStamp" : 0, "lastOffloadFailureTimeStamp" : 1754057114270, "ongoingTxnCount" : 0, "abortedTxnCount" : 0, "committedTxnCount" : 0, "publishers" : [ ], "waitingPublishers" : 0, "subscriptions" : { "oc" : { "msgRateOut" : 68.99998524895315, "msgThroughputOut" : 4284.515750708608, "bytesOutCounter" : 2932073, "msgOutCounter" : 47504, "msgRateRedeliver" : 0.0, "messageAckRate" : 69.18331873911225, "chunkedMessageRate" : 0.0, "msgBacklog" : 0, "backlogSize" : 0, "earliestMsgPublishTimeInBacklog" : 0, "msgBacklogNoDelayed" : 0, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "msgInReplay" : 0, "unackedMessages" : 0, "type" : "Exclusive", "activeConsumerName" : "Mg6JN", "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 1754058624027, "lastConsumedTimestamp" : 1754058626885, "lastAckedTimestamp" : 1754058626906, "lastMarkDeleteAdvancedTimestamp" : 1754058626906, "consumers" : [ { "appId" : "admin", "msgRateOut" : 68.99998524895315, "msgThroughputOut" : 4284.515750708608, "bytesOutCounter" : 2781984, "msgOutCounter" : 44996, "msgRateRedeliver" : 0.0, "messageAckRate" : 69.18331873911225, "chunkedMessageRate" : 0.0, "consumerName" : "Mg6JN", "availablePermits" : 504, "unackedMessages" : 0, "avgMessagesPerEntry" : 1, "blockedConsumerOnUnackedMsgs" : false, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "address" : "/172.16.124.11:43604", "connectedSince" : "2025-08-01T14:11:01.081608912Z", "clientVersion" : "Pulsar-Java-v4.0.5", "lastAckedTimestamp" : 1754058626906, "lastConsumedTimestamp" : 1754058626885, "lastConsumedFlowTimestamp" : 1754058624027, "metadata" : { }, "lastAckedTime" : "2025-08-01T14:30:26.906Z", "lastConsumedTime" : "2025-08-01T14:30:26.885Z" } ], "isDurable" : true, "isReplicated" : false, "allowOutOfOrderDelivery" : false, "consumersAfterMarkDeletePosition" : { }, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "subscriptionProperties" : { }, "filterProcessedMsgCount" : 0, "filterAcceptedMsgCount" : 0, "filterRejectedMsgCount" : 0, "filterRescheduledMsgCount" : 0, "dispatchThrottledMsgEventsBySubscriptionLimit" : 0, "dispatchThrottledBytesEventsBySubscriptionLimit" : 0, "dispatchThrottledMsgEventsByTopicLimit" : 0, "dispatchThrottledBytesEventsByTopicLimit" : 0, "dispatchThrottledMsgEventsByBrokerLimit" : 0, "dispatchThrottledBytesEventsByBrokerLimit" : 0, "durable" : true, "replicated" : false }, "wallah" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "msgRateRedeliver" : 0.0, "messageAckRate" : 0.0, "chunkedMessageRate" : 0.0, "msgBacklog" : 49000, "backlogSize" : 3021708, "earliestMsgPublishTimeInBacklog" : 0, "msgBacklogNoDelayed" : 49000, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "msgInReplay" : 0, "unackedMessages" : 0, "type" : "Exclusive", "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 1754024182067, "lastConsumedTimestamp" : 0, "lastAckedTimestamp" : 0, "lastMarkDeleteAdvancedTimestamp" : 0, "consumers" : [ ], "isDurable" : true, "isReplicated" : false, "allowOutOfOrderDelivery" : false, "consumersAfterMarkDeletePosition" : { }, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "subscriptionProperties" : { }, "filterProcessedMsgCount" : 0, "filterAcceptedMsgCount" : 0, "filterRejectedMsgCount" : 0, "filterRescheduledMsgCount" : 0, "dispatchThrottledMsgEventsBySubscriptionLimit" : 0, "dispatchThrottledBytesEventsBySubscriptionLimit" : 0, "dispatchThrottledMsgEventsByTopicLimit" : 0, "dispatchThrottledBytesEventsByTopicLimit" : 0, "dispatchThrottledMsgEventsByBrokerLimit" : 0, "dispatchThrottledBytesEventsByBrokerLimit" : 0, "durable" : true, "replicated" : false } }, "replication" : { }, "deduplicationStatus" : "Disabled", "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "compaction" : { "lastCompactionRemovedEventCount" : 0, "lastCompactionSucceedTimestamp" : 0, "lastCompactionFailedTimestamp" : 0, "lastCompactionDurationTimeInMills" : 0 }, "ownerBroker" : "pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:8080" }

Internal stats:

{ "entriesAddedCounter" : 55500, "numberOfEntries" : 55500, "totalSize" : 3426254, "currentLedgerEntries" : 5499, "currentLedgerSize" : 342410, "lastLedgerCreatedTimestamp" : "2025-08-01T14:30:47.189Z", "waitingCursorsCount" : 1, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "851:5498", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 843, "entries" : 1, "size" : 59, "offloaded" : false, "underReplicated" : false }, { "ledgerId" : 849, "entries" : 50000, "size" : 3083785, "offloaded" : false, "underReplicated" : false }, { "ledgerId" : 851, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "oc" : { "markDeletePosition" : "851:5498", "readPosition" : "851:5499", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 55500, "cursorLedger" : 850, "cursorLedgerLastEntry" : 421, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2025-08-01T14:08:09.063Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "wallah" : { "markDeletePosition" : "843:-1", "readPosition" : "843:0", "waitingReadOp" : false, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2025-08-01T04:56:22.042Z", "state" : "NoLedger", "active" : false, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : false, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } }

Aug 01 '25 14:08 oren-cohen

It looks like the topic is empty. As if it offloaded but nothing send to s3..so...

I have a subscription and I am constantly flooding it with messages now and still no s3 transfers.

The offloading won't happen until the topic's active ledger rolls over to a new one. Here are some of the related settings: https://github.com/apache/pulsar/blob/dedba1a9bf0becb1b3f80f1d89a779451734fbe6/conf/broker.conf#L1247-L1289 docs: https://pulsar.apache.org/docs/4.0.x/tiered-storage-aws/

Aug 05 '25 12:08 lhotari

Hello, thanks for responding. I tried to manually close the active ledger but still it didn't post any data to s3. What would be a good way to test this if not doing that?

Aug 09 '25 20:08 oren-cohen

Hello, thanks for responding. I tried to manually close the active ledger but still it didn't post any data to s3. What would be a good way to test this if not doing that?

Have you seen https://pulsar.apache.org/docs/4.0.x/tiered-storage-aws/#configure-aws-s3-offloader-to-run-manually ?

Aug 11 '25 12:08 lhotari

Yes, I have tried that before. Just did it again. Every time I trigger it I get the same message. So it's not actually offloading anything.

Offload triggered for persistent://public/api/requests for messages before 1064:0:-1

No errors in log and nothing appeared in s3. This is the log I saw when I triggered it:

2025-08-11T16:12:36,718+0000 [pulsar-web-40-4] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [11/Aug/2025:16:12:36 +0000] "PUT /admin/v2/persistent/public/api/requests/offload HTTP/1.1" 307 0 "-" "Pulsar-Java-v4.0.5" 24

Edit: My config is:

[conf/broker.conf] Applying config managedLedgerOffloadDriver = aws-s3 [conf/broker.conf] Applying config offloadersDirectory = /pulsar/offloaders [conf/broker.conf] Applying config s3ManagedLedgerOffloadBucket = pulsar-offload-bucket [conf/broker.conf] Applying config s3ManagedLedgerOffloadRegion = eu-west-1 [conf/broker.conf] Applying config s3ManagedLedgerOffloadServiceEndpoint = https://s3.eu-west-1.amazonaws.com [conf/broker.conf] Adding config managedLedgerOffloadDeletionLagInMillis = 0 [conf/broker.conf] Adding config managedLedgerOffloadThresholdInBytes = 1

Aug 11 '25 16:08 oren-cohen

Hello, I have not heard back and this issue is still open? Any suggestions on how to test s3 offloading?

Sep 01 '25 00:09 oren-cohen

Hello, I have not heard back and this issue is still open? Any suggestions on how to test s3 offloading?

managedLedgerMinLedgerRolloverTimeMinutes: "5" 
managedLedgerMaxLedgerSize: "256M"

It should offload within 5 minutes when a ledger is closed to my knowledge. I tried running commands to roll it over manually and it never rolled over. I tried consuming and it didn't make a difference.

In addition, you would need to set managedLedgerMaxLedgerRolloverTimeMinutes to be "5" if you want to rollover within 5 minutes. For testing, you can set managedLedgerMinLedgerRolloverTimeMinutes: "0" so that it's not preventing ledger rollover.

The rules for rollover isn't very obvious, it's explained here: https://github.com/apache/pulsar/blob/0903f45b9cfd97816545f8210d8e6b7dab9cdd39/conf/broker.conf#L1307-L1319

A ledger rollover is triggered after the min rollover time has passed and one of the following conditions is true:

The max rollover time has been reached

The max entries have been written to the ledger

The max ledger size has been written to the ledger

For various commands, you can see how s3 offloading is tested in Pulsar integration tests: https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/offload/TestS3Offload.java https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/offload/TestBaseOffload.java#L40

Sep 16 '25 14:09 lhotari

Thanks for the help. It got me to find this error:

KarpenterNodeRole-stg/i-0d5d3f2396b3ca609 is not authorized to perform: s3:PutObject on resource: "arn:aws:s3::offload-bucket

So I realized the Service account it was using is incorrect. When I tried to specify a service account in the broker config in helm it did not work properly FYI:

broker:
  service_account:
    name: pulsar-sa
    annotations:
      eks.amazonaws.com/role-arn:<my role arn>

It still was using this sa which I found in kubernetes: pulsar-broker-acct

Once I added that to my AWS trust policy for my role it finally worked!

Just curious why when I specify a service account for the broker and restart it, it does not pick it up or create the SA in the namespace?

Sep 19 '25 18:09 oren-cohen