[QUERY] The buffered producer took too long to start

[zhunipingan]

Library name and version

Azure.Messaging.EventHubs" Version="5.11.5"

Query/Question

Hi, I met this error when call the method in the SDK of eventhub. Azure.Messaging.EventHubs.EventHubsException(ServiceTimeout): The buffered producer took too long to start. (xxxxxx). For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot at Azure.Messaging.EventHubs.Producer.EventHubBufferedProducerClient.StartPublishingAsync(CancellationToken cancellationToken) at Azure.Messaging.EventHubs.Producer.EventHubBufferedProducerClient.EnqueueEventsAsync(IEnumerable1 events, EnqueueEventOptions options, CancellationToken cancellationToken) at MYSERVICE.EventHubBufferedPublisher.SendAsync[T](IEnumerable1 events, String partitionKey, CancellationToken cancellationToken)

I think it is one transient problem, not all time I will get this error. I want to know If I get this error, will the event will be lost. In this scenario, it seems the SendEventBatchFailedAsync is not called. Under what circumstances will the FailedAsync or SucceedAsync methods be called? When will the retrytime we configured be used? Does this scenario experience retry? Do I need to customize a retry at the outermost level?

Appreciate for your help.

Environment

No response

[github-actions[bot]]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

Hi @zhunipingan. Thanks for reaching out and we regret that you're experiencing difficulties. This error generally means that the producer was unable to establish a connection to the Event Hubs service. Often this is due to the AMQP ports (5671, 5672), being blocked in your environment, an intercepting proxy being present, or transient network failures.

The events that were enqueued are not lost in this case. They are still held in the internal buffers and will be sent once a connection can be established.

[github-actions[bot]]

Hi @zhunipingan. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[zhunipingan]

@jsquire thank you. But how can I know whether the events were enqueued successfully. I just get the exception, but I dont get any other info. If there is some event loss, I will get other exceptions?

I also have another question, In this scenario, it seems the SendEventBatchFailedAsync is not called. Under what circumstances will the FailedAsync or SucceedAsync methods be called? Thank you!

[jsquire]

The contract for the enqueue methods is straightforward and simple. If the call throws, the event was not enqueued. If the call returns successfully, the event was enqueued and the producer has responsibility for it.

You won't see the send success/fail handlers invoked when your producer fails to start. This is for two reasons:

1. Your event was not enqueued successfully, so there's nothing to attempt a send.
2. The producer is not running, as it failed to start.

Once an event is successfully enqueued, the producer is confirmed running, and you'll see the handlers fire each time a send is attempted.

[zhunipingan]

Thanks! @jsquire According to your explanation, for the exception I met, the reason is the event was not enqueued successfully, so the event will be lost. How can I do something to avoid the loss, should I add another retry when I call the enqueue() method? When will the retrytime we configured be used? Does this scenario experience retry? Do I need to customize a retry at the outermost level?

[jsquire]

@zhunipingan : All Azure SDK clients implicitly retry failures that are not considered terminal according to the configured retry policy. If your application sees an error surface, that indicates that we have exhausted all eligible retries. There's usually no value in implementing a second-level retry versus changing your built-in retry configuration.

How can I do something to avoid the loss, should I add another retry when I call the enqueue() method?

This is something you'll have to decide based on what is right for your application. From the SDK perspective, you're welcome to retry calls as you'd like. However, each call (and any implicit retries) are distinct to the client. It does not have any awareness of past calls and their success; your application would need to understand "every single time that I've called this it has failed. This isn't transient and I should go figure out what's wrong in my app/environment."

[github-actions[bot]]

Hi @zhunipingan, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.