amqp evensource not working with single log "started processing the AMQP event source..."

[ArieLevs]

Describe the bug similar to #1545, the issue still happening

no error message, nothing crashing, logs from event source:

namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=starting eventsource server
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=starting metrics server
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=Not the LEADER, stand by ...
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=Becoming a Candidate, stand by ...
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=I'm the LEADER, starting ...
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=Starting event source server...
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=eventBusConfig: {NATS:&NATSConfig{URL:nats://eventbus-default-stan-svc:4222,ClusterID:*eventbus-de
fault,Auth:*token,AccessSecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:eventbus-default-client,},Key:client-auth,Optional:nil,},} JetStream:nil}
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=NATS auth strategy: Token
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=Connected to NATS server.
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=Connected to NATS streaming server.
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=Eventing server started.
namespace=classifier, eventSourceName=rabbitmq-develop, level=info, time=2022-07-25T20:44:48Z, msg=starting eventbus connection daemon...
namespace=classifier, eventSourceName=rabbitmq-develop, eventSourceType=amqp, eventName=newTask, level=info, time=2022-07-25T20:44:48Z, msg=started processing the AMQP event source...
namespace=classifier, eventSourceName=rabbitmq-develop, eventSourceType=amqp, eventName=newTask, level=info, time=2022-07-25T20:44:48Z, msg=started processing the AMQP event source...
namespace=classifier, eventSourceName=rabbitmq-develop, eventSourceType=amqp, eventName=newTask, level=info, time=2022-07-25T20:44:48Z, msg=started processing the AMQP event source...
namespace=classifier, eventSourceName=rabbitmq-develop, eventSourceType=amqp, eventName=newTask, level=info, time=2022-07-25T20:44:48Z, msg=started processing the AMQP event source...

EventSource resource looks like:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: rabbitmq-develop
spec:
  amqp:
    newTask:
      url: amqps://****:5671
      jsonBody: true
      exchangeName: "test"
      exchangeType: direct
      routingKey: hello
      connectionBackoff:
        duration: 10s
        steps: 5
        factor: 2
        jitter: 0.2
      exchangeDeclare:
        durable: true
        autoDelete: false
        internal: false
        noWait: false
      queueDeclare:
        name: "test"
        durable: true
        autoDelete: false
        exclusive: true
        noWait: false
      auth:
        username:
          name: rabbitmq-secrets
          key: username
        password:
          name: rabbitmq-secrets
          key: password

Expected behavior Connection succeeds or an error is thrown.

Environment:

• Kubernetes: v1.22.9
• Argo: None
• Argo Events: v1.7.1

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[whynowy]

It should be something wrong with your connection to AMQP, but it's bad not to display the logs...

[ArieLevs]

I agree but I have no way of investigating what is wrong :sweat_smile: wrong pass, timeouts, cert validation etc, absence of an errors in logs makes debugging really hard

[whynowy]

Agreed.

[vermaxik]

The same to me, did you @ArieLevs find a way how to debug it?

[whynowy]

There's a bug in the logging mechanism during retry, I'll put a fix there, so that it will be easier for debugging.

[ArieLevs]

Hi @vermaxik unfortunately not, this issue blocked my poc with argo workflows (still using self implemented solution using Keda)

[maxisam]

I got this issue as well, I don't see any error logs with v1.7.2

[whynowy]

@maxisam - could you post all the logs you can see?

[maxisam]

it is identical with the original post. I did solve my issue in the end by checking RabbitMQ's log. Apparently I set the exclusive flag as true but it is not set as true in queue. After I fix the configuration it works. However, there is no error from argo-event. I think it might be caused by the upstream amqp library.

[ArieLevs]

@vermaxik I'm not sure what changed from my last attempt (since there are still no logs with version 1.7.2) but I got it working using latest version, might be related to the issue @maxisam raised as I also set exclusive: false

[whynowy]

@maxisam @ArieLevs - Thank you both, I'll keep looking into the log issue.