iothub icon indicating copy to clipboard operation
iothub copied to clipboard

Connection drops after a while, how to refresh it

Open tappoz opened this issue 4 years ago • 3 comments

I'm creating a cloud-to-device link via AMQP with:

ihQueueClient, err := iotservice.NewFromConnectionString(iotHubConnStr)

I can use this successfully for 1 hour or so, but then after that I start seeing:

*Error{Condition: amqp:unauthorized-access, Description: Token or Certificate is invalid., Info: map[com.microsoft:is-filtered:true com.microsoft:tracking-id:<<<TRACKING_ID>>>-G:6-TimeStamp:02/10/2021 16:05:08]}

Looks like the AMQP connection is being dropped due to inactivity or just because 1 hour is passed after it has been created. Is there any way I can avoid this? How can I specify to automatically refresh or reconnect to the IoT Hub?

I can see in here https://github.com/amenzhinsky/iothub/blob/e718b48d126cf72b9696022028762ccd1d9ee1fe/iotservice/client.go#L44

there's this TLS option, but it's not exactly related to timeouts, that's more about extra security checks when using TLS:

// WithTLSConfig sets TLS config that's used by REST HTTP and AMQP clients.
func WithTLSConfig(config *tls.Config) ClientOption {
	return func(c *Client) {
		c.tls = config
	}
}

tappoz avatar Feb 11 '21 11:02 tappoz

If I recall correctly we had this problem before, the client should maintain token actuality in the background:

https://github.com/amenzhinsky/iothub/blob/master/iotservice/client.go#L147

amenzhinsky avatar Feb 12 '21 08:02 amenzhinsky

There is this stack of calls:

  • SendEvent https://github.com/amenzhinsky/iothub/blob/master/iotservice/client.go#L400
  • getSendLink https://github.com/amenzhinsky/iothub/blob/master/iotservice/client.go#L419
  • newSession https://github.com/amenzhinsky/iothub/blob/master/iotservice/client.go#L433
  • putToken https://github.com/amenzhinsky/iothub/blob/master/iotservice/client.go#L162
  • putTokenContinuously https://github.com/amenzhinsky/iothub/blob/master/iotservice/client.go#L132

That is not using consistently a Go context - it is recreating in a few different places context.Background(). It would be useful to be able to create a Go context with a timeout so we can invalidate a session. Also these are mostly unexposed so I can not call them from outside the Go package where they are written.

This does not allow to refresh the IoT Hub token and the effect is that having a long running process that is supposed to be constantly connected to the IoT Hub to send "could-to-device" messages does not work. This long running process stops working after around 1 hour from when it was started.

I ended up working around this issue with:

func (ms *myStruct) IotHubRefreshConnection(iotHubConnStr string) {
	if ms.ihsClient != nil {
		log.Info.Println("Closing the IoT Hub connection...")
		ms.ihsClient.Close()
	}
	log.Printf("Connecting to the IoT Hub with: %v", iotHubConnStr)
	ihQueueClient, err := iotservice.NewFromConnectionString(
		iotHubConnStr,
		iotservice.WithLogger(
			logger.New(
				logger.LevelDebug,
				func(lvl logger.Level, s string) {
					log.Println(fmt.Sprint("*IH-CLIENT* ", lvl.String(), " ", s))
				},
			),
		),
	)
	if err != nil {
		log.Error.Printf("Cannot establish the IoT Hub connection: %+v", err)
		panic(err)
	}
	ms.ihsClient = ihQueueClient
}

func syncRefreshSchedule(ms *myStruct, iotHubConnStr string) {
	for {
		log.Info.Println("Wating to refresh the Iot Hub connection...")
		select {
		case <-time.After(10 * time.Minute):
			log.Info.Println("Recurrent time to refresh the IoT Hub connection...")
			ms.IotHubRefreshConnection(iotHubConnStr)
		}
	}
}

Then in my constructor of myStruct I do this:

ms.IotHubRefreshConnection(iotHubConnStr)
go syncRefreshSchedule(&ms, iotHubConnStr)

This way I make sure there is always a working IoT Hub session. Even after 1 hour.

tappoz avatar Feb 15 '21 14:02 tappoz

Can you try the latest master, there was simply a defer error that closed amqp session before it tired to renew token.

amenzhinsky avatar Feb 28 '21 14:02 amenzhinsky