[V2] Unable to reconnect to iothub after long offline period (> 1 hour)

[villepalo]

After more than one hour disconnection, connected is not established anymore. This happens occasionally, not always when I try.

Host OS: Ubuntu 14.04

2018-02-26 17:20:40.538 +00:00 [ERR] - Unable to authenticate device test-device/ingester Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Error updating identity for device test-device/ingester ---> System.AggregateException: One or more errors occurred. (Operation timeout expired.) (Unable to connect to the remote server) ---> System.TimeoutException: Operation timeout expired. at Microsoft.Azure.Devices.Client.DeviceClient.<>c__DisplayClass79_0.<ApplyTimeout>b__2(Task t) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.<CreateAndOpenDeviceClient>d__23.MoveNext() in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 178 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.Edge.Util.Fallback.<ExecuteAsync>d__3`1.MoveNext() --- End of inner exception stack trace --- at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.<ConnectToIoTHub>d__22.MoveNext() in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 153 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.<GetCloudProxyAsync>d__21.MoveNext() in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 146 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.<>c__DisplayClass20_0.<<CreateOrUpdateAsync>b__1>d.MoveNext() in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 109 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection.<CreateOrUpdateAsync>d__20.MoveNext() in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudConnection.cs:line 135 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager.<>c__DisplayClass25_0.<<CreateOrUpdateCloudConnection>b__0>d.MoveNext() in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/ConnectionManager.cs:line 115 --- End of inner exception stack trace --- 2018-02-26 17:20:40.543 +00:00 [ERR] - Unable to generate identity for clientId test-device/ingester and username virtualbox/test-device/ingester/api-version=2017-06-30&DeviceClientType=Microsoft.Azure.Devices.Client%2F1.6.0-preview-001%20%28.NET%20Core%204.6.00001.0%3B%20Linux%204.4.0-112-generic%20%23135~14.04.1-Ubuntu%20SMP%20Tue%20Jan%2023%2020%3A41%3A48%20UTC%202018%3B%20X64%29 2018-02-26 17:20:40.543 +00:00 [INF] - ClientNotAuthenticated, Client ID: test-device/ingester; Username: virtualbox/test-device/ingester/api-version=2017-06-30&DeviceClientType=Microsoft.Azure.Devices.Client%2F1.6.0-preview-001%20%28.NET%20Core%204.6.00001.0%3B%20Linux%204.4.0-112-generic%20%23135~14.04.1-Ubuntu%20SMP%20Tue%20Jan%2023%2020%3A41%3A48%20UTC%202018%3B%20X64%29, 76b31fee 2018-02-26 17:20:41.006 +00:00 [INF] - Attempting to connect to IoT Hub for client test-device/ingester via AMQP...

[avranju]

Thanks for reporting this. We'll investigate.

[sjkp]

@avranju Any status on this, we are running field tests now. And this is killing our gateways connectivity, quite regularly, when network conditions are poor, which I assumed was one of the main motivators for using IoT Edge.

[chipalost]

Hi guys, have you tried reproing this on GA bits? The change in our architecture may have addressed this issue.