Telemetry data not received by IoT Hub

[gillermaney]

The tutorial page at: https://azure.github.io/Industrial-IoT/tutorials/tut-publish-data.html Includes the text below. Besides the text being a bit off, after performing these steps, telemetry data is not being received by IoT Hub

Next steps Once the configuration ofr the published data is done, the endpoints may be disconnected from the engineering tool. Please not though that endpoints need to be reconnected, so that browsing is possible, when modifications to the publishing data-points are required.

The telemetry data from the OPC UA Servers on the field is now flowing into the IIoT platform and is stored in the TSI storage respective in the Azure DataLake Gen 2 in CDM format.

[marcschier]

We are working to add better diagnostics in milestone 2.8 so we can understand what is happening. In the meantime, the version of the engineering tool (displayed in the UX) would be very helpful for us to try and reproduce.

[gillermaney]

Using the June 1st canned deployment script unchanged and deploying to a variety of regions, the "device to cloud messages" displayed on the bottom of the IoT Hub page shows spikes every 6 hours, but no messages between spikes. There are 5 variables being published at either 1 or 5 second intervals according to the engineering tool. The height of each spike is about 35 messages. Device twin operations count fluctuates around 30. The telemetry data lake does not have any containers. The data lake ingress graph shows peeks every hour, but the graph is a dotted line. Of course there is no model.json file as no container was created. TSI shows no ingress at all since deploying.

[cristipogacean]

@gillermaney this looks like a connectivity issue between the publisher module and the opc ua server you're trying to connect to. Can you have look on the publisher's logs and check if there are issues there?

[gillermaney]

I am using the simulators that are deployed as part of the canned deployment script.

[cristipogacean]

you can try to restart the simulation VMs. if that does not help, please send us the publisher logs

[gillermaney]

I have reset the VMs in the past and that didn't work. Can you see the publisher logs in Application Insights? If so, what is the path to get there? Here is a reoccurring error from Application Insight Exceptions:

6/3/2020, 6:57:02.405 PM	Microsoft.Azure.EventHubs.ReceiverDisconnectedException at Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<OnReceiveAsync>d__13.MoveNext	Microsoft.Azure.EventHubs.ReceiverDisconnectedException	Microsoft.Azure.EventHubs, Version=4.2.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c	Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<OnReceiveAsync>d__13.MoveNext	Microsoft.Azure.EventHubs.ReceiverDisconnectedException	New receiver 'host-920c6f86-b9d7-49d5-884c-a74671edb94d' with higher epoch of '2' is created hence current receiver 'host-6f385b94-bb66-4e1a-9450-43535abb172a' with epoch '1' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:f9764352000339ec001b72ec5ed7f247_G11_B32, SystemTracker:eventhubnamespace-mw7uby:eventhub:eventhub-mw7uby~8191|telemetryux, Timestamp:2020-06-03T18:57:02	Microsoft.Azure.EventHubs, Version=4.2.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c	Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<OnReceiveAsync>d__13.MoveNext	2	[{"parsedStack":[{"method":"Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<OnReceiveAsync>d__13.MoveNext","level":0,"line":0,"assembly":"Microsoft.Azure.EventHubs, Version=4.2.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c"},{"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":1,"line":0,"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"},{"method":"Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<OnReceiveAsync>d__13.MoveNext","level":2,"line":0,"assembly":"Microsoft.Azure.EventHubs, Version=4.2.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c"},{"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":3,"line":0,"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"},{"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":4,"line":0,"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"},{"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":5,"line":0,"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"},{"method":"Microsoft.Azure.EventHubs.PartitionReceiver+<ReceiveAsync>d__30.MoveNext","level":6,"line":0,"assembly":"Microsoft.Azure.EventHubs, Version=4.2.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c"},{"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":7,"line":0,"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"},{"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":8,"line":0,"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"},{"method":"Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<ReceivePumpAsync>d__18.MoveNext","level":9,"line":0,"assembly":"Microsoft.Azure.EventHubs, Version=4.2.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c"}],"severityLevel":"Warning","outerId":"0","message":"New receiver 'host-920c6f86-b9d7-49d5-884c-a74671edb94d' with higher epoch of '2' is created hence current receiver 'host-6f385b94-bb66-4e1a-9450-43535abb172a' with epoch '1' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:f9764352000339ec001b72ec5ed7f247_G11_B32, SystemTracker:eventhubnamespace-mw7uby:eventhub:eventhub-mw7uby~8191|telemetryux, Timestamp:2020-06-03T18:57:02","type":"Microsoft.Azure.EventHubs.ReceiverDisconnectedException","id":"54357625"}]	exception	{"MessageTemplate":"Processor {id} error","SourceContext":"Microsoft.Azure.IIoT.Hub.Processor.Services.EventProcessorFactory","ProcessorId":"40cfe513-9d79-4584-817c-321ccb9608b7","id":"40cfe513-9d79-4584-817c-321ccb9608b7"}	PC	0.0.0.0	Washington	Virginia	United States	6e05638783a5.(none)	26625d77-5126-4bbb-a1ef-5d0af17375a9

[gillermaney]

Connecting log analytics to the gateway and simulator VMs works, but what is the path to the publisher log data using the log anlytics tool? I don't see the publisher module in the list of kusto tables. Do I need to use SSH to log into the IoT Gateway? If so, how do I find the public IP address of the IoT Gateway? Would it be possible to add a short write up about using SSH to log into the VMs if this is how you are supposed to trouble shoot them? I can connect using Bastion with a user name and password, but don't know where the log files are on disk.

[gillermaney]

We figured out the issue with messages not reaching IoTHub. It has to do with the iothubmessagesize and iothubsendintervale parameters. WRT message size, the default is 262144 bytes. Since we are only subscribing to relatively small number of messages, we don't hit this default value quickly. It would be good to make this a settable parameter in the engineering tool and describe the issue in the tutorial. WRT send interval, we had to set this to zero. It would be good to make this a settable parameter in the engineering tool and describe the issue in the tutorial.

Lastly, connecting to the VM's would seem to be easiest using Bastion so a short write up about this would be good. Also it would be good to document the location of the publisher log files.

[barnstee]

Please check out our new Ops Manual for Publisher documentation, too

[barnstee]

Remaining feature request is for "It would be good to make this a settable parameter in the engineering tool and describe the issue in the tutorial."