Out of order pings after non-200 and connection errors

[st3fan]

The test endpoint at https://basement.sateh.com/telemetry-incoming will return a 500 for every sequence number that is a multiple of 5 and will abort/close the connection for every sequence number that is a multiple of 13.

I am seeing the following happing:

(Event pings removed - the simulated errors only happen for core pings)

2018/04/18 19:03:47 500 E1E27683-D658-438B-8411-C713B9451D7F core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           75 (23h3m47.610418714s)


2018/04/18 19:03:55 200 E1E27683-D658-438B-8411-C713B9451D7F core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           75 (23h3m55.200843168s)
2018/04/18 19:03:55 500 85836623-A476-40C7-8951-9173EEE5C568 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           70 (23h3m55.209219732s)


2018/04/18 19:04:01 200 85836623-A476-40C7-8951-9173EEE5C568 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           70 (23h4m1.035697222s)
2018/04/18 19:04:01 200 A0156696-61BC-41CA-B46B-4B7A962A1B56 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           87 (23h4m1.054416382s)
2018/04/18 19:04:01 200 E41AD980-3596-4B1D-91B9-A2910EE35C89 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           74 (23h4m1.063136299s)
2018/04/18 19:04:01 200 72674670-7201-416A-B3A9-4FA291934433 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           88 (23h4m1.070717299s)
2018/04/18 19:04:01 200 B98F1816-C547-4A71-9B81-1DDAA27D66B8 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           79 (23h4m1.0792575s)
2018/04/18 19:04:01 ERR 9A52432F-5C33-40A0-8BE6-3D4BBFA12027 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           91 (23h4m1.087181501s)

2018/04/18 19:04:05 200 1E713146-E8BF-4AFA-876B-E6AA6242E7E3 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           92 (23h4m5.680228191s)
2018/04/18 19:04:05 200 9A52432F-5C33-40A0-8BE6-3D4BBFA12027 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           91 (23h4m5.687184832s)
2018/04/18 19:04:05 500 DE838166-24FD-4AAE-8F1A-A57D01CF572E core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           80 (23h4m5.694552486s)

2018/04/18 19:04:09 200 DE838166-24FD-4AAE-8F1A-A57D01CF572E core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           80 (23h4m9.459854604s)

2018/04/18 19:04:09 500 2304F04A-17B8-4581-9377-709D20F26C8D core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           90 (23h4m9.472049816s)

2018/04/18 19:04:12 200 2304F04A-17B8-4581-9377-709D20F26C8D core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           90 (23h4m12.323785544s)
2018/04/18 19:04:12 200 847C0074-BFCB-4B14-A2D6-EC736F844A9D core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           94 (23h4m12.352921581s)
2018/04/18 19:04:12 200 7D13F97F-E8FA-49AC-8FA8-E9DB19925C37 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           73 (23h4m12.360999897s)
2018/04/18 19:04:12 200 AEDA24B8-C6FB-4292-913A-572B7BCABC32 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           89 (23h4m12.367838278s)
2018/04/18 19:04:12 200 D9B304B3-EFF4-436B-93A6-289FE038CE02 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           84 (23h4m12.374474652s)
2018/04/18 19:04:12 200 6856123D-E9B6-4AE9-9795-60BFCB1EAC69 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           93 (23h4m12.381737926s)
2018/04/18 19:04:12 200 0E474E03-B551-4441-A0F0-953E39B42994 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           81 (23h4m12.391548084s)
2018/04/18 19:04:12 200 7DBC633D-ADFB-4DEF-B69F-513633C91201 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           71 (23h4m12.403456643s)
2018/04/18 19:04:12 200 C8E4C7DC-89A7-4085-AA0F-B243CC7F5406 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           77 (23h4m12.412524504s)
2018/04/18 19:04:12 500 F9F70C3B-2292-40F7-8EF7-462EFDA1527B core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           85 (23h4m12.422551256s)

2018/04/18 19:04:12 500 D15F9376-3EBE-4FE0-9DDC-234CBDA0E436 core           D2F01E77-9AE2-4470-BE33-7FE34E3A5C3F 2018-04-18           95 (23h4m12.662410591s)

Both HTTP and Connection Errors are retried correctly but are out of order.