Eli Ribble

This is to address https://github.com/vaidik/commentjson/issues/43

Alright, looks like I captured some more information. At UTC 19:22 our automated systems started alerting that one of our services wasn't reachable. We had 3 nodes in the cluster...

Here's more logs in case it helps: Just had a repro again, this time couldn't reach services in `qbio2455qa8aysx15p31aqdc8 `. Log from the afflicted node: ``` Jun 20 22:02:04 ip-10-0-109-151...

My guess at this point is that the issue has to do with the line: `Jun 21 15:23:42 ip-10-0-109-151 dockerd[18631]: time="2017-06-21T15:23:42.160320276Z" level=error msg="Bulk sync to node ip-10-0-101-134-756771be03b7 timed out"` as...

Alright, I started looking at the logs on the other nodes to see if there was anything obvious in them. This seems suspcious - this is from the system that...

``` root@ip-10-0-46-77:/home/eliribble# docker service ps rl90o2zc17b14r294p0a4dr7v --no-trunc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 1z516vsiax1j81gmr0vb3y6av maxillo.1 authentise/maxillo:latest@sha256:a0b50d597dbc1a27b3c418b4066e9099ca4c22eb550c1985ad7999bed87db9ac ip-10-0-101-134 Running Starting 7 seconds ago uvtomhf2kq43b8zwhn1j6p3jt \_ maxillo.1 authentise/maxillo:latest@sha256:a0b50d597dbc1a27b3c418b4066e9099ca4c22eb550c1985ad7999bed87db9ac...

I hadn't noticed before that maxillo was failing to start. That's been resolved now. I believe it was a red-herring, we've been having the problem for days and the maxillo...

Based on this discussion thread https://groups.google.com/forum/#!topic/consul-tool/dQSHf2R93lI I'm starting to wonder if the issue has to do with periodic UDP packet failure so that one node is failing to indicate it...

I'm seeing the same thing using Keycloak. NextCloud properly navigates to the Keycloak login, I authenticate, and NextCloud starts part of the login flow. The failure is after POST to...

I hit the same error and ended up just subclassing the Protocol to dodge the problem. The major issue for us is that we have sentry in place which emits...