WebSocket connection failure and defunct chrome process

[ggrossetie]

The mermaid companion container on https://kroki.io is unstable. For unknown reasons, the chrome process is defunct (zombie) and the connection via WebSocket between the Node process and the Chrome instance is failing:

Exception during convert ErrorEvent {
  target: WebSocket {
    _events: [Object: null prototype] { open: [Function], error: [Function] },
    _eventsCount: 2,
    _maxListeners: undefined,
    _binaryType: 'nodebuffer',
    _closeCode: 1006,
    _closeFrameReceived: false,
    _closeFrameSent: false,
    _closeMessage: '',
    _closeTimer: null,
    _extensions: {},
    _protocol: '',
    _readyState: 3,
    _receiver: null,
    _sender: null,
    _socket: null,
    _bufferedAmount: 0,
    _isServer: false,
    _redirects: 0,
    _url: 'ws://127.0.0.1:39883/devtools/browser/42c3f509-6ca8-41c9-bc52-e455bc2e80cd',
    _req: null,
    [Symbol(kCapture)]: false
  },
  type: 'error',
  message: 'connect ECONNREFUSED 127.0.0.1:39883',
  error: Error: connect ECONNREFUSED 127.0.0.1:39883
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '127.0.0.1',
    port: 39883
  }
}

We might need to enable some traces on Chrome to understand what's going on.

[heyLu]

We had the same issue in our internal instance running in Kubernetes. Increasing the Kubernetes memory request and limit from 128M to 512M has so far fixed the problem for us. (The instance has been running okay for 24h now.)

[mariotoffia]

@heyLu is this what you did to make it work (I use docker compose instead of Kubernetes)?

NOTE I run a Macbook M1 (arm64) and version 0.18.0 of the containers. Version 0.17.2 works like a charm! :)

Docker compose file:

image: yuzutech/kroki-excalidraw
mem_limit: 512m
mem_reservation: 128M

when running the command

cat test.excalidraw | curl -H 'Content-Type: text/plain' -H 'Accept: image/svg+xml' -X POST --data-binary @- http://localhost:8000/excalidraw/svg > _assets/test.svg

I get the following (for excalidraw) in my logs:

{"timestamp":"1669961084681","level":"INFO","thread":"vert.x-eventloop-thread-1","mdc":{"path":"/excalidraw/svg","method":"POST","service_name":"excalidraw","action":"request_received","bytes_read":"286716","user_agent":"curl/7.84.0"},"logger":"io.kroki.server.service.DiagramHandler","message":"Request received POST /excalidraw/svg","context":"default"}
kroki-excalidraw-1   | e ErrorEvent {
kroki-excalidraw-1   |   [Symbol(kTarget)]: WebSocket {
kroki-excalidraw-1   |     _events: [Object: null prototype] { open: [Function], error: [Function] },
kroki-excalidraw-1   |     _eventsCount: 2,
kroki-excalidraw-1   |     _maxListeners: undefined,
kroki-excalidraw-1   |     _binaryType: 'nodebuffer',
kroki-excalidraw-1   |     _closeCode: 1006,
kroki-excalidraw-1   |     _closeFrameReceived: false,
kroki-excalidraw-1   |     _closeFrameSent: false,
kroki-excalidraw-1   |     _closeMessage: <Buffer >,
kroki-excalidraw-1   |     _closeTimer: null,
kroki-excalidraw-1   |     _extensions: {},
kroki-excalidraw-1   |     _paused: false,
kroki-excalidraw-1   |     _protocol: '',
kroki-excalidraw-1   |     _readyState: 3,
kroki-excalidraw-1   |     _receiver: null,
kroki-excalidraw-1   |     _sender: null,
kroki-excalidraw-1   |     _socket: null,
kroki-excalidraw-1   |     _bufferedAmount: 0,
kroki-excalidraw-1   |     _isServer: false,
kroki-excalidraw-1   |     _redirects: 0,
kroki-excalidraw-1   |     _url: 'ws://127.0.0.1:38081/devtools/browser/d445ad2e-b535-4092-8c56-9048d3e131b6',
kroki-excalidraw-1   |     _originalSecure: false,
kroki-excalidraw-1   |     _originalHost: '127.0.0.1:38081',
kroki-excalidraw-1   |     _req: null,
kroki-excalidraw-1   |     [Symbol(kCapture)]: false
kroki-excalidraw-1   |   },
kroki-excalidraw-1   |   [Symbol(kType)]: 'error',
kroki-excalidraw-1   |   [Symbol(kError)]: Error: connect ECONNREFUSED 127.0.0.1:38081
kroki-excalidraw-1   |       at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16) {
kroki-excalidraw-1   |     errno: -111,
kroki-excalidraw-1   |     code: 'ECONNREFUSED',
kroki-excalidraw-1   |     syscall: 'connect',
kroki-excalidraw-1   |     address: '127.0.0.1',
kroki-excalidraw-1   |     port: 38081
kroki-excalidraw-1   |   },
kroki-excalidraw-1   |   [Symbol(kMessage)]: 'connect ECONNREFUSED 127.0.0.1:38081'
kroki-excalidraw-1   | }

Excalidraw file

{
  "type": "excalidraw",
  "version": 2,
  "source": "https://excalidraw.com",
  "elements": [
    {
      "id": "l9cH3xUXnD7x8ot0O4yPn",
      "type": "rectangle",
      "x": 2246.1794425288995,
      "y": -60.53904093184536,
      "width": 353.80859375,
      "height": 227.14843750000006,
      "angle": 0,
      "strokeColor": "#000000",
      "backgroundColor": "#a61e4d",
      "fillStyle": "hachure",
      "strokeWidth": 1,
      "strokeStyle": "solid",
      "roughness": 2,
      "opacity": 100,
      "groupIds": [],
      "strokeSharpness": "sharp",
      "seed": 638167397,
      "version": 23,
      "versionNonce": 1421406853,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1669961294045,
      "link": null,
      "locked": false
    }
  ],
  "appState": {
    "gridSize": null,
    "viewBackgroundColor": "#ffffff"
  },
  "files": {}
}

I get the following output file:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg xmlns="http://www.w3.org/2000/svg" width="146" height="26" viewBox="0 0 146 26" version="1.1">
<rect fill="#fff5f7" width="146" height="26"/>
<text id="text" xml:space="preserve" font-weight="bold" fill="#cd0930" font-family="BlinkMacSystemFont, -apple-system, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Fira Sans', 'Droid Sans', 'Helvetica Neue', 'Helvetica', Arial, sans-serif" font-size="16px" x="0" y="5" dy="0">
<tspan x="10" dy="16">Error 400: OK</tspan>
</text>
<rect fill="#ff3860" width="3" height="26"/>
</svg>

[heyLu]

@mariotoffia not sure, our problem was only with the mermaid container and we increased both the request and limit to 512M. Here's a snippet of what this looks like in Kubernetes' YAML format:

    - image: yuzutech/kroki-mermaid:0.18.0
      imagePullPolicy: IfNotPresent
      name: mermaid
      resources:
        limits:
          memory: 512M
        requests:
          cpu: 10m
          memory: 512M

With this our Kroki instance (and the kroki-mermaid container in it) has been running without the crash described in this issue since I posted that message, both on 0.17.2 and 0.18.0.

So maybe try increasing mem_reservation to 512M as well?