Upgrade Socket.io engine

[hassy]

Current Socket.io engine uses Socket.io v2. We need to add support for v3/v4.

@ptejada built https://github.com/ptejada/artillery-engine-socketio-v3, which also improves the API of the engine in addition to adding v3 support. It's awesome and we should just adopt that.

• [ ] The API change will need to be documented. We don't have support for versioning in the docs yet, so that will need to be added - or API differences in v1.6.x vs v2 documented clearly in current documentation.
• [ ] The engine will need to be updated to use the new metrics API in v2

Ref: https://github.com/ptejada/artillery-engine-socketio-v3/issues/1

[0GiS0]

Hi all,

Do you know if there is an engine for socket.io v4?

I was using the engine from @ptejada but it's for v3 and I getting this errors:

"engine unsupported protocol version"

So I assume that it's because of this.

thanks

[hassy]

hi @0GiS0 👋 v3 and v4 should be compatible, according to https://socket.io/docs/v4/troubleshooting-connection-issues/#The-client-is-not-compatible-with-the-version-of-the-server. Do you see "engine unsupported protocol version" errors in Artillery's output?

[0GiS0]

hi @0GiS0 👋 v3 and v4 should be compatible, according to https://socket.io/docs/v4/troubleshooting-connection-issues/#The-client-is-not-compatible-with-the-version-of-the-server. Do you see "engine unsupported protocol version" errors in Artillery's output?

Hi @hassy!

No, I saw that error in the server side when I execute the tests. (I enabled DEBUG=* because I'm seeing 400 errors all the time)

I enabled affinity with the server on Azure App Service, and this is the config from socket.io server:

const appInsights = require("applicationinsights");
appInsights.setup('APP_INSIGTHS_INSTRUMENTATION_KEY').start();

const express = require('express');
const app = express();
const http = require('http');
const server = http.createServer(app);
const { Server } = require('socket.io');
const { instrument } = require("@socket.io/admin-ui");
const redisAdapter = require("socket.io-redis");
const path = require('path');
require('dotenv').config();
var redis = require("redis");
var socketpub = redis.createClient(process.env.REDIS_PORT, process.env.REDIS_HOSTNAME, { auth_pass: process.env.REDIS_KEY, return_buffers: true });
var socketsub = redis.createClient(process.env.REDIS_PORT, process.env.REDIS_HOSTNAME, { auth_pass: process.env.REDIS_KEY, return_buffers: true });

const port = process.env.PORT || 3000;

server.listen(port, () => {
    console.log('Server listening at port %d', port);
});

// Routing
app.use(express.static(path.join(__dirname, 'public')));

// Chatroom
let numUsers = 0;

const io = new Server(server, {
    // transports: ["websocket"],
    perMessageDeflate: false, //https://docs.microsoft.com/es-es/azure/app-service/faq-app-service-linux#language-support
    cors: {
        origin: ["https://admin.socket.io"], credentials: true
    }
});


io.adapter(redisAdapter({ pubClient: socketpub, subClient: socketsub }));
instrument(io, { auth: false }); //Admin web: https://admin.socket.io/

and here the Artillery YAML:

config:
  target: "https://chat-socketio.azurewebsites.net"
  ensure:
    maxErrorRate: 1 # fail if error rate exceeds 1%
    max: 500 # fail if max response time exceeds 500ms
  engines:
    socketio-v3: { }    
  phases:
    - duration: 120
      arrivalRate: 10
      rampTo: 50
      name: "Warm up phase"
    - duration: 600
      arrivalRate: 50
      name: "Sustained max load"
  variables:
    greeting:
      [
        "hola",
        "¡Buenos días!",
        "Hola gente",
        "buenas",
        "ya estamos todos",
        "hola",
      ]
  processor: "./functions.js"
scenarios:
  - name: "Usuarios que solo entran"
    weight: 10
    # engine: "socketio-v3"
    engine: "socketio"
    flow:
      # - get:
      #     url: "/"
      #     # afterResponse: "printStatus"
      - emit:
          channel: "add user"
          data: "lurker-{{$randomString()}}"
      - think: 2
  - name: "Usuarios que solo saludan"
    weight: 20
    # engine: "socketio-v3"
    engine: "socketio"
    flow:
      # - get:
      # url: "/"
      # afterResponse: "printStatus"
      - emit:
          channel: "add user"
          data: "quiet-{{ $randomString() }}"
      - think: 5
      - emit:
          channel: "new message"
          data: "{{ greeting }}"
      - think: 5
  - name: "Usuarios que hablan todo el tiempo"
    weight: 70
    # engine: "socketio-v3"
    engine: "socketio"
    flow:
      # - get:
      # url: "/"
      # afterResponse: "printStatus"
      - emit:
          channel: "add user"
          data: "chatty-{{ $randomString() }}"
      - emit:
          channel: "new message"
          data: "{{ greeting }}"
      - loop:
          - function: "setMessage"
          - emit:
              channel: "new message"
              data: "{{ message }}"
          - think: 10
        count: 10
      - think: 1

This is the artillery output:

Any ideas? Thank you so much!

[0GiS0]

Hi again!

I notice this only happens when I have two nodes but if i have only one with RedisAdapter and all It works perfect. So i was wondering if this was tested with environment which requires sticky session and if during the negotiation the engine sends the cookie needed.

Thanks again!

[josekudiyirippil]

Hi team,

any update on this ticket? we are having the same issue. Our artillery tests are failing with 404s

[peterzanetti]

Does artillery no longer work for testing socket.io v2?

I previously used artillery without issue, but when trying today I see: Error: It seems you are trying to reach a Socket.IO server in v2.x with a v3.x client

Is there a version of artillery that still works with v2?