Persistent subscriptions and clustered eventstore.

[henriiik]

Hello!

I am trying to use spear create/connect to persistent subscriptions in a clustered eventstoredb setup.

If the Spear.Connection is connected to a non-leader node an error like this is returned, which contains the info needed to re-connect to the leader node:

%Spear.Connection.Response{
  data: "",
  headers: [
    {"content-type", "application/grpc"},
    {"date", "Thu, 02 Mar 2023 08:08:04 GMT"},
    {"server", "Kestrel"},
    {"content-length", "0"},
    {"exception", "not-leader"},
    {"leader-endpoint-host", "127.0.0.1"},
    {"leader-endpoint-port", "2111"},
    {"grpc-status", "5"},
    {"grpc-message", "Leader info available"}
  ],
  status: 200,
  type: {:event_store_db_gpb_protobufs_persistent,
   :"event_store.client.persistent_subscriptions.CreateResp"}
}

The call to Spear.create_persistent_subscription returns an error like this, which was very confusing to me at first:

%Spear.Grpc.Response{
  data: "",
  message: "Leader info available",
  status: :not_found,
  status_code: 5
}

I was able to reproduce the issue by switching the docker setup in the spear repo to use a clustered eventstoredb, and with some additional logging i get this:

❯ mix test test/spear_test.exs:310
Compiling 2 files (.ex)
Excluding tags: [:test]
Including tags: [line: "310"]

%Spear.Connection.Response{
  data: <<0, 0, 0, 0, 18, 10, 16, 8, 6, 26, 12, 8, 237, 169, 129, 128, 1, 16,
    237, 169, 129, 128, 1>>,
  headers: [
    {"content-type", "application/grpc"},
    {"date", "Thu, 02 Mar 2023 08:13:01 GMT"},
    {"server", "Kestrel"},
    {"grpc-status", "0"}
  ],
  status: 200,
  type: {:event_store_db_gpb_protobufs_streams,
   :"event_store.client.streams.AppendResp"}
}
%Spear.Connection.Response{
  data: "",
  headers: [
    {"content-type", "application/grpc"},
    {"date", "Thu, 02 Mar 2023 08:13:01 GMT"},
    {"server", "Kestrel"},
    {"content-length", "0"},
    {"exception", "not-leader"},
    {"leader-endpoint-host", "127.0.0.1"},
    {"leader-endpoint-port", "2111"},
    {"grpc-status", "5"},
    {"grpc-message", "Leader info available"}
  ],
  status: 200,
  type: {:event_store_db_gpb_protobufs_persistent,
   :"event_store.client.persistent_subscriptions.CreateResp"}
}
%Spear.Grpc.Response{
  data: "",
  message: "Leader info available",
  status: :not_found,
  status_code: 5
}


  1) test given a stream contains events info about a psub to :all can be fetched (SpearTest)
     test/spear_test.exs:310
     Assertion with == failed
     code:  assert Spear.create_persistent_subscription(c.conn, :all, group, settings) == :ok
     left:  {:error, %Spear.Grpc.Response{data: "", message: "Leader info available", status: :not_found, status_code: 5}}
     right: :ok
     stacktrace:
       test/spear_test.exs:314: (test)



Finished in 1.3 seconds (1.3s async, 0.00s sync)
64 tests, 1 failure, 63 excluded

Randomized with seed 439068

edit:

I accidentally submitted this issue before i was done writing it.

Is there recomended way to handle this? Look up leader before creating the subscription? or is this a bug?