Usage locally against GCP PubSub emulator

[plasticine]

I would like to be able to develop locally and have Kane use the GCP PubSub emulator (gcloud beta emulators pubsub start) endpoint.

This seems like something I could set up via the following config;

config :kane,
  endpoint: "http://127.0.0.1:8085"

...which seems to work OK, however I’m stuck with errors coming out of Goth.

I’m wondering if this restriction is intentional or just an oversight?

[plasticine]

After digging a bit more it seems like the following is possible;

config :kane,
  token: Goth.Token,
  endpoint: "http://127.0.0.1:8085"

config :goth,
  json: nil,
  project_id: "foo"

[jeffdeville]

To get this to work, I had to include /v1 in the kane endpoint. Here's my config:

config :goth, json: {:system, "GCP_CREDENTIALS"}
config :kane, :endpoint, "http://#{System.get_env("PUBSUB_EMULATOR_HOST")}/v1"

(For anyone curious, I collapsed my credentials.json file to 1 line, and exported it in GCP_CREDENTIALS)

[medikent]

@jeffdeville Did you have to use valid GCP credentials in order to get this to work? If so, can you provide an explanation or example? Trying all the methods you and the author suggested brings me to Could not retrieve token invalid_grant errors from Goth.

[matehat]

@medikent you don't need working credentials to talk to pubsub emulator. Here's what I did to make all of that work:

defmodule PubSub.Emulator do
  @moduledoc """
  Provide fake tokens when running with PubSub emulator
  """

  def for_scope(_) do
    {:ok, %{type: "Bearer", token: "dummy"}}
  end
  
  def detect() do
    emulator_host = System.get_env("PUBSUB_EMULATOR_HOST")

    unless is_nil(emulator_host) do
      Application.put_env(:kane, :token, PubSub.Emulator)
      Application.put_env(:kane, :endpoint, "http://#{emulator_host}/v1")

      Logger.info("Using pubsub emulator")
    end
  end
end

So I just add the line PubSub.Emulator.detect() in by Application.start/2 callback function and 1) it Just Works™️ , and I get a nice little log Using pubsub emulator to confirm this is activated.

[medikent]

@matehat Thanks for the update. I ended up going with Weddell to take advantage of gRPC. The gRPC API provides lower tail latency (see this ). See Weddell. Once Batching (https://github.com/cjab/weddell/issues/14) and StreamingPull (https://github.com/cjab/weddell/issues/15) are added to Weddell it will support the low-latency, point-to-point messaging needs I have.

[matehat]

Yeah I looked at Weddell quite a bit, but that statement worries me:

If the beta/experimental status of Weddell worries you Kane may be a better choice.

You've had good experience with it? Is it running in production?

[medikent]

Yes, we've had good experience with it. My team runs it in production. I assume cjab, the maintainer, is being conservative in his description of the library. Once we have batching and StreamingPull working then it will be closer to being a full-featured 1.0 release.

[matehat]

Awesome, thanks for the status! Will start to switch our code very soon 😃