Go Client: Connect must be enabled in order to use this endpoint

[RickoNoNo3]

Overview of the Issue

Error when use Golang client. It seems like the CA certification is opened by default? But I don't have any configuration for it.

Reproduction Steps

Create a cluster with at least 3 server nodes as Docker container, and use these configs:

version: '2'

services:
  consul-server1:
    image: hashicorp/consul:1.11.2
    container_name: consul-server1
    restart: always
    volumes:
     - /root/consul/server1.json:/consul/config/server1.json:ro
    networks:
      - consul
    ports:
      - "8500:8500"
      - "8600:8600/tcp"
      - "8600:8600/udp"
    command: "agent -bootstrap-expect=3"

  consul-server2:
    image: hashicorp/consul:1.11.2
    container_name: consul-server2
    restart: always
    volumes:
     - /root/consul/server2.json:/consul/config/server2.json:ro
    networks:
      - consul
    ports:
      - "8501:8500"
      - "8601:8600/tcp"
      - "8601:8600/udp"
    command: "agent -bootstrap-expect=3"

  consul-server3:
    image: hashicorp/consul:1.11.2
    container_name: consul-server3
    restart: always
    volumes:
     - /root/consul/server3.json:/consul/config/server3.json:ro
    networks:
      - consul
    ports:
      - "8502:8500"
      - "8602:8600/tcp"
      - "8602:8600/udp"
    command: "agent -bootstrap-expect=3"

  consul-client:
    image: hashicorp/consul:1.11.2
    container_name: consul-client
    restart: always
    volumes:
     - /root/consul/client.json:/consul/config/client.json:ro
    networks:
      - consul
    command: "agent"

networks:
  consul:
    driver: bridge

Takes server1.json as an example:

{
    "node_name": "consul-server1",
    "server": true,
    "ui_config": {
        "enabled" : true
    },
    "data_dir": "/consul/data",
    "addresses": {
        "http" : "0.0.0.0"
    },
    "retry_join":[
        "consul-server2",
        "consul-server3"
    ]
}

Write a Go program on the same machine as server1, build and run:

import (
	"fmt"
	"net/http"

	"github.com/hashicorp/consul/api"
	"github.com/hashicorp/consul/connect"
)

func main() {
	client, _ := api.NewClient(api.DefaultConfig())
	svc, _ := connect.NewService("MyService", client)
	defer svc.Close()

	if err := http.ListenAndServe(":8890", nil); err != nil {
		fmt.Println("HTTP serving error: ",err)
	}
}

View error in Go console:

2022-08-30T15:01:44.550+0800 [ERROR] connect.watch: Watch errored: service=MyService type=connect_roots error="Unexpected response code: 500 (rpc error making call: Connect must be enabled in order to use this endpoint)" retry=3m0s
2022-08-30T15:01:44.551+0800 [ERROR] connect.watch: Watch errored: service=MyService type=connect_leaf error="Unexpected response code: 500 (rpc error making call: Connect must be enabled in order to use this endpoint)" retry=3m0s
2022-08-30T15:04:44.650+0800 [ERROR] connect.watch: Watch errored: service=MyService type=connect_roots error="Unexpected response code: 500 (rpc error making call: Connect must be enabled in order to use this endpoint)" retry=3m0s
2022-08-30T15:04:44.650+0800 [ERROR] connect.watch: Watch errored: service=MyService type=connect_leaf error="Unexpected response code: 500 (rpc error making call: Connect must be enabled in order to use this endpoint)" retry=3m0s

View error in Server1:

2022-08-30T06:40:12.239Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/connect/ca/leaf/MyService from=172.24.0.1:35240 error="rpc error making call: Connect must be enabled in order to use this endpoint"
2022-08-30T06:40:12.239Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/connect/ca/roots from=172.24.0.1:35242 error="rpc error making call: Connect must be enabled in order to use this endpoint"
2022-08-30T06:41:32.295Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/connect/ca/roots from=172.24.0.1:35242 error="rpc error making call: Connect must be enabled in order to use this endpoint"
2022-08-30T06:41:32.295Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/connect/ca/leaf/MyService from=172.24.0.1:35240 error="rpc error making call: Connect must be enabled in order to use this endpoint"
2022-08-30T06:43:37.326Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/connect/ca/leaf/MyService from=172.24.0.1:37488 error="rpc error making call: Connect must be enabled in order to use this endpoint"
2022-08-30T06:43:37.326Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/connect/ca/roots from=172.24.0.1:37490 error="rpc error making call: Connect must be enabled in order to use this endpoint"
2022-08-30T06:46:37.410Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/connect/ca/leaf/MyService from=172.24.0.1:38972 error="rpc error making call: Connect must be enabled in order to use this endpoint"
2022-08-30T06:46:37.410Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/connect/ca/roots from=172.24.0.1:38974 error="rpc error making call: Connect must be enabled in order to use this endpoint"

Consul info

Server info

agent:
        check_monitors = 0
        check_ttls = 1
        checks = 3
        services = 2
build:
        prerelease =
        revision = 37c7d06b
        version = 1.11.2
consul:
        acl = disabled
        bootstrap = false
        known_datacenters = 1
        leader = false
        leader_addr = 172.24.0.2:8300
        server = true
raft:
        applied_index = 21093
        commit_index = 21093
        fsm_pending = 0
        last_contact = 4.067366ms
        last_log_index = 21093
        last_log_term = 2
        last_snapshot_index = 16387
        last_snapshot_term = 2
        latest_configuration = [{Suffrage:Voter ID:13ca4bb5-02c1-6352-194f-acf2c8d69554 Address:172.24.0.2:8300} {Suffrage:Voter ID:b1232bfd-2eb5-a462-ad29-5477a522e739 Address:172.24.0.5:8300} {Suffrage:Voter ID:f5e1271d-0c5f-a264-59c1-0f4680a4a83e Address:172.24.0.7:8300}]
        latest_configuration_index = 0
        num_peers = 2
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Follower
        term = 2
runtime:
        arch = amd64
        cpu_count = 4
        goroutines = 165
        max_procs = 4
        os = linux
        version = go1.17.5
serf_lan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 2
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 4
        members = 4
        query_queue = 0
        query_time = 1
serf_wan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 4
        members = 3
        query_queue = 0
        query_time = 1

Operating system and Environment details

Ubuntu 18.04 amd64

[jkirschner-hashicorp]

Hi @RickoNoNo3,

Welcome to the Consul community! Thank you for reaching out about this.

Can you clarify what you wanted to accomplish with the command below? IsMyService intended to be a service in Consul's service mesh?

svc, _ := connect.NewService("MyService", client)

[RickoNoNo3]

Yes, I want to register my own program as a service of Consul.

[jkirschner-hashicorp]

Are you looking to deploy "MyService" with a sidecar proxy in a service mesh? Or just register "MyService" with Consul to make it discoverable to other programs ("service discovery")?

Are there any guides or documentation you were following?

I ask because there may be an opportunity to make Consul's documentation and/or log messages clearer.

"connect" is the subsystem in Consul that provides its service mesh capabilities. Therefore, connect.newService can't be used without enabling the connect subsystem in the Consul server agent configuration JSON. For example:

{
    "node_name": "consul-server1",
    "server": true,
    "ui_config": {
        "enabled" : true
    },
    "data_dir": "/consul/data",
    "addresses": {
        "http" : "0.0.0.0"
    },
    "retry_join":[
        "consul-server2",
        "consul-server3"
    ],
    "connect":{
      "enabled": true
    }
}

[RickoNoNo3]

I am sorry that I didn’t fully understand the concept of Connect and Mesh. I just looked for a method to register a consul service (that is called, now I know, service discovery) in Go, Then I accidentally entered https://www.consul.io/docs/connect/native/go by mistake so I did above things.

Now it seems that client.Agent().ServiceRegister() is more suitable for me, rather than connect.NewService(). Is it right?

[jkirschner-hashicorp]

It's the role of our documentation to ensure concepts like Connect and Mesh are clear for our users :) Your experience points out there may be things we can do to make it clearer.

That Go integration page you found does show connect.NewService() without anything indicating it requires service mesh (connect enabled), or indicating that an alternative should be used for service discovery. Maybe we can make a small docs change there.

Yes, client.Agent().ServiceRegister() sounds right to me.

[RickoNoNo3]

Thank you!