kvstore.consul.acl_token has no effect on Distributor

[jakubgs]

Describe the bug I've provided the Consul host and token using the flags so far, because it was a good way to avoid cluttering up my config with needlessly repeated sections of kvstore config. Today while trying to use ETCD alongside Consul using Multi KV I encountered this error when trying to use kvstore.consul.acl_token instead in distributor.ring config:

level=error
caller=cortex.go:448
msg="module failed"
module=ring
err="invalid service state:
  Failed, expected:
    Running, failure:
      unable to initialise ring state:
        Unexpected response code:
          403 (rpc error making call:
            Permission denied:
              token with AccessorID '00000000-0000-0000-0000-000000000002'
              lacks permission 'key:read' on \"collectors/ring\"
          )"

After re-adding the -consul.acl-token flag the error went away, which suggests to me that the acl_token config doesn't work.

To Reproduce Steps to reproduce the behavior:

1. Start Cortex 1.13.0 distributor
2. Use kvstore.consul.acl_token in configuration
3. See error above

Expected behavior It works.

Environment:

• Infrastructure: Ubuntu
• Deployment tool: Ansible

[alvinlin123]

@jakubgs so if you use the -<prefix>.consul.acl-token command line argument to Cortex everything works, but if you configure the same in YAML everything works?

If possible would you be able to provide examples of config that works and doesn't work?

[jakubgs]

This is the config I was using:

target: 'distributor'
auth_enabled: false

configs:
  database:
    uri: 'memory://'

limits:
  ingestion_rate: 1000000
  ingestion_burst_size: 2000000
  max_chunks_per_query: 7000000
  cardinality_limit: 30000
  max_series_per_metric: 30000
  max_series_per_user: 5000000
  max_label_names_per_series: 30

server:
  http_listen_address: '0.0.0.0'
  http_listen_port: 9092
  grpc_listen_address: '0.0.0.0'
  grpc_listen_port: 9095
  log_level: 'debug'

storage:
  engine: 'blocks'

distributor:
  remote_timeout: '4s'

  ring:
    kvstore:
      store: multi
      etcd:
        username: cortex
        password: cortex
        endpoints: ['10.10.0.10:2379', '10.10.0.11:2379', '10.10.0.12:2379']

      consul:
        host: localhost:8500
        # WARNING: Has no effect currently.
        acl_token: 00000000-0000-0000-0000-000000000000
      multi:
       primary: etcd
       secondary: consul
       mirror_enabled: true
       mirror_timeout: 2s

And I didn't use -<prefix>.consul.acl-token, I used -consul.acl-token, because brevity is the soul of wit. I wish there was a global section in the YAML config file so I could define my ETCD and Consul configuration there for all targets without having to repeat the same section at times.