m3 icon indicating copy to clipboard operation
m3 copied to clipboard

M3DB unable to mark namespace as ready

Open gemiit opened this issue 4 years ago • 1 comments

error messages

{"level":"info","ts":1626163424.4172487,"msg":"dynamic namespace registry updated to version","version":1}
{"level":"info","ts":1626163424.4172852,"msg":"received update from kv namespace watch"}
{"level":"info","ts":1626163424.417325,"msg":"updating database namespaces","adds":"[]","updates":"[default]","removals":"[]"}
{"level":"warn","ts":1626163424.4173307,"msg":"skipping namespace removals and updates (except schema updates and runtime options), restart the process if you want changes to take effect"}
{"level":"info","ts":1626163424.4178169,"msg":"dynamic namespace registry updated to version","version":1}
{"level":"info","ts":1626163424.417843,"msg":"received update from kv namespace watch"}
{"level":"info","ts":1626163424.4178882,"msg":"refreshed cluster namespaces","added":[],"updated":["default"],"removed":[]}
{"level":"info","ts":1626163424.417898,"msg":"namespace has non-ready staging state status","namespace":"default","status":"unknown"}
{"level":"info","ts":1626163424.9263237,"msg":"successfully parsed placement","version":3}
{"level":"info","ts":1626163424.9265494,"msg":"received update from kv topology watch"}
{"level":"info","ts":1626163425.2612078,"msg":"cluster db successfully marked shards as available","shards":[48,52,53,2,39,42,29,50,57,4,10,14,6,28,45,25,35,43,5,16,37,22,24,31,0,7,18,58,11,30,54,23,51,8,12,13,32,38,44,1,46,60,9,40,59,55,61,62,15,26,36,63,3,33,47,27,49,17,19,21,34,20,41,56]}
{"level":"info","ts":1626163425.2613027,"msg":"successfully parsed placement","version":4}
{"level":"info","ts":1626163425.2615163,"msg":"received update from kv topology watch"}
{"level":"warn","ts":1626163484.1795275,"msg":"cluster connect with consistency any","error":"timed out connecting, returning success"}
{"level":"info","ts":1626163484.1797705,"msg":"successfully updated topology","numHosts":3}
{"level":"info","ts":1626163484.1797967,"msg":"received update for topology"}
{"level":"warn","ts":1626163544.2017615,"msg":"cluster connect with consistency any","error":"timed out connecting, returning success"}
{"level":"info","ts":1626163544.2019,"msg":"successfully updated topology","numHosts":3}
{"level":"error","ts":1626163591.7079751,"msg":"unable to mark namespace as ready","rqID":"4917ab5a-dce9-4793-b156-7e2ee41a947c","error":"namepace default not yet ready, err: unable to satisfy consistency requirements, shards=64: failed to meet consistency level unstrict_majority with 0/3 success, 3 nodes responded, errors: [error fetching tagged from host m3db-cluster-test-rep1-0: connection pool has no connections error fetching tagged from host m3db-cluster-test-rep0-0: connection pool has no connections error fetching tagged from host m3db-cluster-test-rep2-0: connection pool has no connections]"}

M3DB yaml

coordinator: {}

db:
  hostID:
    resolver: file
    file:
      path: /etc/hostname
      timeout: 5m
  client:
    writeConsistencyLevel: majority
    readConsistencyLevel: unstrict_majority
  discovery:
    # The type of discovery configuration used, valid options: [config, m3db_single_node, m3db_cluster, m3aggregator_cluster]
    type: m3db_cluster
    # Defines M3DB discovery via etcd
    m3dbCluster:
      env: test/m3db-cluster-test
      endpoints:
      - http://*:2379
      - http://*:2379
      - http://*:2379

gemiit avatar Jul 13 '21 08:07 gemiit

@ItWangS Where did you find this configuration? The syntax of the endpoints don't look like what we typically see in this config, so thinking that could be the issue as the coordinator may not recognize them as they're currently stated in the config (resulting in a timeout).

gibbscullen avatar Jul 21 '21 18:07 gibbscullen