ignored "-proto_dist inet6_tcp"

[sergey-safarov]

Description

I cannot form cluster in ipv6 network

Steps to Reproduce

1. add "-proto_dist inet6_tcp" into "/opt/couchdb/etc/vm.args"
2. open fauxton interface and add host using DNS name that resolved only AAAA record.

Expected Behaviour

CouchDB try resolve AAAA record

Your Environment

CouchDB resolve A record

• CouchDB version used: 2.3.1
• Browser name and version: Google Chrome
• Operating system and version: CentOS 8

Additional Context

None

[wohali]

@dch You did some work on IPv6 a while back, know anything about this?

[CanRau]

Would also love to get CouchDB going using IPv6 to be able to deploy a cluster on Fly.io

[dch]

works fine here on FreeBSD but that shouldn't matter. If you find a working config for fly, let us know.

for each node, just have [email protected] ... [email protected] etc for name field. the rest of your ini & args files should be the same as usual.

vm.args snippet

-name [email protected]
-setcookie ...
-proto_dist inet6_tcp
-kernel inet_dist_listen_min 4370
-kernel inet_dist_listen_max 4400
...

local.ini snippet

[chttpd]
port = 5984
bind_address = ::
...

consider changing the bind_address to something less open once you know what you're doing.

[CanRau]

Trying that I get

{conn_failed,{error,nxdomain}}

when using a899cafd.vm.crcouchdb1.internal as remote host, this is fly's private networking and :: as bind address.

I'm also trying to debug with the Fly folks over at community.fly.io, so far having issues querying the instances IPv6 to try that instead of the hostname as mentioned over there.

Okay so resolved the IP

trying fdaa:0:328d:a7b:7d16:1:f46:2 as remote results in {url_parsing_failed{error,invalid_uri}}

whereas [fdaa:0:328d:a7b:7d16:1:f46:2] returns the same as before {conn_failed,{error,nxdomain}}

all using the UI, haven't tried via API so far

UPDATE seems like same errors happen using the API

[dch]

@CanRau appreciate this is frustrating but I think you should start off ensuring the erlang layer works first, getting distributed erlang working, and posting more of your config/setup, etc.

I've just checked and a bare ipv6 e.g. fca2:...::3 works as well as :: for bind_address, no brackets required. However this isn't important for the clustering part of couchdb, the -name [email protected] bit is.

the nxdomain error shows something is trying to resolve the domain and is being told NO. This isn't a couchdb problem, so I suggest you shell into the container, and start a shell inside to check DNS resolution. something like this:

$ dig AAAA a899cafd.vm.crcouchdb1.internal
$ dig A    a899cafd.vm.crcouchdb1.internal
$ erl
Erlang/OTP 25 [erts-13.0] [source] [64-bit] [smp:32:32] [ds:32:32:10] [async-threads:1] [jit] [sharing-preserving]

Eshell V13.0  (abort with ^G)
1> net:gethostname().
{ok,"wintermute.skunkwerks.at"}
2> inet:getaddr("localhost", inet6).
{ok,{0,0,0,0,0,0,0,1}}
3> inet:getaddr("a899cafd.vm.crcouchdb1.internal", inet6).
{error,nxdomain}

I'm hoping 3 has a better outcome for you.

Once you have that working, set up a pair of distributed erlang nodes using the same lines from vm.args above, and see if that works. If not, it's a networking issue at fly.

Once that's working, you can come back to the couchdb side and see how you get on.

You may find some more inspiration from my older couchdb 2.3 (but still relevant) notes-to-self the first time I set this up, https://hackmd.io/@dch/HJXCjUpeV its just a walk-through from the excellent couch docs anyway.

I hope this works, having couchdb at fly would be a nice improvement.

[big-r81]

@CanRau Is this still an issue?

[CanRau]

Sorry I don't know as I'm not working on this anymore, so at least for me it's not and could be closed.

[sergey-safarov]

I can make a test. Is required to make the test on the latest stable version? Can I test using apache/couchdb:latest docker image?

[big-r81]

Try it with the latest source from main or with 3.2.2. I can't say something to use inside docker, but you can test it...

[dch]

This still works perfectly for me, since 2.3.x til 3.2.2 atm.

[big-r81]

Closing wil close this for now, @sergey-safarov if you have a reproducible test case, please re-open this issue.

[sergey-safarov]

Sorry for the long delay I have reproduced the issue. In my installation, I have 3 node cluster that works IPv6 only (on the server no IPv4 address on the network interface card). But when I try to replicate the database from another server I can see "A" DNS requests. To start replications on my filesystem created /tmp/json.txt with content

{
  "source": {
    "headers": {},
    "url": "http://admin:[email protected]:5984/alerts"
  },
  "target": {
    "headers": {},
    "url": "http://admin:[email protected]:5984/alerts"
  },
  "continuous": false,
  "create_target": false
}

And then use curl command to start replication

curl -s -S \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/javascript' \
  --data-binary @/tmp/json.txt \
  http://admin:[email protected]:5984/_replicate

and in the response has

{"error":"replication_auth_error","reason":"{session_request_failed,\"http://source.local:5984/_session\",\"admin\",\n                        {conn_failed,{error,nxdomain}}}"}

Tested on 2.3.1 and will recheck on the latest docker image

[sergey-safarov]

I have reproduced the issue on apache/couchdb:3.3.2 docker image.

I tarted replicaation using json

{
  "user_ctx": {
    "name": "admin",
    "roles": [
      "_admin",
      "_reader",
      "_writer"
    ]
  },
  "source": {
    "url": "http://ippbx-1a.la.example.com:5984/mydb",
    "headers": {
      "Authorization": "Basic XXXXXXXXXXXX"
    }
  },
  "target": {
    "url": "http://127.0.0.1:5984/mydb",
    "headers": {
      "Authorization": "Basic XXXXXXXXX"
    }
  },
  "create_target": true,
  "continuous": false
}

in the CouchDB output I see

Jul 07 09:21:51 ippbx-1b.la.example.com docker[3677369]: [error] 2023-07-07T09:21:51.599467Z [email protected] <0.6095.0> -------- couch_replicator_httpc: auth plugin initialization failed "http://ippbx-1a.la.example.com:5984/mydb/" {session_request_failed,"http://ippbx-1a.la.example.com:5984/_session","admin",{conn_failed,{error,nxdomain}}} Jul 07 09:21:51 ippbx-1b.la.example.com docker[3677369]: [error] 2023-07-07T09:21:51.599676Z [email protected] <0.6095.0> -------- throw:{replication_auth_error,{session_request_failed,"http://ippbx-1a.la.example.com:5984/_session","admin",{conn_failed,{error,nxdomain}}}}: Replication 69411eb9179c970995746a1ecd781fac+create_target failed to start "http://ippbx-1a.la.example.com:5984/mydb/" -> "http://127.0.0.1:5984/mydb/" doc <<"shards/00000000-7fffffff/_replicator.1688721035">>:<<"477deb9261a9927a1d20e90f2a001b31">> stack:[{couch_replicator_httpc,setup,1,[{file,"src/couch_replicator_httpc.erl"},{line,62}]},{couch_replicator_api_wrap,db_open,3,[{file,"src/couch_replicator_api_wrap.erl"},{line,68}]}]

And the tcpdump output

09:23:15.476762 IP 127.0.0.1.56494 > 127.0.0.1.domain: 63155+ A? ippbx-1a.la.example.com. (40)
09:23:15.476905 IP 127.0.0.1.domain > 127.0.0.1.56494: 63155* 0/1/0 (140)
09:23:15.476996 IP 127.0.0.1.55970 > 127.0.0.1.domain: 52657+ A? ippbx-1a.la.example.com. (40)
09:23:15.477078 IP 127.0.0.1.domain > 127.0.0.1.55970: 52657* 0/1/0 (140)

In the vm.args

[centos@ippbx-1b ~]$ grep proto_dist /opt/couchdb/etc/vm.args
-proto_dist inet6_tcp

[sergey-safarov]

Issue recreated at https://github.com/apache/couchdb/issues/4668