3scale
APIcast hostname resolution issue
[APIcast doesn't resolve hostname without a TLD (.)]
Version
Containers from both APIcast v3.2.0 and v3.3.0 (possibly reproduced on earlier versions as well):
- registry.access.redhat.com/3scale-amp22/apicast-gateway
- registry.access.redhat.com/3scale-amp23/apicast-gateway
Steps To Reproduce
NOTE: The steps below are being reproduced using the Docker implementation of APIcast.
- Create a new Internal Network:
$ docker network create --internal internal_network
- Deploy an Apache (httpd) dummy container with the name "webhost" attached to this network:
$ docker container run --name webhost --network internal_network httpd
-
Create a new API calling "webhost" and using a self-managed APIcast: 3.1. Private Base URL: http://webhost:80 3.2. Staging and Production Public Base URL: http://localhost:8080 3.3. Update the Staging Environment and Promote to Production.
-
Deploy an APIcast self-managed container based on any of the Versions described above (in the example below, 'registry.access.redhat.com/3scale-amp23/apicast-gateway:latest' is being used):
$ docker container run --name apicast -e THREESCALE_PORTAL_ENDPOINT=https://<token>@<user>-admin.3scale.net registry.access.redhat.com/3scale-amp23/apicast-gateway:latest
- Attach the Internal Network to the container:
$ docker network connect internal_network apicast
- Open a shell (bash) into the currently running apicast:
$ docker exec -it apicast bash
- Curl the hostname "webhost" in order to ensure that the O.S. level name resolution is working:
$ curl -X GET -I http://webhost:80
HTTP/1.1 200 OK Date: Fri, 19 Oct 2018 18:18:01 GMT Server: Apache/2.4.34 (Unix) Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT ETag: "2d-432a5e4a73a80" Accept-Ranges: bytes Content-Length: 45 Content-Type: text/html
- Use the same command on APIcast:
$ curl -X GET -I http://localhost:8080?user_key=<user_key>
HTTP/1.1 503 Service Temporarily Unavailable Server: openresty/1.13.6.2 Date: Fri, 19 Oct 2018 18:18:46 GMT Content-Type: text/html Content-Length: 219 Connection: keep-alive`
- Observe the following lines on the APIcast Stdout/Stderr:
2018/10/19 18:19:31 [error] 34#34: *86 [lua] balancer.lua:41: get_peer(): could not select peer: empty peers while connecting to upstream, client: 127.0.0.1, server: _, request: "GET /?user_key=<user_key> HTTP/1.1", host: "webhost" [19/Oct/2018:18:19:33 +0000] localhost:8080 127.0.0.1:56488 "GET /?user_key=<user_key> HTTP/1.1" 503 219 (4.100) 2.1
- The results above differs when using a full domain name with dots - e.g. "webhost.internal_network" as the Private Base URL:
$ curl -X GET -I http://webhost.internal_network
HTTP/1.1 200 OK Date: Fri, 19 Oct 2018 18:23:11 GMT Server: Apache/2.4.34 (Unix) Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT ETag: "2d-432a5e4a73a80" Accept-Ranges: bytes Content-Length: 45 Content-Type: text/html
$ curl -X GET -I http://localhost:8080?user_key=<user_key>
HTTP/1.1 200 OK Server: openresty/1.13.6.2 Date: Fri, 19 Oct 2018 18:24:03 GMT Content-Type: text/html Content-Length: 45 Connection: keep-alive Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT ETag: "2d-432a5e4a73a80" Accept-Ranges: bytes
[19/Oct/2018:18:24:03 +0000] localhost:8080 127.0.0.1:56610 "GET /?user_key=<user_key> HTTP/1.1" 200 45 (0.900) 0`
Current Result
APIcast doesn't resolve hostname, only hostname.domain.
Expected Result
APIcast should resolve both hostname and hostname.domain.
Additional Information
-
Since the O.S. resolves both hostname and hosnamet.domain, we are able to isolate this issue to the APIcast name resolution.
-
Adding the host in /etc/hosts doesn't solve the issue.
@estevaobk thank you! Great steps to reproduce.
Here is some additional research:
Using docker nameserver:
# /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0
;; QUESTION SECTION:
;webhost.internal_network. IN A
;; ANSWER SECTION:
webhost.internal_network. 600 IN A 172.18.0.2
;; Query time: 1 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Mon Oct 22 09:24:14 UTC 2018
;; MSG SIZE rcvd: 82
;; QUESTION SECTION:
;webhost. IN A
;; ANSWER SECTION:
webhost. 600 IN A 172.18.0.2
;; Query time: 2 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Mon Oct 22 09:26:17 UTC 2018
;; MSG SIZE rcvd: 48
And using our dnsmasq:
;; QUESTION SECTION:
;webhost. IN A
;; Query time: 1 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1)
;; WHEN: Mon Oct 22 09:26:41 UTC 2018
;; MSG SIZE rcvd: 25
;; QUESTION SECTION:
;webhost.internal_network. IN A
;; ANSWER SECTION:
webhost.internal_network. 600 IN A 172.18.0.2
;; Query time: 2 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1)
;; WHEN: Mon Oct 22 09:26:57 UTC 2018
;; MSG SIZE rcvd: 82
We are starting dnsmasq with: --listen-address=127.0.0.1 --port=5353 --all-servers --no-host --no-hosts --cache-size=1000 --no-negcache --domain-needed to not resolve naked domains.
That was done in https://github.com/3scale/apicast/commit/b58bab8493d793faed4a7b3853cf0a2081d8fcbf. We need to verify if that is still needed.
@estevaobk as a workaround you can set proper search scope in /etc/resolv.conf like:
search internal
nameserver 127.0.0.11
options ndots:0
In plain docker that is possible by using --dns-search flag when starting the contianer.
@mikz That didn't work for me. I've added the nginx 'resolver' directive to conf/nginx.conf (127.0.0.11) and now my Docker hostnames are properly resolved. We are still on v3.0.0. Maybe my issue is not the same as above. Docker hostnames were not resolved by Lua. Similar to this issue: https://github.com/openresty/lua-resty-redis/issues/59 Setting the RESOLVER env var to 127.0.0.11:53 did not work.
@mikz I've also found your comment here: https://github.com/nmasse-itix/apicast-logger/issues/2#issuecomment-345989096 Sorry for the confusion. Your answer was meant for newer versions.
@ddevaal v3.0.0 is significantly different from now. The logic of resolving changed several times since then.
