st2-docker
st2-docker copied to clipboard
Problem installing or searching pack, temporary failure in name resolution
Describe the problem A clear and concise description of what the problem is.
Versions
- Host OS: Fedora 33 And Ubuntu 18.04
- docker: Docker version 20.10.3, build 48d30b5
- docker-compose: docker-compose version 1.28.2, build 67630359
- stackstorm version: v3.3.0
To Reproduce If necessary, please specify steps to reproduce the behavior.
Login to st2client with:
docker-compose exec st2client bash
And then try to search or install.
It will fail with:
st2 pack search datadog
root@554d99419ce3:/opt/stackstorm# st2 pack search docker
ERROR: 400 Client Error: Bad Request
MESSAGE: No results from the index: tried https://index.stackstorm.org/v1/index.json.
Status: [
{
"url": "https://index.stackstorm.org/v1/index.json",
"packs": 0,
"message": "ConnectionError(MaxRetryError(\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f448e565710>: Failed to establish a new connection: [Errno -3] Lookup timed out',))\",),)",
"error": "unresponsive"
}
] for url: http://st2api:9101/packs/index/search
root@554d99419ce3:/opt/stackstorm# st2 pack install datadog
[ succeeded ] init_task
[ failed ] download_pack
id: 6038397af6cde4fdb05d5c08
action.ref: packs.install
parameters:
packs:
- datadog
python3: false
status: failed
start_timestamp: Thu, 25 Feb 2021 23:57:46 UTC
end_timestamp: Thu, 25 Feb 2021 23:57:59 UTC
result:
errors:
- message: Execution failed. See result for details.
result:
exit_code: 1
result: None
stderr: "Index parsing error: {
"url": "https://index.stackstorm.org/v1/index.json",
"packs": 0,
"message": "ConnectionError(MaxRetryError(\\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8ff5cdc18>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))\\",),)",
"error": "unresponsive"
}
Traceback (most recent call last):
File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 334, in <module>
obj.run()
File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 193, in run
output = action.run(**self._parameters)
File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 90, in run
return self._validate_result(result=result, repo_url=pack_url)
File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 111, in _validate_result
raise Exception(message)
Exception: The pack has not been downloaded from "None".
Errors:
No results from the index: tried https://index.stackstorm.org/v1/index.json.
Status: [
{
"url": "https://index.stackstorm.org/v1/index.json",
"packs": 0,
"message": "ConnectionError(MaxRetryError(\\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8ff5cdc18>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))\\",),)",
"error": "unresponsive"
}
]
"
stdout: ''
task_id: download_pack
type: error
output:
conflict_list: null
message: ''
packs_list: null
warning_list: null
+--------------------------+------------------------+---------------+----------------+-------------------------------+
| id | status | task | action | start_timestamp |
+--------------------------+------------------------+---------------+----------------+-------------------------------+
| 6038397b1ac1577e7e12dae3 | succeeded (0s elapsed) | init_task | core.noop | Thu, 25 Feb 2021 23:57:47 UTC |
| 6038397c1ac1577e7e12daf3 | failed (11s elapsed) | download_pack | packs.download | Thu, 25 Feb 2021 23:57:48 UTC |
If try to do some curl... from st2api container:
root@2bc662f52cab:/opt/stackstorm# curl https://index.stackstorm.org/v1/index.json|wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0curl: (6) Could not resolve host: index.stackstorm.org
0
Modifying resolve.conf for adding 1.1.1.1 after docker resolution improves the thing but still.
curl https://index.stackstorm.org/v1/index.json|wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 678k 0 678k 0 0 406k 0 --:--:-- 0:00:02 --:--:-- 406k
16298
From s2client container on the other hand it's inmediate.
curl https://index.stackstorm.org/v1/index.json|wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 678k 0 678k 0 0 3879k 0 --:--:-- --:--:-- --:--:-- 3857k
16298
For some reason, s2api container it's taking more time to resolv... But even with that it's no reason for getting a Temporary failure in name resolution.
Am i installing them wrong?
By default bridge networks dont forward to the outside world and need to have additional parameters set on the host for this to work correctly.
https://docs.docker.com/network/bridge/#enable-forwarding-from-docker-containers-to-the-outside-world
@minsis I tried it, and indeed also tried changing the type of network, but the problem is still the same.
@JustDevZero We also faced some of these issues (not with StackStorm) when ipv6 was enabled and ultimately disabled it. Do you also have ipv6 enabled?
I'll check it as soon as I'm at my office hours, but I think you're maybe having a point on the ipv6, AFAIK we have ipv6 disabled on whole network.
El jue., 4 mar. 2021 22:34, Dennis Whitney [email protected] escribió:
@JustDevZero https://github.com/JustDevZero We also faced some of these issues (not with StackStorm) when ipv6 was enabled and ultimately disabled it. Do you also have ipv6 enabled?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/StackStorm/st2-docker/issues/216#issuecomment-790959942, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABKFV33A6MSV3UXJOM4ZJTTB74GVANCNFSM4YHOXTIQ .
@JustDevZero The network itself can have ipv6 disabled as a whole but that doesn't prevent localhost from having it enabled. Every device has its own link local ipv6 address when it comes up.
It looks that it's ipv6 fault then, host have ipv6 disabled but somehow docker it's still trying to get ipv6 not 4. So... I guess it's matter of dealing with docker config, maybe?
Hi,
I'm also facing the same issue, from curl it works perfectly.
Every action in python 3 that need to resolve a DNS happens the same problem:
@JustDevZero have you manager to find out what was the root cause of this issue?
Versions
- Host OS: Ubuntu 20.04
- docker: Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1
- docker-compose version 1.29.1, build c34c88b2
- stackstorm version: v3.5.0
- Python: v3.6.9
This fixed the issue for me:
diff --git a/docker-compose.yml b/docker-compose.yml
index 3a05ac0..b1c47fd 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -2,6 +2,9 @@ version: '3'
services:
st2web:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2web:${ST2_VERSION:-latest}
restart: on-failure
environment:
@@ -42,6 +45,9 @@ services:
dns_search: .
command: /makesecrets.sh
st2api:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2api:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -65,6 +71,9 @@ services:
- ${ST2_PACKS_DEV:-./packs.dev}:/opt/stackstorm/packs.dev:rw
dns_search: .
st2stream:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2stream:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -76,6 +85,9 @@ services:
- ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
dns_search: .
st2scheduler:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2scheduler:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -88,6 +100,9 @@ services:
- ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
dns_search: .
st2workflowengine:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2workflowengine:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -101,6 +116,9 @@ services:
- stackstorm-keys:/etc/st2/keys:ro
dns_search: .
st2auth:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2auth:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -113,6 +131,9 @@ services:
- ./files/htpasswd:/etc/st2/htpasswd:ro
dns_search: .
st2actionrunner:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2actionrunner:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -133,6 +154,9 @@ services:
- stackstorm-keys:/etc/st2/keys:ro
dns_search: .
st2garbagecollector:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2garbagecollector:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -144,6 +168,9 @@ services:
- ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
dns_search: .
st2notifier:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2notifier:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -156,6 +183,9 @@ services:
- ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
dns_search: .
st2rulesengine:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2rulesengine:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -167,6 +197,9 @@ services:
- ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
dns_search: .
st2sensorcontainer:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2sensorcontainer:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -182,6 +215,9 @@ services:
- stackstorm-packs-configs:/opt/stackstorm/configs:ro
- ${ST2_PACKS_DEV:-./packs.dev}:/opt/stackstorm/packs.dev:ro
st2timersengine:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2timersengine:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -192,6 +228,9 @@ services:
volumes:
- ./files/st2.docker.conf:/etc/st2/st2.docker.conf:ro
st2client:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2actionrunner:${ST2_VERSION:-latest}
restart: on-failure
depends_on:
@@ -222,6 +261,9 @@ services:
- ./files/st2-cli.conf:/root/.st2/config
- ./scripts/st2client-startup.sh:/st2client-startup.sh
st2chatops:
+ dns:
+ - 8.8.8.8
+ - 4.4.4.4
image: ${ST2_IMAGE_REPO:-stackstorm/}st2chatops:${ST2_VERSION:-latest}
restart: on-failure:5
depends_on:
By default ipv4 forwarding is not enabled. This is mostly where the issue is with getting traffic to leave the container.
Add net.ipv4.ip_forward=1
to /etc/sysctl.conf to fix it.