timesketch
timesketch copied to clipboard
google
ELASTIC_HOST/PORT still needed for docker-compose setup to work
Describe the bug When following the below quick start guide to install Timesketch it will not work as it still tries to search for ELASTIC_HOST and ELASTIC_PORT.
To Reproduce Steps to reproduce the behavior:
- Follow https://github.com/google/timesketch/blob/master/docs/guides/admin/install.md
- Login and create a New Investigation -> 500 internal server error
- Check the worker log for the error below
[2022-01-11 11:59:19,838] timesketch.app/ERROR Exception on /api/v1/sketches/1/ [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
<...cut..?
File "/usr/local/lib/python3.8/dist-packages/opensearchpy/connection/http_urllib3.py", line 136, in __init__
super(Urllib3HttpConnection, self).__init__(
File "/usr/local/lib/python3.8/dist-packages/opensearchpy/connection/base.py", line 155, in __init__
if ":" in host: # IPv6
TypeError: argument of type 'NoneType' is not iterable
- Add ELASTIC_HOST and ELASTIC_PORT to timesketch.conf and restart timesketch containers
- Create a new investigation and see it succeed.
Expected behaviour I expect the setup guide to give me a working Timesketch setup ;)
Desktop (please complete the following information):
- OS: MacOS
- Browser Chrome
- Version 96.0.4664.110
Hi, thanks for the report. I can confirm this error. I am looking into it.
I am wondering why this occures. I have another fresh timesketch running since two days and after I recognized the issue#2097 just altered the variable names to OPENSEARCH_HOST and OPENSEARCH_PORT in the etc/timesketch.conf file and after a restart of timesketch by docker-compose everything works fine. So I expected the modification to the deploy-script which bringing the right variable names and settings in the etc/timesketch.conf file will be enough. And I am quite sure I tested that it works fine, yesterday.
Suprisingly the error logs occured in worker.log only the first time. After that errors where logged in wsgi_error.log.
==== worker.log extract ==== 2022-01-11 21:09:53,606] celery.worker.consumer.consumer/WARNING consumer: Connection to broker lost. Trying to re-establish the connection... Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/celery/worker/consumer/consumer.py", line 318, in start blueprint.start(self) File "/usr/local/lib/python3.8/dist-packages/celery/bootsteps.py", line 119, in start step.start(parent) File "/usr/local/lib/python3.8/dist-packages/celery/worker/consumer/consumer.py", line 596, in start c.loop(*c.loop_args()) File "/usr/local/lib/python3.8/dist-packages/celery/worker/loops.py", line 83, in asynloop next(loop) File "/usr/local/lib/python3.8/dist-packages/kombu/asynchronous/hub.py", line 364, in create_loop cb(*cbargs) File "/usr/local/lib/python3.8/dist-packages/kombu/transport/redis.py", line 1083, in on_readable self.cycle.on_readable(fileno) File "/usr/local/lib/python3.8/dist-packages/kombu/transport/redis.py", line 354, in on_readable chan.handlerstype File "/usr/local/lib/python3.8/dist-packages/kombu/transport/redis.py", line 688, in _receive ret.append(self._receive_one(c)) File "/usr/local/lib/python3.8/dist-packages/kombu/transport/redis.py", line 698, in _receive_one response = c.parse_response() File "/usr/local/lib/python3.8/dist-packages/redis/client.py", line 3185, in parse_response response = self._execute(conn, conn.read_response) File "/usr/local/lib/python3.8/dist-packages/redis/client.py", line 3159, in _execute return command(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 700, in read_response response = self._parser.read_response() File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 310, in read_response response = self._buffer.readline() File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 242, in readline self._read_from_socket() File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 187, in _read_from_socket raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR) redis.exceptions.ConnectionError: Connection closed by server.
==== wsgi_error.log ====
[2022-01-11 21:34:21,852] timesketch.app/ERROR Exception on /api/v1/sketches/3/ [GET] Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn conn = connection.create_connection( File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/opensearchpy/connection/http_urllib3.py", line 256, in perform_request response = self.pool.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 376, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.8/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.8/http/client.py", line 1256, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 951, in send self.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe12983f9a0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1935, in dispatch_request return self.view_functionsrule.endpoint File "/usr/local/lib/python3.8/dist-packages/flask_restful/init.py", line 458, in wrapper resp = resource(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 89, in view return self.dispatch_request(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/flask_restful/init.py", line 573, in dispatch_request resp = meth(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 261, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/timesketch/api/v1/resources/sketch.py", line 459, in get filter_labels=self.datastore.get_filter_labels( File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/datastores/opensearch.py", line 716, in get_filter_labels result = self.client.search( File "/usr/local/lib/python3.8/dist-packages/opensearchpy/client/utils.py", line 177, in _wrapped return func(*args, params=params, headers=headers, **kwargs) File "/usr/local/lib/python3.8/dist-packages/opensearchpy/client/init.py", line 1593, in search return self.transport.perform_request( File "/usr/local/lib/python3.8/dist-packages/opensearchpy/transport.py", line 410, in perform_request raise e File "/usr/local/lib/python3.8/dist-packages/opensearchpy/transport.py", line 373, in perform_request status, headers_response, data = connection.perform_request( File "/usr/local/lib/python3.8/dist-packages/opensearchpy/connection/http_urllib3.py", line 271, in perform_request raise ConnectionError("N/A", str(e), e) opensearchpy.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe12983f9a0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe12983f9a0>: Failed to establish a new connection: [Errno 111] Connection refused) [2022-01-11 21:34:45,777] timesketch.lib.sigma/ERROR None /usr/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used return io.open(fd, *args, **kwargs) [2022-01-11 21:34:45,839] timesketch.app/ERROR Exception on /api/v1/sketches/3/ [GET] Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn conn = connection.create_connection( File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/opensearchpy/connection/http_urllib3.py", line 256, in perform_request response = self.pool.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 376, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.8/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.8/http/client.py", line 1256, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 951, in send self.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe129861ac0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1935, in dispatch_request return self.view_functionsrule.endpoint File "/usr/local/lib/python3.8/dist-packages/flask_restful/init.py", line 458, in wrapper resp = resource(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 89, in view return self.dispatch_request(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/flask_restful/init.py", line 573, in dispatch_request resp = meth(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 261, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/timesketch/api/v1/resources/sketch.py", line 459, in get filter_labels=self.datastore.get_filter_labels( File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/datastores/opensearch.py", line 716, in get_filter_labels result = self.client.search( File "/usr/local/lib/python3.8/dist-packages/opensearchpy/client/utils.py", line 177, in _wrapped return func(*args, params=params, headers=headers, **kwargs) File "/usr/local/lib/python3.8/dist-packages/opensearchpy/client/init.py", line 1593, in search return self.transport.perform_request( File "/usr/local/lib/python3.8/dist-packages/opensearchpy/transport.py", line 410, in perform_request raise e File "/usr/local/lib/python3.8/dist-packages/opensearchpy/transport.py", line 373, in perform_request status, headers_response, data = connection.perform_request( File "/usr/local/lib/python3.8/dist-packages/opensearchpy/connection/http_urllib3.py", line 271, in perform_request raise ConnectionError("N/A", str(e), e) opensearchpy.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe129861ac0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe129861ac0>: Failed to establish a new connection: [Errno 111] Connection refused)
==== Files with ELASTIC_HOST and ELASTIC_PORT ====
-> % grep -r "ELASTIC_HOST" config/dpkg/timesketch-server.timesketch.default:ELASTIC_HOST = '127.0.0.1' contrib/deploy_timesketch.ps1:(Get-Content $timesketchconf).replace("ELASTIC_HOST = '127.0.0.1'", "ELASTIC_HOST = '$OPENSEARCH_ADDRESS'") | Set-Content $timesketchconf contrib/deploy_timesketch.sh:sed -i 's#^ELASTIC_HOST = \x27127.0.0.1\x27#ELASTIC_HOST = \x27'$OPENSEARCH_ADDRESS'\x27#' timesketch/etc/timesketch/timesketch.conf timesketch/app.py: sys.stderr.write('Deprecated config field found: ELASTIC_HOST. ' timesketch/app.py: app.config['OPENSEARCH_HOST'] = app.config.get('ELASTIC_HOST')
-> % grep -r "ELASTIC_PORT" config/dpkg/timesketch-server.timesketch.default:ELASTIC_PORT = 9200 contrib/deploy_timesketch.ps1:(Get-Content $timesketchconf).replace("ELASTIC_PORT = 9200", "ELASTIC_PORT = $OPENSEARCH_PORT") | Set-Content $timesketchconf contrib/deploy_timesketch.sh:sed -i 's#^ELASTIC_PORT = 9200#ELASTIC_PORT = '$OPENSEARCH_PORT'#' timesketch/etc/timesketch/timesketch.conf docker/dev/build/docker-entrypoint.sh: echo "Please pass values for the ELASTIC_ADDRESS and ELASTIC_PORT environment variables" timesketch/app.py: sys.stderr.write('Deprecated config field found: ELASTIC_PORT. ' timesketch/app.py: app.config['OPENSEARCH_PORT'] = app.config.get('ELASTIC_PORT')
I checked it again and I still confirm timesketch is running into errors while opening a new sketch.
But I cannot confirm : Add ELASTIC_HOST and ELASTIC_PORT to timesketch.conf and restart timesketch containers
That didn't worked on my system.
Also checked the timesketch at my other system. It is still running fine with the previously patched OPENSEARCH_HOST setting. But as I said I just fixed it on this system directly in the config file and did not run the deployment skript again. But I couldn't figure out yet, what my changes to the deployment scripts variable names and settings might effect in other places. I think it shouldn't effect anything else.
Here are some of my observations in researching the issue with networking issues in the latest timesketch/opensearch release. I hope they help some and reduces the research time for others working on this. I made the updates to the docker-compose.yml that were consistent with the prior release of timesketch/elasticsearch and although the network stats look like the should work, they do not. I provide results before and after the updates to the docker-compose.yml file.
The docker-compose.yml, as created by deploy_timesketch.sh This file does not have ports settings for opensearch. These results which obviously won't work as there is nothing listening on port 9200.
root@ubuntu:/opt/timesketch# ns
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 5934/docker-proxy
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 713/systemd-resolve
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 873/cupsd
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 5872/docker-proxy
tcp6 0 0 :::80 :::* LISTEN 5943/docker-proxy
tcp6 0 0 ::1:631 :::* LISTEN 873/cupsd
tcp6 0 0 :::443 :::* LISTEN 5895/docker-proxy
root@ubuntu:/opt/timesketch#
root@ubuntu:/opt/timesketch#
root@ubuntu:/opt/timesketch# dkproxy
root 5872 1027 0 04:42 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.18.0.4 -container-port 443
root 5895 1027 0 04:42 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 443 -container-ip 172.18.0.4 -container-port 443
root 5934 1027 0 04:42 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.4 -container-port 80
root 5943 1027 0 04:42 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 80 -container-ip 172.18.0.4 -container-port 80
root 6751 4566 0 04:44 pts/0 00:00:00 grep --color=auto docker-proxy
root@ubuntu:/opt/timesketch#
root@ubuntu:/opt/timesketch#
root@ubuntu:/opt/timesketch# dkps
NAMES STATUS PORTS
redis Up 2 minutes 6379/tcp
timesketch-web Up 2 minutes
opensearch Up 2 minutes 9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp
nginx Up 2 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp
timesketch-worker Up 2 minutes
postgres Up 2 minutes 5432/tcp
root@ubuntu:/opt/timesketch#
root@ubuntu:/opt/timesketch#
root@ubuntu:/opt/timesketch# dknwi | grep opensearch -A 4
"Name": "opensearch",
"EndpointID": "5ebf8f40a78029065bb829075fc8255e127f414ee30c9caaac940882962c43e8",
"MacAddress": "02:42:ac:12:00:07",
"IPv4Address": "172.18.0.7/16",
"IPv6Address": ""
root@ubuntu:/opt/timesketch#
From docker-compose.yml The ports were added to this version of the file. The output below is after I updated docker-compose.yml to add ports 9200 & 9300.
opensearch:
container_name: opensearch
image: opensearchproject/opensearch:${OPENSEARCH_VERSION}
restart: always
environment:
- TAKE_FILE_OWNERSHIP=1
- discovery.type=single-node
- "DISABLE_INSTALL_DEMO_CONFIG=true"
- "DISABLE_SECURITY_PLUGIN=true" # TODO: Enable when we have migrated the python client to Opensearch as well.
- "OPENSEARCH_JAVA_OPTS=-Xms${OPENSEARCH_MEM_USE_GB}g -Xmx${OPENSEARCH_MEM_USE_GB}g"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- ./data/opensearch:/usr/share/opensearch/data/
ports:
- "9200:9200"
- "9300:9300"
root@ubuntu:~#
root@ubuntu:~# ns
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1598/docker-proxy
tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN 1543/docker-proxy
tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN 1428/docker-proxy
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 713/systemd-resolve
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 873/cupsd
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1523/docker-proxy
tcp6 0 0 :::80 :::* LISTEN 1616/docker-proxy
tcp6 0 0 :::9200 :::* LISTEN 1553/docker-proxy
tcp6 0 0 :::9300 :::* LISTEN 1455/docker-proxy
tcp6 0 0 ::1:631 :::* LISTEN 873/cupsd
tcp6 0 0 :::443 :::* LISTEN 1528/docker-proxy
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~# dkproxy
root 1428 1027 0 04:04 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9300 -container-ip 172.23.0.5 -container-port 9300
root 1455 1027 0 04:04 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 9300 -container-ip 172.23.0.5 -container-port 9300
root 1523 1027 0 04:04 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.23.0.6 -container-port 443
root 1528 1027 0 04:04 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 443 -container-ip 172.23.0.6 -container-port 443
root 1543 1027 0 04:04 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9200 -container-ip 172.23.0.5 -container-port 9200
root 1553 1027 0 04:04 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 9200 -container-ip 172.23.0.5 -container-port 9200
root 1598 1027 0 04:04 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.23.0.6 -container-port 80
root 1616 1027 0 04:04 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 80 -container-ip 172.23.0.6 -container-port 80
root 4710 4640 0 04:30 pts/1 00:00:00 grep --color=auto docker-proxy
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~# dkps
NAMES STATUS PORTS
timesketch-worker Up 26 minutes
nginx Up 26 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp
opensearch Up 26 minutes 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9600/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp, 9650/tcp
timesketch-web Up 26 minutes
postgres Up 26 minutes 5432/tcp
redis Up 26 minutes 6379/tcp
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~# dknwi | grep opensearch -A 4
"Name": "opensearch",
"EndpointID": "6d8c17843de5f5f599fb808c30eef727956c4a86aca7785d5cd9f4634e08955a",
"MacAddress": "02:42:ac:17:00:05",
"IPv4Address": "172.23.0.5/16",
"IPv6Address": ""
root@ubuntu:~#
Observation - of standard config created by deploy_timesketch.sh
- There no processes listening on port 9200 and 9300 the port for opensearch as seen by running
netstat -plnt. - The docker -compose.yml file does not define ports for opensearch where the previous timesketch/elasticsearch release docker-compose.yml file did.
- How does the Docker container opensearch get it's port settings? When you run docker ps -a you can see the ports that were defined when it docker was run to create the container but where do they come from if they are not in the docker-compose.yml file?
- The timesketch.conf file has the var OPENSEARCH_HOST = 'opensearch'. How does opensearch get resolved? Is it defined in the default network created by docker-compse? Running docker network inspect timesketch_default, I can see opensearch but I'm not sure if that is the container name or a host name. I can't access the opensearch with something like ping opensearch. Not sure if that should even be expected to work put the way it is used in the timesketch.conf as a hostname it might. (OPENSEARCH_HOST = 'opensearch')
- An install of only opensearch, following the steps on opesearch.org , does define the ports in the docker-compose.yml and you can connect to loclhost:9200 from your browser.
- i can not connect to localhost:9200 in the timesketch/opensearch. I an connect to localhost:9200 in the timesketch only install as well as the timesketch / elasticsearch version.
- To state the obvious, looks like we have a networking issue with opensearch
Okay, let's try to filter out some info.
- docker compose creates it's own network, host names are auto resolved as per name in the docker-compose config
- docker compose instances can connect to other instances without having to define ports/export definitions. 'ports' is for exposure to the external host, expose is only for documentation (docker ps etc) and has no effect on actual networking (except for a few edge cases)
So in a docker-compose setup you can have a container (opensearch) start a service listenening on port 9200 and all other containers are able to connect to it using eg nc opensearch 9200. No need to define any port/export in the configuration.
You can test this by getting a shell in one of the containers (eg docker exec -ti [container_id] sh and nc-ing/pinging any of the other containers by name.
Hope that clarifies some networking things. See https://docs.docker.com/compose/networking/#multi-host-networking
hacktobeer, thanks so much. that does help fill in some of the holes in my docker networking knowledge. i was on that path but the issue is that the ping command is not in these containers; neither is apt or apt-get. I'm not sure what the base image is that is being used. These containers are different than the timesketch/elasticsearch containers that do have ping. All of the container to container networking can be verified in those containers because the tool are there. Ideas on other approaches to test container to container networking?
root@ubuntu:~#
root@ubuntu:~# docker exec -u root -it opensearch /bin/bash
bash-4.2# ping timesketch-web
bash: ping: command not found
bash-4.2#
Back the issue at hand. We know that the docker-compse.yml needs to have the ports added to the opensearch section. I did that, but that still does not fix the problem. My simple test is can I reach opensearch from my browser http://localhost:9200 Which I can not do even after making the ports change to the docker-compose.yml and restarting all the containers. docker-compose down && docker-compose up -d
From all the network data the I collected I can't find anything glaring wrong. Thoughts?
Since this is more conversational, what do you think about moving this to the Open Source DFIR Slack Workspace?
Let's move to Slack. Last comments on this:
- opensearch does not need ports defined in the configuration as it does not need exposure to the host (aka outside of the docker-compose network).
- if you can connect from inside one of the timesketch containers to opensearch it all works networking wise.
On Wed, Jan 12, 2022, 18:18 Mark Hallman @.***> wrote:
hacktobeer, thanks so much. that does help fill in some of the holes in my docker networking knowledge. i was on that path but the issue is that the ping command is not in these containers; neither is apt or apt-get. I'm not sure what the base image is that is being used. These containers are different than the timesketch/elasticsearch containers that do have ping. All of the container to container networking can be verified in those containers because the tool are there. Ideas on other approaches to test container to container networking?
@.:~# @.:~# docker exec -u root -it opensearch /bin/bash bash-4.2# ping timesketch-web bash: ping: command not found bash-4.2#
Back the issue at hand. We know that the docker-compse.yml needs to have the ports added to the opensearch section. I did that, but that still does not fix the problem. My simple test is can I reach opensearch from my browser http://localhost:9200 Which I can not do even after making the ports change to the docker-compose.yml and restarting all the containers. docker-compose down && docker-compose up -d
From all the network data the I collected I can't find anything glaring wrong. Thoughts?
Since this is more conversational, what do you think about moving this to the Open Source DFIR Slack Workspace?
— Reply to this email directly, view it on GitHub https://github.com/google/timesketch/issues/2103#issuecomment-1011276194, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABP5D4AA72BMUG4FSUTX3SDUVWZXLANCNFSM5LWVXSWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
AFAIK this bug has been fixed with https://github.com/google/timesketch/pull/2099
If it remained please feel free to re-open
