fast-data-dev
fast-data-dev copied to clipboard
lensesio
Unable to get the application running
Hey all, working on trying to get Kafka / Confluent running with FDD, and am ... well unsuccessful. I'm getting the following when trying to run the application on AWS EC2:
Authenticating with public key "HR-DevOps-HRTech"
ubuntu@ip-172-16-75-233:~$ screen -S kafka -- sudo docker run -it -v /srv/kafka:/tmp --rm --net=host -e ADV_HOST=172.16.75.233 landoop/fast-data-dev 256a5053f0c5: Pull complete df59bd94bdd4: Pull complete 6aac547c4d90: Pull complete 1818c7d8b194: Pull complete 8297d3821b7b: Pull complete 27f5c49688ee: Pull complete 3a13574f4ddd: Pull complete 13d13524515b: Pull complete dd4dddc9dc97: Pull complete 633bee4809e9: Pull complete a6842c7eb2cf: Pull complete 22ae19fc8a44: Pull complete c58290624b9c: Pull complete 09afeafbcca3: Pull complete 30033ace6359: Pull complete 14951d7d8ded: Pull complete 6797a96a7092: Pull complete 5e958485d959: Pull complete 6e77d9fc1e81: Pull complete 99046468067f: Pull complete def900f2c3f5: Pull complete e3298ece27a7: Pull complete 46b35ea8b707: Pull complete 498af3d85513: Pull complete c29a64ba4098: Pull complete d5267660efab: Pull complete 068d760d616b: Pull complete 12c2d93a4a0c: Pull complete Digest: sha256:041358122a185144261c95cb8abfb38d47461d585e1672ed58d3ce9800634629 Status: Downloaded newer image for landoop/fast-data-dev:latest Setting advertised host to 172.16.75.233. Operating system RAM available is 3563 MiB, which is less than the lowest recommended of 5120 MiB. Your system performance may be seriously impacted. Starting services. This is landoop’s fast-data-dev. Kafka 0.11.0.1, Confluent OSS 3.3.1. You may visit http://172.16.75.233:3030 in about a minute. 2018-02-28 14:42:48,528 CRIT Supervisor running as root (no user in config file) 2018-02-28 14:42:48,528 INFO Included extra file "/etc/supervisord.d/01-fast-data.conf" during parsing 2018-02-28 14:42:48,528 INFO Included extra file "/etc/supervisord.d/99-supervisord-sample-data.conf" during parsing 2018-02-28 14:42:48,530 INFO supervisord started with pid 6 2018-02-28 14:42:49,532 INFO spawned: 'sample-data' with pid 89 2018-02-28 14:42:49,534 INFO spawned: 'zookeeper' with pid 90 2018-02-28 14:42:49,535 INFO spawned: 'caddy' with pid 91 2018-02-28 14:42:49,537 INFO spawned: 'broker' with pid 92 2018-02-28 14:42:49,538 INFO spawned: 'smoke-tests' with pid 94 2018-02-28 14:42:49,540 INFO spawned: 'connect-distributed' with pid 95 2018-02-28 14:42:49,541 INFO spawned: 'logs-to-kafka' with pid 96 2018-02-28 14:42:49,543 INFO spawned: 'schema-registry' with pid 97
┌────────────────────────────────────────────────────────────────────┐ │ • MobaXterm 10.4 • │ │ (SSH client, X-server and networking tools) │ │ │ │ ➤ SSH session to [email protected] │ │ • SSH compression : ✔ │ │ • SSH-browser : ✔ │ │ • X11-forwarding : ✔ (remote display is forwarded through SSH) │ │ • DISPLAY : ✔ (automatically set on remote server) │ │ │ │ ➤ For more info, ctrl+click on help or visit our website │ └────────────────────────────────────────────────────────────────────┘
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-1049-aws x86_64)
- Documentation: https://help.ubuntu.com
- Management: https://landscape.canonical.com
- Support: https://ubuntu.com/advantage
Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud
0 packages can be updated. 0 updates are security updates.
The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law.
/usr/bin/xauth: file /home/ubuntu/.Xauthority does not exist
To run a command as administrator (user "root"), use "sudo
ubuntu@ip-172-16-75-233:~$ sudo apt-get update sudo: unable to resolve host ip-172-16-75-233 Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial InRelease Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB] Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB] Get:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/main Sources [868 kB] Get:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/restricted Sources [4,808 B] Get:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/universe Sources [7,728 kB] Get:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/multiverse Sources [179 kB] Get:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,532 kB] Get:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/universe Translation-en [4,354 kB] Get:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [144 kB] Get:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB] Get:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Sources [298 kB] Get:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/restricted Sources [2,524 B] Get:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Sources [191 kB] Get:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [7,968 B] Get:16 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [736 kB] Get:17 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [305 kB] Get:18 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [592 kB] Get:19 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [239 kB] Get:20 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [16.2 kB] Get:21 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [8,052 B] Get:22 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports/main Sources [3,432 B] Get:23 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports/universe Sources [4,900 B] Get:24 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [4,836 B] Get:25 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [3,220 B] Get:26 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [6,628 B] Get:27 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [3,768 B] Get:28 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Get:29 http://security.ubuntu.com/ubuntu xenial-security/main Sources [116 kB] Get:30 http://security.ubuntu.com/ubuntu xenial-security/restricted Sources [2,116 B] Get:31 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [59.1 kB] Get:32 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [1,516 B] Get:33 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [459 kB] Get:34 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [198 kB] Get:35 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [320 kB] Get:36 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [119 kB] Get:37 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3,208 B] Get:38 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [1,408 B] Fetched 24.9 MB in 7s (3,200 kB/s) Reading package lists... Done ubuntu@ip-172-16-75-233:~$ sudo apt-get install \
apt-transport-https \ ca-certificates \ curl \ software-properties-common
sudo: unable to resolve host ip-172-16-75-233 Reading package lists... Done Building dependency tree Reading state information... Done ca-certificates is already the newest version (20170717~16.04.1). software-properties-common is already the newest version (0.96.20.7). The following additional packages will be installed: libcurl3-gnutls The following packages will be upgraded: apt-transport-https curl libcurl3-gnutls 3 upgraded, 0 newly installed, 0 to remove and 48 not upgraded. Need to get 349 kB of archives. After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 curl amd64 7.47.0-1ubuntu2.6 [138 kB] Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libcurl3-gnutls amd64 7.47.0-1ubuntu2.6 [184 kB] Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 apt-transport-https amd64 1.2.25 [26.1 kB] Fetched 349 kB in 0s (13.9 MB/s) (Reading database ... 51122 files and directories currently installed.) Preparing to unpack .../curl_7.47.0-1ubuntu2.6_amd64.deb ... Unpacking curl (7.47.0-1ubuntu2.6) over (7.47.0-1ubuntu2.5) ... Preparing to unpack .../libcurl3-gnutls_7.47.0-1ubuntu2.6_amd64.deb ... Unpacking libcurl3-gnutls:amd64 (7.47.0-1ubuntu2.6) over (7.47.0-1ubuntu2.5) ... Preparing to unpack .../apt-transport-https_1.2.25_amd64.deb ... Unpacking apt-transport-https (1.2.25) over (1.2.24) ... Processing triggers for man-db (2.7.5-1) ... Processing triggers for libc-bin (2.23-0ubuntu10) ... Setting up libcurl3-gnutls:amd64 (7.47.0-1ubuntu2.6) ... Setting up curl (7.47.0-1ubuntu2.6) ... Setting up apt-transport-https (1.2.25) ... Processing triggers for libc-bin (2.23-0ubuntu10) ... ubuntu@ip-172-16-75-233:~$ sudo nano /etc/hosts sudo: unable to resolve host ip-172-16-75-233 ubuntu@ip-172-16-75-233:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo: unable to resolve host ip-172-16-75-233 OK ubuntu@ip-172-16-75-233:~$ sudo apt-key fingerprint 0EBFCD88 sudo: unable to resolve host ip-172-16-75-233 pub 4096R/0EBFCD88 2017-02-22 Key fingerprint = 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88 uid Docker Release (CE deb) [email protected] sub 4096R/F273FCD8 2017-02-22
ubuntu@ip-172-16-75-233:~$ sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu
$(lsb_release -cs)
stable" sudo: unable to resolve host ip-172-16-75-233 ubuntu@ip-172-16-75-233:~$ sudo apt-get update sudo: unable to resolve host ip-172-16-75-233 Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial InRelease Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease Get:4 https://download.docker.com/linux/ubuntu xenial InRelease [65.8 kB] Hit:5 http://security.ubuntu.com/ubuntu xenial-security InRelease Get:6 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages [3,329 B] Fetched 69.1 kB in 1s (67.2 kB/s) Reading package lists... Done ubuntu@ip-172-16-75-233:~$ sudo apt-get install docker-ce sudo: unable to resolve host ip-172-16-75-233 Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: aufs-tools cgroupfs-mount libltdl7 Suggested packages: mountall The following NEW packages will be installed: aufs-tools cgroupfs-mount docker-ce libltdl7 0 upgraded, 4 newly installed, 0 to remove and 48 not upgraded. Need to get 30.3 MB of archives. After this operation, 149 MB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 aufs-tools amd64 1:3.2+20130722-1.1ubuntu1 [92.9 kB] Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4,970 B] Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 libltdl7 amd64 2.4.6-0.1 [38.3 kB] Get:4 https://download.docker.com/linux/ubuntu xenial/stable amd64 docker-ce amd64 17.12.1~ce-0~ubuntu [30.2 MB] Fetched 30.3 MB in 2s (11.2 MB/s) Selecting previously unselected package aufs-tools. (Reading database ... 51122 files and directories currently installed.) Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1ubuntu1_amd64.deb ... Unpacking aufs-tools (1:3.2+20130722-1.1ubuntu1) ... Selecting previously unselected package cgroupfs-mount. Preparing to unpack .../cgroupfs-mount_1.2_all.deb ... Unpacking cgroupfs-mount (1.2) ... Selecting previously unselected package libltdl7:amd64. Preparing to unpack .../libltdl7_2.4.6-0.1_amd64.deb ... Unpacking libltdl7:amd64 (2.4.6-0.1) ... Selecting previously unselected package docker-ce. Preparing to unpack .../docker-ce_17.12.1~ce-0~ubuntu_amd64.deb ... Unpacking docker-ce (17.12.1~ce-0~ubuntu) ... Processing triggers for libc-bin (2.23-0ubuntu10) ... Processing triggers for man-db (2.7.5-1) ... Processing triggers for ureadahead (0.100.0-19) ... Processing triggers for systemd (229-4ubuntu21) ... Setting up aufs-tools (1:3.2+20130722-1.1ubuntu1) ... Setting up cgroupfs-mount (1.2) ... Setting up libltdl7:amd64 (2.4.6-0.1) ... Setting up docker-ce (17.12.1~ce-0~ubuntu) ... Processing triggers for libc-bin (2.23-0ubuntu10) ... Processing triggers for systemd (229-4ubuntu21) ... Processing triggers for ureadahead (0.100.0-19) ... ubuntu@ip-172-16-75-233:~$ sudo groupadd docker sudo: unable to resolve host ip-172-16-75-233 groupadd: group 'docker' already exists ubuntu@ip-172-16-75-233:~$ sudo usermod -aG docker $USER sudo: unable to resolve host ip-172-16-75-233 ubuntu@ip-172-16-75-233:~$ sudo nano /etc/hosts sudo: unable to resolve host ip-172-16-75-233 ubuntu@ip-172-16-75-233:~$ screen -S kafka -- sudo docker run -it -v /srv/kafka:/tmp --rm --net=host -e ADV_HOST=172.16.75.233 landoop/fast-data-dev [detached from 13762.kafka] ubuntu@ip-172-16-75-233:~$ ^C ubuntu@ip-172-16-75-233:~$ ^A? ?: command not found ubuntu@ip-172-16-75-233:~$ ? ?: command not found ubuntu@ip-172-16-75-233:~$ ubuntu@ip-172-16-75-233:~$ logout
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Session stopped
- Press
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-1049-aws x86_64)
- Documentation: https://help.ubuntu.com
- Management: https://landscape.canonical.com
- Support: https://ubuntu.com/advantage
Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud
51 packages can be updated. 17 updates are security updates.
Last login: Wed Feb 28 14:21:54 2018 from 10.8.16.32 ubuntu@ip-172-16-75-233:~$ ^C ubuntu@ip-172-16-75-233:~$ c c: command not found ubuntu@ip-172-16-75-233:~$ screen -v Screen version 4.03.01 (GNU) 28-Jun-15 ubuntu@ip-172-16-75-233:~$ which screen /usr/bin/screen ubuntu@ip-172-16-75-233:~$ screen -r 2018-02-28 18:11:12,481 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:15,291 INFO exited: schema-registry (exit status 1; not expected) 2018-02-28 18:11:15,298 INFO spawned: 'schema-registry' with pid 22032 2018-02-28 18:11:16,303 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:16,673 INFO exited: broker (exit status 1; not expected) 2018-02-28 18:11:16,757 INFO spawned: 'broker' with pid 22056 2018-02-28 18:11:17,762 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:20,143 INFO exited: rest-proxy (exit status 1; not expected) 2018-02-28 18:11:20,228 INFO spawned: 'rest-proxy' with pid 22076 2018-02-28 18:11:21,233 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:22,017 INFO exited: broker (exit status 1; not expected) 2018-02-28 18:11:22,039 INFO spawned: 'broker' with pid 22100 2018-02-28 18:11:23,044 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:25,631 INFO exited: schema-registry (exit status 1; not expected) 2018-02-28 18:11:25,711 INFO spawned: 'schema-registry' with pid 22144 2018-02-28 18:11:25,822 INFO exited: connect-distributed (exit status 1; not expected) 2018-02-28 18:11:25,912 INFO spawned: 'connect-distributed' with pid 22146 2018-02-28 18:11:26,716 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:26,967 INFO success: connect-distributed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:27,290 INFO exited: broker (exit status 1; not expected) 2018-02-28 18:11:27,321 INFO spawned: 'broker' with pid 22170 2018-02-28 18:11:28,326 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:32,567 INFO exited: broker (exit status 1; not expected) 2018-02-28 18:11:32,641 INFO spawned: 'broker' with pid 22194 2018-02-28 18:11:33,658 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:35,469 INFO exited: rest-proxy (exit status 1; not expected) 2018-02-28 18:11:35,520 INFO spawned: 'rest-proxy' with pid 22214 2018-02-28 18:11:35,966 INFO exited: schema-registry (exit status 1; not expected) 2018-02-28 18:11:36,023 INFO spawned: 'schema-registry' with pid 22234 2018-02-28 18:11:36,524 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:37,027 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:37,908 INFO exited: broker (exit status 1; not expected) 2018-02-28 18:11:37,934 INFO spawned: 'broker' with pid 22258 2018-02-28 18:11:38,939 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:41,330 INFO exited: connect-distributed (exit status 1; not expected) 2018-02-28 18:11:41,355 INFO spawned: 'connect-distributed' with pid 22284 2018-02-28 18:11:42,360 INFO success: connect-distributed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:43,182 INFO exited: broker (exit status 1; not expected) 2018-02-28 18:11:43,216 INFO spawned: 'broker' with pid 22308 2018-02-28 18:11:44,221 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:46,274 INFO exited: schema-registry (exit status 1; not expected) 2018-02-28 18:11:46,336 INFO spawned: 'schema-registry' with pid 22328 2018-02-28 18:11:47,391 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2018-02-28 18:11:48,477 INFO exited: broker (exit status 1; not expected) 2018-02-28 18:11:48,550 INFO spawned: 'broker' with pid 22352
I am SO new at this, and a DBA, not an engineer, so I am very much so confused as to how ANY of this works at all.
Given that you are NEW to this, join our Slack channel at http://launchpass.com/landoop-community so we can assist you setting things up
having the same issue, docker on arch linux
docker --version
Docker version 18.06.1-ce, build e68fc7a215
running your docker command docker run -it --name kafka-dev-landoop --net=host landoop/fast-data-dev
turns out that I can sucsesfully start the stack by leaving out the --net=host parameter and adding the port mapping manually as -p 3030:3030.
My full running command is now : docker run -it --name kafka-dev-landoop -p 3030:3030 -p 2181:2181 -p 3031:3031 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 landoop/fast-data-dev though it may contain "too much port mappings"...
https://github.com/Landoop/fast-data-dev/issues/61#issuecomment-449875355 I need to do this when using Docker for Mac as well.
