elastalert
elastalert copied to clipboard
bitsensor
Build Error <mock requires Python '>=3.6' but the running Python is 2.7.15>
Hi,
I'm having trouble building the elastalert docker container on Ubuntu 18.04 using the instructions at https://github.com/bitsensor/elastalert. Initial builds failed with same error message mentioned in https://github.com/bitsensor/elastalert/issues/147 but even after modifying the dockerfile to have "FROM alpine:3.8 as py-ea" instead of "FROM alpine:latest as py-ea" as recommended, builds are still failing but with a different error:
mock requires Python '>=3.6' but the running Python is 2.7.15 You are using pip version 10.0.1, however version 20.0.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. The command '/bin/sh -c sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && python setup.py install && pip install -r requirements.txt' returned a non-zero code: 1 Makefile:6: recipe for target 'build' failed make: *** [build] Error 1
Any help or insight on how to resolve this will be greatly appreciated.
Having the same issue myself on Ubuntu 18.04 after making the change "FROM alpine:latest as py-ea -> FROM alpine:3.8 as py-ea"
Did you have any luck resolving @bcisse ?
I was able to build version v0.2.1 of the elastalert docker image by modifying the dockerfile to install python3 as follows:
*****Dockerfile ***** FROM alpine:latest as py-ea ARG ELASTALERT_VERSION=v0.2.1 ... CHANGE: RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget && \
TO:
RUN apk add --update --no-cache ca-certificates openssl-dev openssl python3-dev python3 py3-pip py3-yaml libffi-dev gcc musl-dev wget &&
...
CHANGE:
COPY --from=py-ea /usr/lib/python2.7/site-packages /usr/lib/python2.7/site-packages
TO: COPY --from=py-ea /usr/lib/python3.8/site-packages /usr/lib/python3.8/site-packages
RUN sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py &&
python3 setup.py install &&
pip3 install -r requirements.txt
When I try to start a container using the image, it fails however after a few seconds. The elastalert indices are created in elasticsearch but container fails shortly after with following error:
16:37:18.170Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json. 16:37:18.173Z INFO elastalert-server: Config: Proceeding to look for normal config file. 16:37:18.174Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config. 16:37:18.185Z INFO elastalert-server: Router: Listening for GET request on /. 16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status. 16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action. 16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status/errors. 16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /rules. 16:37:18.188Z INFO elastalert-server: Router: Listening for GET request on /rules/:id. 16:37:18.189Z INFO elastalert-server: Router: Listening for POST request on /rules/:id. 16:37:18.189Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id. 16:37:18.189Z INFO elastalert-server: Router: Listening for GET request on /templates. 16:37:18.189Z INFO elastalert-server: Router: Listening for GET request on /templates/:id. 16:37:18.189Z INFO elastalert-server: Router: Listening for POST request on /templates/:id. 16:37:18.193Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id. 16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /test. 16:37:18.194Z INFO elastalert-server: Router: Listening for GET request on /config. 16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /config. 16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /download. 16:37:18.194Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type. 16:37:18.195Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index. 16:37:18.195Z INFO elastalert-server: Router: Listening for POST request on /search/:index. 16:37:18.199Z INFO elastalert-server: ProcessController: Starting ElastAlert 16:37:18.199Z INFO elastalert-server: ProcessController: Creating index 16:37:21.448Z INFO elastalert-server: ProcessController: Elastic Version:7 Mapping used for string:{'type': 'keyword'}
16:37:21.449Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/create_index.py", line 275, in
16:37:21.449Z ERROR elastalert-server: ProcessController: Index create exited with code 1 16:37:21.449Z WARN elastalert-server: ProcessController: ElastAlert will start but might not be able to save its data! 16:37:21.450Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none] 16:37:21.459Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 36) 16:37:21.461Z INFO elastalert-server: Server: Server listening on port 3030 16:37:21.462Z INFO elastalert-server: Server: Websocket listening on port 3333 16:37:21.462Z INFO elastalert-server: Server: Server started 16:37:22.338Z ERROR elastalert-server: ProcessController: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
16:37:22.338Z ERROR elastalert-server:
ProcessController: "main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
16:37:22.338Z ERROR elastalert-server: ProcessController: client.start() File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start
16:37:22.338Z ERROR elastalert-server: ProcessController: self.run_all_rules() File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules
16:37:22.339Z ERROR elastalert-server: ProcessController: self.send_pending_alerts() File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts
16:37:22.339Z ERROR elastalert-server: ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit) File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts
16:37:22.340Z ERROR elastalert-server: ProcessController: size=1000) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
16:37:22.340Z ERROR elastalert-server: ProcessController: return func(*args, params=params, **kwargs) TypeError: search() got an unexpected keyword argument 'doc_type'
16:37:22.384Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1 16:37:22.384Z INFO elastalert-server: Server: Stopping server 16:37:22.384Z INFO elastalert-server: ProcessController: ElastAlert is not running 16:37:22.384Z INFO elastalert-server: Server: Server stopped. Bye!
In my testing I saw this error a few times. fwiw, I have been using https://github.com/bitsensor/elastalert/pull/157 and it works great on python3. I would also prehaps prefer a base ubuntu. See the commits for a possible solution
For those who want to stick to python 2.7 the below works
FROM alpine:3.8 as py-ea
ARG ELASTALERT_VERSION=v0.1.39
ENV ELASTALERT_VERSION=${ELASTALERT_VERSION}
# URL from which to download Elastalert.
ARG ELASTALERT_URL=https://github.com/Yelp/elastalert/archive/$ELASTALERT_VERSION.zip
ENV ELASTALERT_URL=${ELASTALERT_URL}
# Elastalert home directory full path.
ENV ELASTALERT_HOME /opt/elastalert
WORKDIR /opt
RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget && \
# Download and unpack Elastalert.
wget -O elastalert.zip "${ELASTALERT_URL}" && \
unzip elastalert.zip && \
rm elastalert.zip && \
mv e* "${ELASTALERT_HOME}"
WORKDIR "${ELASTALERT_HOME}"
# Install Elastalert.
# see: https://github.com/Yelp/elastalert/issues/1654
RUN sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && sed -i 's/mock>=2.0.0/mock==3.0.5/g' setup.py && \
python setup.py install && \
pip install apscheduler>=3.3.0 aws-requests-auth>=0.3.0 blist>=1.3.6 boto3>=1.4.4 cffi>=1.11.5 configparser>=3.5.0 croniter>=0.3.16 elasticsearch>=7.0.0 envparse>=0.2.0 exotel>=0.1.3 "jira>=1.0.10,<1.0.15" jsonschema>=3.0.2 mock==3.0.5 prison>=0.1.2 py-zabbix==1.1.3 PyStaticConfiguration>=0.10.3 "python-dateutil>=2.6.0,<2.7.0" python-magic>=0.4.15 PyYAML>=5.1 requests>=2.0.0 stomp.py==4.1.23 texttable>=0.8.8 thehive4py>=1.4.4 twilio==6.0.0
FROM node:alpine
LABEL maintainer="BitSensor <[email protected]>"
# Set timezone for this container
ENV TZ Etc/UTC
RUN apk add --update --no-cache curl tzdata python2 make libmagic
COPY --from=py-ea /usr/lib/python2.7/site-packages /usr/lib/python2.7/site-packages
COPY --from=py-ea /opt/elastalert /opt/elastalert
COPY --from=py-ea /usr/bin/elastalert* /usr/bin/
WORKDIR /opt/elastalert-server
COPY . /opt/elastalert-server
RUN npm install --production --quiet
COPY config/elastalert.yaml /opt/elastalert/config.yaml
COPY config/elastalert-test.yaml /opt/elastalert/config-test.yaml
COPY config/config.json config/config.json
COPY rule_templates/ /opt/elastalert/rule_templates
COPY elastalert_modules/ /opt/elastalert/elastalert_modules
EXPOSE 3030
ENTRYPOINT ["npm", "start"]
@jibsonline Using your config I get this error. Am I missing something ?
Installed /usr/lib/python2.7/site-packages/croniter-0.3.37-py2.7.egg
Searching for configparser>=3.5.0
Reading https://pypi.org/simple/configparser/
Downloading https://files.pythonhosted.org/packages/3f/e7/9518720c56396179f8c63d7b5924c8463ed423828e54329be7f8cde5c364/configparser-5.0.1.tar.gz#sha256=005c3b102c96f4be9b8f40dafbd4997db003d07d1caa19f37808be8031475f2a
Best match: configparser 5.0.1
Processing configparser-5.0.1.tar.gz
Writing /tmp/easy_install-DYi9wW/configparser-5.0.1/setup.cfg
Running configparser-5.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-DYi9wW/configparser-5.0.1/egg-dist-tmp-rLCXdh
error: Setup script exited with error: 'egg_base' must be a directory name (got src)
The command '/bin/sh -c sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && sed -i 's/mock>=2.0.0/mock==3.0.5/g' setup.py && python setup.py install && pip install apscheduler>=3.3.0 aws-requests-auth>=0.3.0 blist>=1.3.6 boto3>=1.4.4 cffi>=1.11.5 configparser>=3.5.0 croniter>=0.3.16 elasticsearch>=7.0.0 envparse>=0.2.0 exotel>=0.1.3 "jira>=1.0.10,<1.0.15" jsonschema>=3.0.2 mock==3.0.5 prison>=0.1.2 py-zabbix==1.1.3 PyStaticConfiguration>=0.10.3 "python-dateutil>=2.6.0,<2.7.0" python-magic>=0.4.15 PyYAML>=5.1 requests>=2.0.0 stomp.py==4.1.23 texttable>=0.8.8 thehive4py>=1.4.4 twilio==6.0.0' returned a non-zero code: 1
Makefile:6: recipe for target 'build' failed
make: *** [build] Error 1
