airflow-pipeline
airflow-pipeline copied to clipboard
jessie-backports sometimes gives hash sum mismatch
Depending on the internet connection, sometimes fetching from jessie-backports
can give hashsum mismatch (logs at the end). Guide suggests that this could be due to ISP caching: https://www.reddit.com/r/debian/comments/64xk33/jessiebackports_is_giving_a_hash_sum_mismatch/.
Can consider using the newer Debian stretch
that doesn't require the above additional package, see example: https://github.com/datagovsg/airflow-pipeline/tree/debian-upgrade.
Error Log:
> docker build . -t datagovsg/airflow-pipeline --build-arg SPARK_VERSION=2.1.2 --build-arg HADOOP_VERSION=2.6.5 --build-arg SPARK_PY4J=python/lib/py4j-0.10.4-src.zip
...
Step 3/59 : RUN set -ex && (echo 'deb http://deb.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/backports.list) && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y --force-yes vim-tiny libsasl2-dev libffi-dev gosu krb5-user && rm -rf /var/lib/apt/lists/* && pip install --no-cache-dir "apache-airflow[devel_hadoop, crypto]==1.9.0" psycopg2 && pip install --no-cache-dir sqlalchemy==1.1.17
---> Running in 2b9b735eccda
+ echo deb http://deb.debian.org/debian jessie-backports main
+ apt-get update
Get:1 http://security.debian.org jessie/updates InRelease [94.4 kB]
Ign http://deb.debian.org jessie InRelease
Get:2 http://deb.debian.org jessie-updates InRelease [145 kB]
Get:3 http://deb.debian.org jessie-backports InRelease [166 kB]
Get:4 http://deb.debian.org jessie Release.gpg [2434 B]
Get:5 http://deb.debian.org jessie Release [148 kB]
Get:6 http://security.debian.org jessie/updates/main amd64 Packages [622 kB]
Get:7 http://deb.debian.org jessie-updates/main amd64 Packages [23.0 kB]
Get:8 http://deb.debian.org jessie-backports/main amd64 Packages [1170 kB]
Get:9 http://deb.debian.org jessie/main amd64 Packages [9064 kB]
Fetched 11.4 MB in 6s (1663 kB/s)
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
The command '/bin/sh -c set -ex && (echo 'deb http://deb.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/backports.list) && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y --force-yes vim-tiny libsasl2-dev libffi-dev gosu krb5-user && rm -rf /var/lib/apt/lists/* && pip install --no-cache-dir "apache-airflow[devel_hadoop, crypto]==1.9.0" psycopg2 && pip install --no-cache-dir sqlalchemy==1.1.17' returned a non-zero code: 100