codeship
Fixed that distro download stops on sporadic 404 error
(ticket is https://support.cloudbees.com/hc/en-us/requests/198007)
There is an intermittent issue with Elasticsearch installations (most likely started after 9c905d5f838d697fbe98769bc7b03f2277391f1a), when elasticsearh.sh sometimes gets 404 from ES CDN, while the next time it downloads it w/o any issues.
The ideal would be to use --retry-on-http-error:
wget --retry-on-http-error=404 --tries=10 --waitretry=3
--continue --output-document "${CACHED_DOWNLOAD}" "${ELASTICSEARCH_DL_URL}"
However, this doesn't help on practice, wget was failing with 404 all 10 attempts:
--2021-02-22 16:04:02-- https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
Retrying.
--2021-02-22 16:04:03-- (try: 2) https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-linux-x86_64.tar.gz
Reusing existing connection to artifacts.elastic.co:443.
HTTP request sent, awaiting response... 404 Not Found
Retrying.
...
--2021-02-22 16:04:26-- (try:10) https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-linux-x86_64.tar.gz
Reusing existing connection to artifacts.elastic.co:443.
HTTP request sent, awaiting response... 404 Not Found
Giving up.
My theory is that wget in this mode does not try to reconnect, uses the same IP to fetch file from, (and yeah the root cause is somewhere in codeship network DNS, artifacts.elastic.co domain I bet is resolving onto old IP in one of your routers)
So switched here to shell loop, and that helped:
Making attempt # 1
--2021-02-22 16:16:59-- https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-02-22 16:16:59 ERROR 404: Not Found.
Making attempt # 2
--2021-02-22 16:17:00-- https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 318776692 (304M) [application/x-gzip]
Saving to: ‘/home/rof/cache/elasticsearch-7.10.0.tar.gz’
/home/rof/cache/ela 100%[===================>] 304.01M 105MB/s in 2.9s
2021-02-22 16:17:03 (105 MB/s) - ‘/home/rof/cache/elasticsearch-7.10.0.tar.gz’ saved [318776692/318776692]
