hostsblock hosts.block not being updated

hostsblock had not updated hosts.block since I installed version 0.999.8.1 8 days ago. The only blocklist I’m using is StevenBlack (last updated 5 days ago). The timer is enabled, the computer was running every day. I have not changed hostsblock.conf.

First, I edited the timer to run every 5 minutes. hosts.block remained unchanged.

I then tried to manually update:

$ sudo hostsblock
Checking blocklists and/or redirectlists for updates...
No new changes. DONE.
[HERE IS MY PS1]   Changes found to https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.

(Note: The last line was output after my PS1 was already displayed.)

After this, hosts.block was still not updated, so I used the -u flag:

$ sudo hostsblock -u
Checking blocklists and/or redirectlists for updates...
Changes found among blocklists and/or redirectlists. Extracting to /var/lib/hostsblock/hosts.block.new...
  Extracting blocklists...
  Appending blacklist entries...
/var/lib/hostsblock/hosts.block successfully compiled. DONE.
[HERE IS MY PS1]

This worked.

Why did hostsblock require the -u flag? Might this also be the reason why the timed execution fails to update hosts.block?

Apr 16 '19 10:04 Markus00000

Try deleting all your files in /var/lib/hostsblock/cache and then running hostsblock -u -d and post what it outputs here. It looks like something might be up with how hostsblock manages downloads. (The "Changes found" dialog line shouldn't come up after the script completes, which means something might be up with how hostsblock tracks parallel downloads.

Apr 16 '19 12:04 gaenserich

I think hostsblock initially complained about files not being found in the cache directory after I installed the hostsblock package from the AUR. No cache directory was created, so I created it manually:

$ ls -ld /var/lib/hostsblock/cache
drwx------ 2 hostsblock hostsblock 70 2019-04-16 14:40 /var/lib/hostsblock/cache/

After that, the initial update worked. This might be a separate issue that the cache directory should either be part of the package or created by hostsblock if it does not exist.

On to your comment. After emptying the cache directory, I ran hostsblock -u -d. Note that my PS1 is, once more, in the middle of the output (just search for “PS1”).

% sudo hostsblock -u -d
++ id -un
+ '[' hostsblock '!=' hostsblock ']'
+ '[' ']'
++ id -un
+ '[' hostsblock = hostsblock ']'
+ '[' -f /var/lib/hostsblock/hostsblock.conf ']'
+ . /var/lib/hostsblock/hostsblock.conf
+ TMPDIR=/tmp/hostsblock
+ mkdir -p -v -- /tmp/hostsblock
+ '[' 1 -eq 1 ']'
+ touch /tmp/hostsblock/changed
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v chmod
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v cksum
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v cp
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v curl
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v cut
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v file
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v find
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v grep
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v id
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v mkdir
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v mv
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v rm
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v sed
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v sort
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v tee
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v touch
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v tr
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v wc
+ for _depends in chmod cksum cp curl cut file find grep id mkdir mv rm sed sort tee touch tr wc xargs
+ command -v xargs
+ _return=0
+ '[' '' ']'
+ _job
+ command -v unzip
+ _unzip_available=1
+ command -v 7zr
+ _un7zip_available=1
+ '[' 1 -eq 1 ']'
+ _notify 1 'Checking blocklists and/or redirectlists for updates...'
+ '[' 3 -ge 1 ']'
+ printf '%s\n' 'Checking blocklists and/or redirectlists for updates...'
Checking blocklists and/or redirectlists for updates...
+ '[' -d /tmp/hostsblock/downloads/ ']'
+ mkdir -p -v -- /tmp/hostsblock/downloads/
mkdir: created directory '/tmp/hostsblock/downloads/'
+ sed 's/#.*//g' /var/lib/hostsblock/block.urls
+ grep '[[:alnum:]]'
+ read _url
+ '[' 8 -gt 0 ']'
++ find /tmp/hostsblock/downloads -type f
++ wc -l
+ '[' 0 -ge 8 ']'
+ read _url
+ _job_download_list https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
+ wait
++ printf %s raw.githubusercontent.com/StevenBlack/hosts/master/hosts
++ tr '/%&+?=' .
++ find /tmp/hostsblock/downloads -type f
++ wc -l
+ _outfile=/var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts
+ '[' '!' -d /tmp/hostsblock/downloads ']'
+ touch /tmp/hostsblock/downloads/raw.githubusercontent.com.StevenBlack.hosts.master.hosts
+ '[' 0 -gt 0 ']'
+ rm -rf -v -- /tmp/hostsblock/downloads
+ '[' -f /var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts ']'
+ curl -v --compressed -L --connect-timeout 60 --retry 0 -z /var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -o /var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts
removed '/tmp/hostsblock/downloads/raw.githubusercontent.com.StevenBlack.hosts.master.hosts'
removed directory '/tmp/hostsblock/downloads'
+ '[' -f /tmp/hostsblock/changed ']'
+ _notify 1 'Changes found among blocklists and/or redirectlists. Extracting to /var/lib/hostsblock/hosts.block.new...'
+ '[' 3 -ge 1 ']'
+ printf '%s\n' 'Changes found among blocklists and/or redirectlists. Extracting to /var/lib/hostsblock/hosts.block.new...'
Changes found among blocklists and/or redirectlists. Extracting to /var/lib/hostsblock/hosts.block.new...
+ '[' 0 '!=' 0 ']'
+ '[' /var/lib/hostsblock/block.urls ']'
+ _notify 1 '  Extracting blocklists...'
+ '[' 3 -ge 1 ']'
+ printf '%s\n' '  Extracting blocklists...'
  Extracting blocklists...
+ _job_extract_from_cachefiles /var/lib/hostsblock/block.urls
+ sed 's/#.*//g' /var/lib/hostsblock/block.urls
+ read _url
+ grep -ve ' localhost$' -ve ' localhost\.localdomain$' -ve ' broadcasthost$' -ve '.* .* .*'
+ grep -Fvf /var/lib/hostsblock/white.list
+ sort -u
Warning: Illegal date format for -z, --time-cond (and not a file name). 
Warning: Disabling time condition. See curl_getdate(3) for valid date syntax.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0+ grep '[[:alnum:]]'
++ printf %s raw.githubusercontent.com/StevenBlack/hosts/master/hosts
++ tr '/%&+?=' .
+ _cachefile=/var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts
++ file -bi /var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts
+ _cachefile_type='cannot open `/var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts'\'' (No such file or directory)'
+ printf %s 'cannot open `/var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts'\'' (No such file or directory)'
+ grep -Fq application/zip
+ printf %s 'cannot open `/var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts'\'' (No such file or directory)'
+ grep -Fq application/x-7z-compressed
+ grep -qE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}[[:space:]]' /var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts
grep: /var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts: No such file or directory
+ _sanitize_raw
/usr/lib/hostsblock.sh: line 60: /var/lib/hostsblock/cache/raw.githubusercontent.com.StevenBlack.hosts.master.hosts: No such file or directory
+ read _url
+ '[' 1 -ne 0 ']'
+ _notify 0 'FAILED TO COMPILE BLOCK/REDIRECT ENTRIES FROM URLS IN /var/lib/hostsblock/block.urls INTO /var/lib/hostsblock/hosts.block. EXITING...'
+ '[' 3 -ge 0 ']'
+ printf '%s\n' 'FAILED TO COMPILE BLOCK/REDIRECT ENTRIES FROM URLS IN /var/lib/hostsblock/block.urls INTO /var/lib/hostsblock/hosts.block. EXITING...'
FAILED TO COMPILE BLOCK/REDIRECT ENTRIES FROM URLS IN /var/lib/hostsblock/block.urls INTO /var/lib/hostsblock/hosts.block. EXITING...
+ exit 2
[MY PS1 HERE] *   Trying 151.101.12.133...
* TCP_NODELAY set
* Connected to raw.githubusercontent.com (151.101.12.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [112 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3182 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=www.github.com
*  start date: Mar 23 00:00:00 2017 GMT
*  expire date: May 13 12:00:00 2020 GMT
*  subjectAltName: host "raw.githubusercontent.com" matched cert's "*.githubusercontent.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
} [5 bytes data]
> GET /StevenBlack/hosts/master/hosts HTTP/1.1
> Host: raw.githubusercontent.com
> User-Agent: curl/7.64.1
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
{ [5 bytes data]
< HTTP/1.1 200 OK
< Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< ETag: "3a9b5ef4cdabd22b621e76887c7659fc8ee025d2"
< Content-Type: text/plain; charset=utf-8
< Cache-Control: max-age=300
< X-Geo-Block-List:
< X-GitHub-Request-Id: BBD0:5836:9B420:B0698:5CB5CC46
< Content-Encoding: gzip
< Content-Length: 294878
< Accept-Ranges: bytes
< Date: Tue, 16 Apr 2019 12:40:50 GMT
< Via: 1.1 varnish
< Connection: keep-alive
< X-Served-By: cache-fra19168-FRA
< X-Cache: HIT
< X-Cache-Hits: 1
< X-Timer: S1555418451.546959,VS0,VE1
< Vary: Authorization,Accept-Encoding
< Access-Control-Allow-Origin: *
< X-Fastly-Request-ID: 6a18ac0c936e455525cf820c9454476b8496c24b
< Expires: Tue, 16 Apr 2019 12:45:50 GMT
< Source-Age: 264
< 
{ [1895 bytes data]
100  287k  100  287k    0     0  1024k      0 --:--:-- --:--:-- --:--:-- 1024k
* Connection #0 to host raw.githubusercontent.com left intact
* Closing connection 0
++ cksum
+ _new_cksum='2953408015 1149641'
+ '[' '' '!=' '2953408015 1149641' ']'
+ _notify 1 '  Changes found to https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.'
+ '[' 3 -ge 1 ']'
+ printf '%s\n' '  Changes found to https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.'
  Changes found to https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
+ '[' '!' -d /tmp/hostsblock ']'
+ touch /tmp/hostsblock/changed
+ rm -rf -v -- /tmp/hostsblock/downloads/raw.githubusercontent.com.StevenBlack.hosts.master.hosts

Apr 16 '19 12:04 Markus00000

I think I see where the problem is, but it shouldn't be a problem. I'm going to see if I can replicate it on my end, but in the meanwhile, here's a work around:

Work-around

Open /usr/lib/hostsblock.sh with your favorite editor as root or hostsblock.

Go to line 618, which will look like this

        _job_download_list "$_url" &

Here's the context leading up to it if you can't find it:

    # DOWNLOAD BLOCKLISTS AND/OR REDIRECT LISTS
    _notify 1 "Checking blocklists and/or redirectlists for updates..."

    [ -d "$tmpdir"/downloads/ ] && rm -rf $_v -- "$tmpdir"/downloads/
    mkdir -p $_v -- "$tmpdir"/downloads/
    sed "s/#.*//g" $blocklists $redirectlists | grep "[[:alnum:]]" | \
      while read _url; do
        if [ $max_simultaneous_downloads -gt 0 ]; then
            while [ $(find "$tmpdir"/downloads -type f | wc -l) -ge $max_simultaneous_downloads ]; do
                sleep 0.1
            done
        fi
        _job_download_list "$_url" &

Remove the "&" at the end of the line, so it looks like this:

        _job_download_list "$_url"

This removes your ability to download more than one file in parallel, but since you download only one, this will make absolutely no difference.

P.S. Thanks so much for testing this! We're getting closer to a 1.0 release because of you!

The issue (a note to myself or anyone interested):

The loop I used to download blocklists in parallel uses a temporary directory to keep track of how many downloads are in progress at a particular moment. The function _job_download_list, which handles the downloading of individual files, will create a temporary file in this temporary directory before it downloads and then will delete the file once it is done. The loop which invokes _job_download_list for each to-be-downloaded file checks in on this directory and counts the number of files therein to check to see if can start another process or if it should wait. Once the loop has started all the _job_download_list subprocesses for all the to-be-downloaded files, it ends and then we hit a wait, which should wait for all subprocesses to complete before continuing. However, I noticed that this wait did not seem to wait for all download processes to stop before exiting, so I added another loop after it which checks the temporary directory for the temporary files created by _job_download_list.

However, if there are not a lot of to-be-downloaded blocklists, both loops seem to exit before the first _job_download_list creates its temporary file. Possible ways to fix the issue of the top of my head:

Create a dummy temporary file before the first loop starts that the first _job_download_list deletes after creating its own temporary file. (sounds like the best option)
Strip out parallel downloading (boo!)
?

Apr 16 '19 13:04 gaenserich

hostsblock hostsblock copied to clipboard

hosts.block not being updated

Work-around

The issue (a note to myself or anyone interested):

hostsblock
hostsblock copied to clipboard