Ultimate.Hosts.Blacklist
Ultimate.Hosts.Blacklist copied to clipboard
Ultimate-Hosts-Blacklist
[FEATURE] RPZ Format
Harry @alsyundawy see this issue. We all left some good ideas there how to do it yourself,so try it,see how it goes 👍 Meanwhile ping @funilrys for assist 👍
There have been other request for this in the past latest by @Somebodyisnobody wrote a request for wildcarding the records where possible, but not many raised there hands to maintain hose lists :smirk: In addition to this, I do maintain wildcard lists in the My Privacy DNS and they are all properly hosted over DNS, See bind script example below. While UHB contains a hole lot more records, as they are collected from various lists, then Mypdns is a handcrafted user contributed lists where all records are manually confirmed. Just never forget your private whitelists when using mypdns :smiley:
If you follow @dnmTX link then you get all these lines vs 2 lines and the code is NOT RPZ COMPLIANT...
- zone-name is missing from all records.
- SOA record is missing. Which WILL cause stalled cache
https://github.com/AdguardTeam/cname-trackers/blob/4e8f42456cb616adfc1f4a17faed1a4f0a4dc2b8/combined_disguised_trackers_rpz.txt#L679-L1027
The two right lines for actonservice.com would be looking like this + SOA line (en contra 348 lines).
tracking.mypdns.cloud. 86400 IN SOA ns1.mypdns.org. need.to.know.only. 2021042867 300 60 604800 3600
actonservice.com.tracking.mypdns.cloud CNAME .
*.actonservice.com.tracking.mypdns.cloud CNAME .
Bind setup example of https://github.com/DNS-RPZ/dns-rpz-integration (Outdated use example below)
// RPZ
response-policy { zone "adult.mypdns.cloud";
zone "adware.mypdns.cloud";
zone "coinblocker.mypdns.cloud";
zone "gambling.mypdns.cloud";
zone "malicious.mypdns.cloud";
zone "phishing.mypdns.cloud";
zone "pirated.mypdns.cloud";
zone "redirector.mypdns.cloud";
zone "rpz.mypdns.cloud";
zone "safesearch.mypdns.cloud";
zone "spam.mypdns.cloud";
zone "spyware.mypdns.cloud";
zone "tracking.mypdns.cloud";
zone "typosquatting.mypdns.cloud";
zone "whitelist.mypdns.cloud";
};
zone "adult.mypdns.cloud" IN {
type slave;
file "slave/adult.mypdns.cloud";
masters port 5303 {188.176.49.219;};
};
zone "adware.mypdns.cloud" {type slave; file "slave/adware.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "coinblocker.mypdns.cloud" {type slave; file "slave/coinblocker.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "gambling.mypdns.cloud" {type slave; file "slave/gambling.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "malicious.mypdns.cloud" {type slave; file "slave/malicious.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "phishing.mypdns.cloud" {type slave; file "slave/phishing.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "pirated.mypdns.cloud" {type slave; file "slave/pirated.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "redirector.mypdns.cloud" {type slave; file "slave/redirector.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "rpz.mypdns.cloud" {type slave; file "slave/rpz.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "safesearch.mypdns.cloud" {type slave; file "slave/safesearch.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "spam.mypdns.cloud" {type slave; file "slave/spam.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "spyware.mypdns.cloud" {type slave; file "slave/spyware.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "tracking.mypdns.cloud" {type slave; file "slave/tracking.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "typosquatting.mypdns.cloud" {type slave; file "slave/typosquatting.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "whitelist.mypdns.cloud" {type slave; file "slave/whitelist.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
logging{
channel normal-log{
// alternatively use default_syslog above to log
// everything apart from RPZ info to syslog and omit
// the file statement below
file "/var/log/named/named.log" versions 3 size 1m;
severity info;
};
channel named-rpz {
// change path as appropriate
file "/var/log/named/rpz.log" versions 3 size 250k;
severity error;
};
channel rpzlog {
file "/var/log/named/rpz.log" versions unlimited size 100m;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
category rpz {
rpzlog;
};
category rpz{
named-rpz;
};
// everything else
category default{
normal-log;
};
@spirillen I wrote the patch to generate the cname-trackers list in RPZ format.
The zone-name is not missing from the records. The records do not have a trailing . which, by DNS convention, makes them part of the containing zone.
I omitted the SOA (and NS) record on purpose. The response policy zones live in the namespace of the administrative domain setting the policy. They will be different for every user of the data. To make the zone "RPZ compliant", all you need to do is prepend correct SOA and NS records for your administrative domain. My patch documented this requirement in the README.md file of the cname-trackers repo.
Distributing the data in "RPZ format" without SOA/NS records is the most flexible solution. Different administrative domains will feed the data to their DNS servers differently. The simplest setups only need to cat the data with a fixed preamble. More complicated setups can generate the records they need as part of their pipelines.
For what it's worth, this is (part of) how I integrate the cname-trackers list in my DNS:
# This script runs from cron and updates the zone if the source files change
next_serial() {
d=$(date +%Y%m%d)
awk "/; Serial/{
if (\$0 ~ /${d}/) \$1++; else \$1=${d}00; print \$1
}" ${1} 2>/dev/null || echo "${d}00"
}
write_header() {
serial=$1
url=$2
srchash=$3
echo "\$TTL 300"
echo ""
echo "; Generated from ${url}"
echo "; srchash: ${srchash}"
echo ""
echo "@ SOA ${mname}. ${rname}. ("
echo " ${serial}"
echo " 43200"
echo " 900"
echo " 604800"
echo " 3600 )"
echo ""
echo "@ NS ${mname}."
echo ""
}
update_from_plain_rpz() {
url=$1
outfile=$2
srchash=$(fetch_source ${url} ${outfile}) || return $?
serial=$(next_serial ${outfile})
write_header ${serial} ${url} ${srchash} > ${tmpdir}/blacklist.rpz
cat ${tmpdir}/blacklist.txt >> ${tmpdir}/blacklist.rpz
cat ${tmpdir}/blacklist.rpz > ${outfile}
echo ${serial}
}
The simplest setups only need to cat the data with a fixed preamble.
No the simplest is to run on DNS port :53 where DNS servers are exchanging data based on a SOA record. CAT / wget / curl or what ever are for people who doesn't understand DNS and it's sub process RPZ and think you are turning a perfect DNS axfr funktion into a plain a hosts file that requirs to download all data to update just one record (kb vs MB).
The zone-name is not missing from the records.
Yes it is, while you are speaking of the a particular DNS program Bind9 then the real zone data exchange are performed in ABSOLUTE records. Yes you can cheat that in Bind9 by adding $zone-name in the top of a local zone record to have it applied to ALL zone records within that file. However that does NOT correct the failed zone record linked above as it is not a fully complaint record. and ONLY supported by bind9+
For what it's worth, this is (part of) how I integrate the cname-trackers list in my DNS:
RPZ + DNS are not to be run from a cron job... is it a dumb hosts file you are operating here?? or is it a DNS zone which holds RPZ records updated in accordance to the SOA serial?
The fact is, your script only partial support Bind9's DNS software. While if you IE. tried to setup one of the RPZ zone files the right way DNS software to DNS software exchange you'll see they are operating properly without any CRON based commend but simply uses the SOA record to determine if a new exchange is needed.
I omitted the ~SOA~ (and NS) record on purpose.
True the NS record isn't needed in RPZ zones, in my cases they are added as a simple workaround to satisfy pdnsutil (Bash tool for PowerDNS)
you can do a simple test on your degraded Bind9 installation a do a drill axfr rpz-zone-name.tld and see how the exchange for looks like!!
Here is a example of coinblocker.srv
drill axfr coinblocker.srv @35.156.219.71 | head
coinblocker.srv. 86400 IN SOA need.to.know.only. hostmaster.deteque.com. 1628812861 28800 60 432000 60
coinblocker.srv. 86400 IN NS localhost.
seed.ac.coinblocker.srv. 86400 IN CNAME .
bittrex.academy.coinblocker.srv. 86400 IN CNAME .
hostingcloud.accountant.coinblocker.srv. 86400 IN CNAME .
www.hostingcloud.accountant.coinblocker.srv. 86400 IN CNAME .
miner.ad.coinblocker.srv. 86400 IN CNAME .
www.miner.ad.coinblocker.srv. 86400 IN CNAME .
aba.ae.coinblocker.srv. 86400 IN CNAME .
www.aba.ae.coinblocker.srv. 86400 IN CNAME .
Notice the FULL zone record in the following syntax: rpz-record.$zone.name TTL RPZ-record-types VALUE
Full RPZ-record-types scheme can be found here
UPDATE: You can of curse also read the documentation for Bind9+RPZ at http://www.zytrax.com/books/dns/ch9/rpz.html for how to actually operate the software and the a RPZ zone.
When I said all of this, then yes, an zone file can be exchanged in a flat-file style, but it have to be a fully valid zone file like https://urlhaus.abuse.ch/downloads/rpz/ or https://sslbl.abuse.ch/blacklist/sslbl.rpz with a fixed update frequents like 300 seconds
@spirillen I'm sorry my example confused you. Those two functions are part of a script that runs from cron which generates RPZ zones for one of my networks. The RPZ zones are served by NSD and distributed to my resolvers (which run unbound) using AXFR.
As for the FQDN discussion: Every DNS server I have encountered which consumes zonefiles in a format like that described in RFC 1035, interprets the absence of an empty terminal label to mean the name belongs to the last start of authority rather than as an FQDN.
Those two functions are part of a script that runs from cron which generates RPZ zones for one of my networks
I know, and as said, I believe it is as wrong as it can be :wink:
IF you are unable to retain the data with AXFR, then it should be possible to "just" download it and the reload the zone. I do this for the PowerDNS like this
sslbl.abuse.ch.sh
#!/usr/bin/env bash
zoneName="sslbl.abuse.ch.rpz.zone"
dir="/var/lib/powerdns/zones.slave.d"
zoneFile="${dir}/${zoneName}"
cd ${dir}
lftp -c "set xfer:clobber on; set xfer:use-temp-file on; set xfer:timeout 15; get https://sslbl.abuse.ch/blacklist/sslbl.rpz -o ${zoneName}"
As for the FQDN discussion: Every DNS server I have encountered which consumes zonefiles in a format like that described in RFC 1035, interprets the absence of an empty terminal label to mean the name belongs to the last start of authority rather than as an FQDN.
That's true :+1: But if you are sticking to relay on the abuse of such rules, you will get a number of errors over time. I can and will not recommand anybody to relay on this versus generating a fully qualified zone file.
Interesting, you are using NSD to serve RPZ ??? hmm will have to re-read the docs before commenting on the syntax for that.
Non the less, I do believe the best would be to generate that flat-file to a fully compliant bind zone syntax with absolute records to avoid any issues.
The RPZ zones are served by NSD and distributed to my resolvers (which run unbound) using AXFR.
Is these public available servers for other to use, in that case I believe you should announce them.
This was some detour with how to import badly formatted records with a cron Versus just have a valid RPZvia AXFR from a primary server.
Any further discussion on this should be in another thread as it is OT to OP.
i have custom domain name to block using rpz but i dont undertand about scripting using bash, sed aor awk, can anybody help me to make like
input domain.txt abcd.com aaskakas.net bsbs.mkns.org *.bsjlwlw.info
output domain.rpz
abcd.com IN CNAME dns-rpz aaskakas.net IN CNAME dns-rpz bsbs.mkns.org IN CNAME dns-rpz *.bsjlwlw.info IN CNAME dns-rpz
You can not use CNAME dns-rpz to anything...
The valid syntax's and codes can be found here: https://mypdns.org/mypdns/support/-/wikis/RPZ-record-types
The short version for how to do this is:
for lines in domain.txt
do
echo -e "$lines\tCNAME . ; UHB"
done
You can then prepend the script with the write_header suggestion from @ppaeps in https://github.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist/issues/623#issuecomment-898090312
But as mentioned earlier, without a list of records that should be handled by wildcard, you actually doesn't have any benefit by converting this into a RPZ, the server load would be the same as if you served it any other way
Alternatively, I have been building on a hosts2RPZ in the past: https://github.com/DNS-RPZ/mypdns-hosts2rpz Can't remember if I had it working, but give it a spin and let me now. Could be we could spend some time on fixing it if it isn't working
You can not use
CNAME dns-rpzto anything...The valid syntax's and codes can be found here: https://mypdns.org/mypdns/support/-/wikis/RPZ-record-types
The short version for how to do this is:
for lines in domain.txt do echo -e "$lines\tCNAME . ; UHB" doneYou can then prepend the script with the
write_headersuggestion from @ppaeps in #623 (comment)But as mentioned earlier, without a list of records that should be handled by wildcard, you actually doesn't have any benefit by converting this into a RPZ, the server load would be the same as if you served it any other way
Alternatively, I have been building on a hosts2RPZ in the past: https://github.com/DNS-RPZ/mypdns-hosts2rpz Can't remember if I had it working, but give it a spin and let me now. Could be we could spend some time on fixing it if it isn't working
found it using python3 script from @iffanux
import os
print('Merge the last database')
inputFile = "input.rpz.tmp"
outputFile = "output.rpz.zone"
# Change your subdomain
my_string = "IN CNAME dns-rpz"
file = open("input.rpz.tmp", "r")
with open(inputFile, 'r') as inFile:
with open(outputFile, 'w+') as outFile:
for line in inFile:
line = line.rstrip("\n")
outFile.write(line + my_string + "\n")
file.close()
# Delete a tmp file
os.remove("input.rpz.tmp")
print('Database merged successfully')
That most have been a non standard program he uses that for.
Because this one will not be working for anything standard RPZ
for line in inFile:
line = line.rstrip("\n")
outFile.write(line + my_string + "\n")
file.close()
I can help you cheat on this one
https://github.com/markster101/Unbound/blob/be23f351b51289e6c995a982c719ee1731c1e962/zone-load#L206
https://archive.mypdns.org/w/recursorserver/#rpz-urlhaus-abuse-ch just replace the DNS server ip with 188.176.49.219 :wink:
You might also run into some issues here, as when they are blocked you won't get the HTTP code 200 and the response could just as well be 301/302/307/308
https://github.com/markster101/Unbound/blob/be23f351b51289e6c995a982c719ee1731c1e962/zone-load#L139
Just a few quick notes as I'm hungry.
These scripts I use with Unbound DNS and nice thing with latest version is that they have fixed the issue with auto uploading rpm with URL like urlhaus.abuse.ch http://urlhaus.abuse.ch/ so I don’t need to include this rpm in my manual load anymore.
Line 139 is is part of ValidateURL function. If the url is blocked the function with exit with return 1. I have been running this code for a long time with great success. Many parts can be re-used for non unbound configuration. Things like zone-control -a register -f rpm.demo will create an empty and proper header rpm file for local use. Other functions that I have coded that can be re-used are conversion and auto/manual loads.
On Sep 11, 2021, at 11:11 AM, spirillen @.***> wrote:
I can help you cheat on this one
https://github.com/markster101/Unbound/blob/be23f351b51289e6c995a982c719ee1731c1e962/zone-load#L206 https://github.com/markster101/Unbound/blob/be23f351b51289e6c995a982c719ee1731c1e962/zone-load#L206 https://archive.mypdns.org/w/recursorserver/#rpz-urlhaus-abuse-ch https://archive.mypdns.org/w/recursorserver/#rpz-urlhaus-abuse-ch just replace the DNS server ip with 188.176.49.219 😉
You might also run into some issues here, as when they are blocked you won't get the HTTP code 200 and the response could just as well be 301/302/307/308
https://github.com/markster101/Unbound/blob/be23f351b51289e6c995a982c719ee1731c1e962/zone-load#L139 https://github.com/markster101/Unbound/blob/be23f351b51289e6c995a982c719ee1731c1e962/zone-load#L139 Just a few quick notes as I'm hungry.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist/issues/623#issuecomment-917440521, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOQWTTITX4IEAKQHNIADTQDUBOETDANCNFSM5CAH4H7A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Mhh, It's maybe time to publish our Bind9 RPZ zones.
Publish? do you mean releasing information for IXFR for the zone(s)?
Well, at the time we (@mitchellkrogza and I) developed and imagined our DNS infrastructure, I had to write a piece of tool that generate the RPZ zones - statically. So, the RPZ zone actually exists within our server ... It's just not public ... There is a tool that automatically fetches all data from here and parses /put them into multiple RPZ zones.
There is a tool that automatically fetches all data from here and parses /put them into multiple RPZ zones.
I do not quit understand what you are trying to say here, but if you do have the RPZ zones inside your bind, you only have to setup IXFR for all and share the connection information to for IXFR transfer (not AXFR), that will make the zones works like ordinary zone exchange as any other DNS zone from master to slaves (zone transfer)
It not even resource expensive on a bind as it is flat files over the network
http://www.zytrax.com/books/dns/ch9/rpz.html#config & http://www.zytrax.com/books/dns/ch7/rpz.html
If you like we can setup a ixfrdist as proxy and I would help sharing it as I would append it to my own DNS if I knew the the zone names you have been using
It's never incremental.
There you have your next project :wink:
PS: powerdns have a nice api for this.... :smirk:
when rpz files officially released
https://infosec.exchange/@mypdns/110845441169549255