spf-tools
spf-tools copied to clipboard
spf-tools
.spf-toolsrc ignored?
Hello
I created a file /root/.spf-toolsrc:
DOMAIN=our-domain.eu
ORIG_SPF=spf-orig.our-domain.eu
DESPF_SKIP_DOMAINS=
DNS_TIMEOUT=5
DNS_SERVER=1.1.1.1
TOKEN=cloudflare_token_here
[email protected]
If I run this from /opt/spf-tools, where I checked out your repository:
./despf.sh | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./cloudflare.sh
Absolutely nothing happens our appears.
It only works a bit if I specify it like this:
./despf.sh spf-orig.our-domain.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh our-domain.eu
I get the idea that the config file isn't loaded or ignored or something. If I add | ./cloudflare.sh at the end, it also results in just nothing.
Hi! Let me have a look. I have not used the cloudflare.sh script for a while...
First thing, the cloudflare.sh script was not verbose enough if it was missing jq or other tool. Please make sure you have jq installed.
Second, configuration file was read too late. Both fixed in following merge.
After the patches it works for me...
lomidrevo:~/src/spf-tools$ ./despf.sh | ./normalize.sh | ./simplify.sh | ./ipran
ge.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf1.jasan.tk
Getting spf2.jasan.tk
Getting spf3.jasan.tk
Changing jasan.tk^"v=spf1 with id ... OK
Changing spf1.jasan.tk^"v=spf1 with id ... OK
Changing spf2.jasan.tk^"v=spf1 with id ... OK
Changing spf3.jasan.tk^"v=spf1 with id ... OK
I just tried and I can confirm that it works. Thank you very much for responding so quickly and fixing this!
Hmm, there seems to be something still off:
root@appserver:/opt/spf-tools# ./despf.sh | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf1.our-domain.eu.eu
jq: error: Cannot iterate over null
Changing jasan.tk with id ... OK
It did something with the correct domain since it fetched spf1.our-domain.eu, but then in the last phase it went to the default jasan.tk after all.
If I specify the domain to start off:
root@appserver:/opt/spf-tools# ./despf.sh spf-orig.our-domain.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
jq: error: Cannot iterate over null
Changing spf8.jasan.tk with id ... OK
Changing spf7.jasan.tk with id ... OK
Changing spf6.jasan.tk with id ... OK
Changing spf5.jasan.tk with id ... OK
Changing spf4.jasan.tk with id ... OK
Changing spf3.jasan.tk with id ... OK
Changing spf2.jasan.tk with id ... OK
Changing spf1.jasan.tk with id ... OK
Changing jasan.tk with id ... OK
@MaartenUreel yes, my fault. I moved the line that reads the spf-toolsrc, but then the defaults rode over the values from the rc file.
Nope:
root@appserver:/opt/spf-tools# ./despf.sh spf-orig.our-domain.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
Changing jasan.tk^"v=spf1 with id ... OK
Changing spf1.jasan.tk^"v=spf1 with id ... OK
Changing spf2.jasan.tk^"v=spf1 with id ... OK
Changing spf3.jasan.tk^"v=spf1 with id ... OK
Changing spf4.jasan.tk^"v=spf1 with id ... OK
Changing spf5.jasan.tk^"v=spf1 with id ... OK
Changing spf6.jasan.tk^"v=spf1 with id ... OK
Changing spf7.jasan.tk^"v=spf1 with id ... OK
Changing spf8.jasan.tk^"v=spf1 with id ... OK
Also if I run with the mkzoneent, it also gives the wrong domain:
root@appserver:/opt/spf-tools# ./despf.sh spf-orig.telsmart.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
spf8.jasan.tk. 1800 IN TXT "v=spf1 ip4:89.234.34.154 ip4:89.234.34.164 ip4:94.236.44.253 ip4:94.236.44.254 ip4:94.245.120.64/26 ip4:95.138.143.148 ip6:2001:489a:2202::/48 ip6:2a01:111:f400::/48 ~all"
[..snip..]
jasan.tk. 1800 IN TXT "v=spf1 ip4:103.2.140.0/22 ip4:103.36.108.0/22 ip4:103.47.204.0/22 ip4:104.130.122.0/23 ip4:104.130.96.0/28 ip4:104.47.0.0/17 ip4:109.68.161.215 ip4:109.68.163.0/26 ip4:109.68.163.128/26 ip4:109.68.163.192/26 ip4:109.68.167.96/27 include:spf1.jasan.tk ~all"
@MaartenUreel please check 64b980d
Seems you are the first one to really run this full chain of scripts :-) I mean others probably run only ./despf.sh and I did not realize the rest does not work really.
OK, the mkzoneent now returns the correct domains.
However, I still have to pass my spf-orig.our-domain.eu as a parameter to despf.sh while it is also in the config file.
The records are not being created / updated on CloudFlare neither though.. :)
root@appserver:/opt/spf-tools# ./despf.sh | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.google.com
Getting _netblocks.google.com
Getting _netblocks2.google.com
Getting _netblocks3.google.com
Now retrieves a bunch of stuff where I don't know where he got it, for sure not from the domain configured in ORIG_SPF. If I pass it as a parameter to despf.sh it's still OK.
Perhaps add a flag that outputs some verbose info to the cloudflare script, so I can give you more info?
lomidrevo:~/src/spf-tools$ ./despf.sh | ./normalize.sh | ./simplify.sh | ./ipran
ge.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.google.com
Getting _netblocks.google.com
Getting _netblocks2.google.com
Getting _netblocks3.google.com
Changing jasan.tk with id a2f0c894e7a71759508ea034772cc26b... OK
Changing spf1.jasan.tk with id 940b5ae32e24350407b208ff3ba68b03... OK
Changing spf2.jasan.tk with id 7f6c6730a9ffdfe637d1d6a563f9e17d... OK
Changing spf3.jasan.tk with id 0e9af188245dd693617a7a91f0ab71ae... OK
lomidrevo:~/src/spf-tools$ export USE_UPSTREAM=1
lomidrevo:~/src/spf-tools$ ./compare.sh
Getting spf1.jasan.tk
Getting spf2.jasan.tk
Getting spf3.jasan.tk
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.google.com
Getting _netblocks.google.com
Getting _netblocks2.google.com
Getting _netblocks3.google.com
Everything OK
@MaartenUreel Thank you very much for bringing this up! It really needed some care.
root@appserver:/opt/spf-tools# ./despf.sh spf-orig.telsmart.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
./cloudflare.sh: 54: shift: can't shift that many
rm: cannot remove ‘/tmp/cloudflare-zone-9QQY-data’: No such file or directory
Also thank you for building this stuff. I'm a good Python programmer but Bash isn't my thing :)
You also do not need to supply the domain name on the command line of despf.sh anymore (given it is set either in ~/spf-toolsrc or in environment variable DOMAIN).
I can indeed drop the domain now, that part works. Cloudflare part not yet:
root@appserver:/opt/spf-tools# ./despf.sh | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
./cloudflare.sh: 54: shift: can't shift that many
rm: cannot remove ‘/tmp/cloudflare-zone-Pnn4-data’: No such file or directory
Btw, do you need to pipe through mkzoneent.sh before Cloudflare, or not? What format does it expect?
Are you planning to do any work on this? Just to know, because otherwise I'll have to write some python scripts to take care of this :)
@MaartenUreel excuse me for silence. I do not know how to reproduce the error since it works for me. I can have a look but please provide me with more details.
Maybe relevant output when you run everything with sh -x will help. There should be DEBUG environment variable used in most scripts so first try to export DEBUG=1 and run the same way as you did before. If it doesn't help, add sh -x before the last (cloudflare) script.
And to answer your question, piping through mkzoneent.sh is not needed. That script outputs the zone entries in format for ISC BIND DNS server.
@MaartenUreel This works for me:
lomidrevo:~/src/spf-tools$ ./despf.sh | ./normalize.sh | ./simplify.sh | ./ipran
ge.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.google.com
Getting _netblocks.google.com
Getting _netblocks2.google.com
Getting _netblocks3.google.com
Changing jasan.tk with id a2f0c894e7a71759508ea034772cc26b... OK
Changing spf1.jasan.tk with id 940b5ae32e24350407b208ff3ba68b03... OK
Changing spf2.jasan.tk with id 7f6c6730a9ffdfe637d1d6a563f9e17d... OK
Changing spf3.jasan.tk with id 0e9af188245dd693617a7a91f0ab71ae... OK
lomidrevo:~/src/spf-tools$ cat ~/.spf-toolsrc
DOMAIN=jasan.tk
ORIG_SPF=spf-orig.jasan.tk
DESPF_SKIP_DOMAINS=
DNS_TIMEOUT=5
DNS_SERVER=8.8.8.8
TOKEN=<mytoken>
[email protected]
lomidrevo:~/src/spf-tools$ git rev-parse HEAD
2bee0b54ec18ea2cb49f6a9b000f587e5e9de9d9