certtools
Cymru whois expert: handle strange ASN symbols better
I discovered a slight issue with the Cymru whois expert bot with some strange AS names. For instance, AS266522.
$ dig +short TXT AS266522.asn.cymru.com
"266522 | BR | lacnic | 2017-01-20 | Realnet Telecomunicacoes e Multimidia ltda\226\128\143 , BR"
\226\128\143, in unicode, decodes to u+200f, the right-to-left mark. When the IMQ bot attempts to decode it with result['as_name'] = items[4].encode('latin1').decode('utf8'), it runs into this issue:
UnicodeEncodeError: 'latin-1' codec can't encode character '\u200f' in position 42: ordinal not in range(256)
A solution which just permits decoding special symbols like RTL marks might not be the best, though just discarding these events into the dump file is also not a good option.
Various online tools for finding public data on ASes, found online via a quick search, give varying results.
- ipinfo.io displays it in 3 different ways ("injected" RTL,
â€andaâ) - qrator displays it as
‏ - bigdatacloud displays it as
8207 - bgpview displays it as
ÂÂ - etc.
I suppose there's no existing expectation of invalid UTF-8 sequences or special symbols like RTL marks in AS names, nor any "standard" way of handling them, and I feel iffy about passing arbitrary symbols like that on to other systems.
Are there any ideas for how to handle cases like this? The simplest option would be to simply pass the undecoded string on in case the decoding step fails.
I had some fun with the encoding issues in AS names in the past already (https://github.com/certtools/intelmq/commit/fff0b6e5f3998ee6621350f270c95e5c3bd15111#diff-016f562585e3b1dd9bbc46a808574681f44f9f1f522890f9a8740182ea61e81f https://github.com/certtools/intelmq/commit/3f0983dcdca9a87f784195e9cbc71bd053caf764#diff-016f562585e3b1dd9bbc46a808574681f44f9f1f522890f9a8740182ea61e81f https://github.com/certtools/intelmq/commit/cb4948bab7735fa81a7f76c1982a38cd35228403#diff-016f562585e3b1dd9bbc46a808574681f44f9f1f522890f9a8740182ea61e81f #307) and I would now be in favour of doing .decode(errors='ignore') to decode everything we can and not fail with the other bogus data like this example.
