pdns icon indicating copy to clipboard operation
pdns copied to clipboard
verified publisher icon PowerDNS

Tiny issues in data result in "500 Internal Server Error" in API, while the server responds correctly via DNS protocol

Open martenlehmann opened this issue 2 years ago • 7 comments

  • Program: Authoritative
  • Issue type: Bug report

Short description

I noticed in several cases, that PowerDNS raises an exception in the log and returns 500 Internal Server Error in the API on records that it serves just fine via DNS queries. These exceptions happen at least on a zone info and on zone export.

This should never happen, as it's not obvious which other situations and data formats might result in similar exceptions, so these exceptions can't be prevented for sure (there might be lots of unexpected input despite some validation). However, 500 errors in an API prevent debugging the problem and resuming normal operations while doing so. On top of that, the log message uses the term "should", which means it gives a recommendation that is optional. But as the code makes it actually mandatory and raises exceptions otherwise, the handling needs to be considered broken.

Environment

  • Operating system: Ubuntu 22.04.2 LTS
  • Software version: 4.7.3-1pdns.jammy
  • Software source: PowerDNS repository

Steps to reproduce

  1. Insert a TXT record with a minor issue (not using quotes for your content for example, which is redundant anyway when using a database backend)
  2. Try to retrieve the zone via API and get a 500 Internal Server Error
  3. Find a log message like HTTP ISE for "/api/v1/servers/localhost/zones/example.com.": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'v=spf1 include:spf.example.com ?all'
  4. Resolve the TXT record via DNS just fine without errors and without log messages for comparison

Expected behaviour

Log the problem as a warning (the underlying issue should be for debate anyway), yet have all interfaces operating smoothly.

Actual behaviour

500 Internal Server Error responses, which break automated systems from operating and applying / verifying changes.

martenlehmann avatar Mar 16 '23 02:03 martenlehmann

Did you notice this issue with types other than TXT?

Habbie avatar Mar 16 '23 09:03 Habbie

I traced down the exception regarding the missing quotes to RecordTextException() in rcpgenerator.cc. There are other cases in which this exception is raised, but none that are relevant to our set of record types.

There is another case in which an exception is raised on unknown records that don't start with \# (Unknown record was stored incorrectly, first part should be '\#', got '#'). It's raised via MOADNSException() in dnsparser.cc.

I'm going to compare the behavior on these two.

When querying a TXT record without quotes:

  • Response: Just normal, PowerDNS adds leading and trailing quotes on the fly
  • Log: Nothing

When querying an unknown record without leading \#:

  • Response: SERVFAIL
  • Log:
Exception building answer packet for type65280.example.com/TYPE65280 (Unknown record was stored incorrectly, first part should be '\#', got '#') sending out servfail

When requesting AXFR for a zone with a TXT record without quotes:

  • Response: Just normal, full zone, PowerDNS adds leading and trailing quotes on the fly
  • Log: Nothing

When requesting AXFR for a zone with an unknown record without leading \#:

  • Response: communications error to <127.0.0.1>#53: end of file after SOA record (and additional RRSIG for SOA with DNSSEC enabled)
  • Log (twice, as dig seems to attempt a second query automatically):
AXFR-out zone 'example.com', client '127.0.0.1', transfer initiated
TCP Connection Thread died because of STL error, cycling backend: Unknown record was stored incorrectly, first part should be '\#', got '#'
AXFR-out zone 'example.com', client '127.0.0.1', transfer initiated
TCP server is without backend connections in doAXFR, launching
TCP Connection Thread died because of STL error, cycling backend: Unknown record was stored incorrectly, first part should be '\#', got '#'

When requesting /export via API for a zone with a TXT record without quotes:

  • Response: 500 Internal Server Error
  • Log:
HTTP ISE for "/api/v1/servers/localhost/zones/example.com./export": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'raw data'

When requesting /export via API for a zone with an unknown record without leading \#:

  • Response: 500 Internal Server Error
  • Log:
HTTP ISE for "/api/v1/servers/localhost/zones/example.com./export": STL Exception: Unknown record was stored incorrectly, first part should be '\#', got '#'

When executing pdnsutil check-zone on a zone with a TXT record without quotes: No issues.

When executing pdnsutil check-zone on a zone with an unknown record without leading \#:

[Error] Following record had a problem: "type65280.example.com IN TYPE65280 # 13 076578616d706c65036e657400"
[Error] Error was: Unknown record was stored incorrectly, first part should be '\#', got '#'

Interestingly a zone lacking NS records receives a message of severity "Error" as well, yet no exception or Internal Server Error:

[Error] No NS record at zone apex in zone 'example.com'

Also interesting: There doesn't seem to be an endpoint in the API to provide access to the results of pdnsutil check-zone.

So what worries me is:

  • These two exceptions have different effects on operation and the handling seems inconsistent.
  • There is no way for to retrieve the error messages / exceptions via the API.
  • While these errors seem to be only two affecting us right now, this might change in the future, just like unknown record without leading \# hasn't been an issue in previous releases when PowerDNS was more tolerant. PowerDNS already shows warnings regarding issues with trailing dots. So if these warnings are raised to errors and exceptions with a future release, that's like a damocles sword, as this would break chronological processing of jobs in a queue.

martenlehmann avatar Mar 21 '23 01:03 martenlehmann

Apr 3 16:12:49 ns1 pdns_server[1010]: [webserver] d8ed8dc6-66bb-482c-85bd-56fb6b4f728d HTTP ISE for "/api/v1/servers/localhost/zones/custom.zone.": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'unescapeddnsstring'

This error cannot be identified by using pdnsutil check-all-zones

I came across it writing sanity checks for my dns setup, by querying all zones via the http api I happened to encounter a 500 internal error

Producing line: https://github.com/PowerDNS/pdns/blob/master/pdns/rcpgenerator.cc#L592

For me, there was no fix beside direct sql editing

pdnsutil edit-zone refuses to open affected zones

dnsdist/unknown,now 1.7.3-1pdns.jammy amd64 [installed]
pdns-backend-mysql/unknown,now 4.7.3-1pdns.jammy amd64 [installed]
pdns-recursor/now 4.8.3-1pdns.jammy amd64 [installed,upgradable to: 4.8.4-1pdns.jammy]
pdns-server/unknown,now 4.7.3-1pdns.jammy amd64 [installed]

zones were created using zone2sql migrating from a bind9 text configuration, data source was configured this way. Would be nice if zone2sql attempted to fix.

ckbaker10 avatar Apr 04 '23 05:04 ckbaker10

Still present in 4.9.1. Actually breaking UI using the API.

grinapo avatar Oct 03 '24 17:10 grinapo

Still present in 4.9.1. Actually breaking UI using the API.

I'm having the same issue with 4.9.1.

Is there a solution? I'm using poweradmin web ui on same host to edit zones (with pdnsutil backend, not API), and adding quotes to TXT entries doesn't do anything, it returns the value like it was, without quotes.

When I use a separate poweradmin webui configured to use the API, it doesn't allow me to edit zones because of the issue.

HTTP ISE for "/api/v1/servers/localhost/zones/example.com": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'v=DKIM1; k=rsa;<snip>

when I run pdnsutil check-zone it shows no errors

pdnsutil check-zone example.com
Checked 28 records of 'example.com', 0 errors, 0 warnings.

so the API gives error but cli does not.

bretton avatar Apr 03 '25 10:04 bretton

Still present in 4.9.1. Actually breaking UI using the API.

I'm having the same issue with 4.9.1.

Is there a solution? I'm using poweradmin web ui on same host to edit zones (with pdnsutil backend, not API), and adding quotes to TXT entries doesn't do anything, it returns the value like it was, without quotes.

When I use a separate poweradmin webui configured to use the API, it doesn't allow me to edit zones because of the issue.

HTTP ISE for "/api/v1/servers/localhost/zones/example.com": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'v=DKIM1; k=rsa;<snip>

when I run pdnsutil check-zone it shows no errors

pdnsutil check-zone example.com
Checked 28 records of 'example.com', 0 errors, 0 warnings.

so the API gives error but cli does not.

edit the data directly in your database

ckbaker10 avatar Apr 03 '25 13:04 ckbaker10

edit the data directly in your database

thanks, I've started doing that after finding this link: https://www.claudiokuenzler.com/blog/1357/analyzing-fixing-internal-server-error-powerdns-api-opera-dns-ui

and then verifying I get JSON out (instead of validation error) with

curl -v -H 'X-API-Key: $APIKEY' http://MYHOST:8081/api/v1/servers/localhost/zones/example.com | jq .

however, poweradmin webui using API gives a new error when trying to edit domains:

Unable to parse request: HTTP version not supported

Will tackle in new issue, can't find anything relating to it in the archives

bretton avatar Apr 03 '25 14:04 bretton