Special character in PostgreSQL password breaks uri configuration

[Edefritz]

Hi,

I noticed that tegola breaks on startup when the connection uri has a special character with this message:

could not register providers: postgis: parse "postgres://db_user:db^password@my_db_url:5432/db_name": net/url: invalid userinfo

In my specific case I suspect the "^" character leading to the failure. The solution for me is to downgrade to 0.19.0 where passing the connection params as separate properties was still not deprecated. The problematic password doesn't seem to be an issue here. Of course I could change my password but when PostgreSQL allows this as a valid password, so probably should tegola.

Is suppose the underlying go package expects the connection string to be URL encoded somehow (potentially here?)

I found this thread which might be related: https://stackoverflow.com/questions/48671938/go-url-parsestring-fails-with-certain-user-names-or-passwords

If you think this is a valid issue, I'm happy to try my best on a PR.

[iwpnd]

As per url-spec it reads like it's the callers job to encode special characters. That's probably the reason why Golangs net/url package does not do it for you on .Parse(). But I can see how this is not really friendly. It also only happens on very specific special characters which makes it even more confusing for the user. I can look into it on the weekend. 👍

update: to unblock you use %5E instead of the caret. @Edefritz

update2: dug into that a little, ran some tests on different passwords. pretty sure its on the caller to encode everything that does not align with reserved characters. for tegola it would mean we'd have to manually parse the uri, get username and password, escape them, and hand them off to url.Parse for validation and finally to pgx to create the configuration. That's awkward. Double checked directly with pgx and it would error on their end as well, independently from tegola.

// Valid:
! $ & ' ( ) * + , - . / : ; = ? @ _ ~
% (should be followed by two hex chars to be completely valid (e.g. %7C))

// Invalid:
" < > [ \ ] ^ ` { | }

wdyt @ARolek?

[Edefritz]

Wow, thanks for the quick response. It's not super urgent for me, I can use 0.19.0 in the mean time, no problem.

However, as for your proposed temporary fix: It's not like I hardcode the password into the config. It comes for an environment variable that is derived from a kubernetes secret which is also used in other places. But I will check if I can apply some pre-processing and replace the problematic character as you suggested.

Thanks!

[ARolek]

@iwpnd / @Edefritz I have ran into this problem before and the solution is to url encode the password. The connection string is a URI so it does expect proper encoding.

It comes for an environment variable that is derived from a kubernetes secret which is also used in other places

I'm interested in what those other systems are doing behind the scenes. Do you have a reference to their handling of the URI?

[gdey]

As per url-spec it reads like it's the callers job to encode special characters. That's probably the reason why Golangs net/url package does not do it for you on .Parse(). But I can see how this is not really friendly. It also only happens on very specific special characters which makes it even more confusing for the user. I can look into it on the weekend. 👍

update: to unblock you use %5E instead of the caret. @Edefritz

update2: dug into that a little, ran some tests on different passwords. pretty sure its on the caller to encode everything that does not align with reserved characters. for tegola it would mean we'd have to manually parse the uri, get username and password, escape them, and hand them off to url.Parse for validation and finally to pgx to create the configuration. That's awkward. Double checked directly with pgx and it would error on their end as well, independently from tegola.

// Valid:
! $ & ' ( ) * + , - . / : ; = ? @ _ ~
% (should be followed by two hex chars to be completely valid (e.g. %7C))

// Invalid:
" < > [ \ ] ^ ` { | }

wdyt @ARolek?

I think we should err on the side of the user on this one. It's probably too confusing and complicates the documentation, for explicitly calling out the very few characters.

I do want to know the answer to @ARolek's question about what other systems are doing as well? Are they accepting the url as it. What about psql what does it expect?

Update:

https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS

It looks like they expect it to be encoded... So, maybe we state that. Alternative would be if it fails to connect due to password/user issue; we check to see if the invalid characters are in the URL if they are make an statement about invalid encoding, and try again with it encoded, or fail.