thunderbird
Autoconfig mechanism K-9 does not pass the email address
I was looking into this and found:
Thunderbird's Autoconfig mechanism has now been available in K-9 Mail 6.7xx beta versions for a while.
Originally posted by @cketti in https://github.com/thunderbird/thunderbird-android/issues/865#issuecomment-1798419231
Currently K9 ask server with "GET" http://autoconfig.example.com/mail/config-v1.1.xml which is incorrect. It should ask with "[email protected]" as parameter according to Mozilla Guidelines: https://wiki.mozilla.org/Thunderbird:Autoconfiguration
Related issue: https://github.com/rseichter/automx2/issues/27
Hey, thanks for filing the report. The autoconfig docs are likely not complete, I don't think anyone has made a concerted effort to adjust these for a while. There are some efforts to standardize autoconfig, but I don't think this work has been completed.
Looking at the Thunderbird Desktop implementation, the email address seems to be sent by default, but there is an about:config option to disable it. So I think implementations should expect that the emailaddress parameter may be missing. K-9's implementation may be a bit more conservative in never sending the email address.
Can automx be adjusted to accept not passing an email address instead?
Can automx be adjusted to accept not passing an email address instead?
Why ever would it?
K9 is the only MUA that has been reported to not send the required email address since automx2 was first released in 2019. The retired predecessor automx also expected the address to be sent. Transmitting an email address is also a requirement for Autodiscover (Microsoft) and Mobileconfig (Apple). Dynamic configuration often depends on a user's email address, it's as simple as that.
If K9 does not conform to a specification which has been in active use for the last decade or thereabouts, it is K9 which needs to be changed, not existing third party software. automx2 works just fine with a multitude of other MUAs across different platforms.
I think k9 use non-standard behaviour so it should (but will it) act according to mozilla own standards or documentation need to be updated/expanded to current standards based on mozilla and those standards need to be implemented.
The standards work I was mentioning is https://datatracker.ietf.org/doc/draft-bucksch-autoconfig/ which I don't think is how Thunderbird Desktop works today. If this is about updating the documentation on the wiki to say that the email address may not be sent, then that should be easy to do.
I acknowledge this can be useful in various situations where config depends on the email address. We've chosen to be more conservative in the initial implementation. If this feature is added to K-9 I think we should double check the data sharing implications, as we'd be sending an email address of the user which might not be what they expect. We may also need to make some changes to the privacy policy.
I'm not insinuating that automx is doing anything wrong, I was just wondering if there is a way for it to be more lenient in case no email is passed, in the meanwhile. I don't believe we can get to this very soon, as we'd need to check the mentioned data questions, and it may need some UX in case we should give the user the choice if they'd like to not send their email.
The technical requirement for the Autoconfig process is a GET request containing
[email protected] . It must be a syntactically valid address, and automx2 matches the domain part against a locally configured set of domains. No domain match means "not my responsibility", and hence no configuration data will be returned. It is quite common to define different servers for different domains.
It does not matter to automx2 if the address is provided by the MUA or by a webserver acting as a proxy, as long as it is a valid address with a matching domain. If somebody were to engage in some clever HTTP request rewriting, automx2 would be none the wiser.
There are however certain aspects of the dynamic configuration which are not possible if some dummy address is used instead of "real" addresses. The mapping of email addresses to login names via LDAP comes to mind.
@kewisch the link you provided is draft: https://datatracker.ietf.org/doc/draft-ietf-mailmaint-autoconfig/ but even....
This protocol is in production use since 15 years by major email
clients, and the config database (used as fallback) contains
configurations for over 50% of all email accounts.
Currently, this protocol or parts of it has been implemented by:
...
* K9 Mail (https://k9mail.app)
...
And letter:
First step is to directly ask the mail provider and allow it to
return the configuration. This step ensures that the protocol is
decentralized and the mail provider is in control of the
configuration issued to mail clients.
* 1.1. https://autoconfig.%EMAILDOMAIN%/mail/config-
v1.1.xml?emailaddress=%EMAILADDRESS% **(Required)**
* 1.2. https://%EMAILDOMAIN%/.well-known/autoconfig/mail/config-
v1.1.xml **(Optional)**
* 1.3. http://autoconfig.%EMAILDOMAIN%/mail/config-v1.1.xml
**(Optional)**
So from my understanding DRAFT is telling that K9 does use this protocol, yet, it does not do the Required which is a bit strange, even for draft.
I understand that one could have privacy concern, yet we don't argue if we want send mail to MX server. If there is danger in sending mail address via parameter to domain that points Organization/Business/Person that is providing us the MX then I don't know what to think.
If one owns domain and want to use 3rd party server for MX then the one owning domain need to point DNS to that 3rd party Service (with or without autoconfig subdomain) so IMO in that case 3rd party already "know" the emailaddress of account.
In case of self-hosted/hosted on same domain via VPS - there is not real privacy danger.
Only case I would see privacy danger with sending emailaddress via parameter would be:
- in case that 3rd party gain control over MX/autoconfig domain/server but then the discovering email address is the least important issue.
- in case using 3rd party proxy there is always danger with sending anything that has value.
Also, the last thing I wanted to say is that imo, I never saw any client email application that would ask you if you want to send you "email" to "autoconfig domain", we can go one step further and say "why thunderbird send our domain to thunderbird server to get autoconfig there ??? " it didn't ask me if I want to... yet it does send and in most case failed to get configuration because most of those domains are small/medium mx servers.
The only setting I would see that has any sense here would be " [ ] Autoconfiguration " then according to standards that you propose in the draft client (thunderbird/k9) would send emailaddress to proper domain, and people that are afraid of loosing their privacy by sharing emailaddress to known domain could opt-out of that.
As @rseichter I can imagine that some configuration could be different for different user or per domain, but per domain basis is solved because autoconfig domain is the domain part from emailaddress, so even if MX domain is different from emailaddress domain then asking emailaddress autoconfig domain anything reveal use domain.
(hope I wrote it in a way you guys can understand it)
As @bigretromike mentioned, I believe that it boils down to this: Either K-9 uses Autoconfig and sends the required emailaddress parameter, or K-9 does not use Autoconfig over the Internet at all. K-9 could ask the user a Yes/No type question, but that's as far as it gets. I don't think that offering this choice would even be beneficial.
Sending the email address is conditio sine qua non. I also agree that sending a user's address to a webserver run by their selected mail service provider — via HTTPS request — does not pose any risk or grounds for privacy concerns.
Hi,
When I first reported the issue... https://github.com/thunderbird/thunderbird-android/pull/7655#issuecomment-1954675557 I was told, that K-9 Mail 6.603 doesn't support autoconfig despite this article https://blog.thunderbird.net/2023/06/thunderbird-for-android-k-9-mail-may-2023-progress-report that seems to say that Thunderbird’s Autoconfiguration (https://mzla.link/autoconfig) was integrated since May 2023. It was then further discussed here https://github.com/thunderbird/thunderbird-android/issues/4721 to finally be redirected to https://forum.k9mail.app/t/autoconfig-url-emailaddress-not-supported/8336 where it was argued that it is unsafe to send email address so K9-Mail does not do it.
As argued in last article above, it appears to me essential for Thunderbird for Android to be enabled for autoconfig, even if it means to transmit the email address over. The Desktop version do it, so why not the Mobile app?
As a safeguard, I support like rseichter the idea to simply ask the end user to confirm the sending of email address via a Yes(configure automatically)/No(configure manually). This is a workaround that could already be implemented, though Thunderbird for Desktop does not do it so why would the mobile version need to do so? As an end-user I would expect Desktop and Mobile apps to behave the same way for autoconfig.
Currently while setting up a new account on Mobile app, by entering email address, pressing next and receiving a Configuration Not Found is really not user friendly nor intuitive.
Better more secure implementation of autoconfig can be worked on, but until widely publicly available, the old already working autoconfig process should be kept working.
Regards,
@richardleger Since you mentioned me explicitly: I want to emphasise that I don't think offering the end user a choice in this matter is beneficial in any way, shape or form.
In particular, choosing to opt out of autoconfig to avoid sending one's email address to an organisation which handed out said address in the first place is utterly nonsensical to me.
sending one's email address to an organisation which handed out said address in the first place is utterly nonsensical to me.
It is not that simple, because the address transit via a GET request and HTTP protocol rather than a POST request over HTTPS, the request and therefore the email address is visible in clear on any server it transit through over Internet till it reaches destination which causes a Privacy and Security concerned as it can be capture/sniffed in the process... Same for the answer with configuration details. That is my understanding of the issue, and it does make sense. I would add that in TB Desktop this is done automatically without user consent at the moment, so adding a consent step could be a temporary workaround the issue, giving choice to end-user via an informed decision.
While some remedy shall address those concerns at some point, in the meantime, Thunderbird for Android shall behave like Thunderbird for Desktop in term of auto-configuration and provide the same user experience by enabling it already.
kewisch tagged this Post as enhancement so hopefully that would put it on the roadmap for improvement, but that should really be considered as a bug in the software and be fixed quickly.
It is not that simple, because the address transit via a GET request and HTTP protocol rather than a POST request over HTTPS
I have a pretty good grasp of how autoconfig works, if I say so myself (see here). HTTPS connections are used first. HTTP is merely a fallback, dating back to times when obtaining TLS certificates was a hassle (i.e. pre Let's Encrypt).
If a mail provider runs an autoconfig service without HTTPS, well, that's not a provider I would choose, although transmitting an email address in the clear is not something that worries me personally. The address will become "visible" anyway, or what is the point of having it?
In any case, K-9 can forgo the fallback to HTTP and use only HTTPS. It is also easy to check if the provider supports HTTPS in the first place, and inform the K-9 user if it does not.
Is there any update to this? My mail server (stalwart) requires this field to return valid information since I am serving multiple domains.
In HTTPS, URL path is encrypted, and GET params. No security issue on this side.
Providing email for many domains, without the GET param, requires one to do some redirect magic and this is long to find and cumbersome when added to all the other mail clients specific behaviors one already has to deal with...
PLEASE add this trivial, required thing, and move on.
FTR, the Apache config that adds domain parameter in addition to emailaddress thanks to SERVER_NAME and QSA (especially required since K9Mail web client removes the referer so you cannot access its domain in the redirected request) :
RewriteOptions InheritDown
<IfModule mod_rewrite.c>
RewriteEngine On
# Check if the request is for /.well-known/autoconfig/mail/config-v1.1.xml
RewriteCond %{REQUEST_URI} ^/\.well-known/autoconfig/mail/config-v1\.1\.xml$
# Redirect to the target URL and append the query string
RewriteRule ^.*$ https://autoconfig.hoster.com/autoconfig.php?domain=%{SERVER_NAME} [R=302,L,QSA]
</IfModule>
If you have a concern about privacy that supersedes the need for individual mail configs, maybe you can do it like Evolution and send &[email protected]
Sending a manipulated e-mail address doesn't help, when the Autoconfig-URL needs the real e-mail address to determine the value of the <username>-field in the Autoconfig-XML.
One cannot assume, that the username always matches the e-mail address (or the local part of it).
Not necessarily as there is the syntax <username>%EMAILADDRESS%</username>. This would be enough in most cases as account-specific configs are rare.
The purpose of the Autoconfig-URL is to work on every setup and not just some setups.
I am the postmaster of a setup where <username>%EMAILADDRESS%</username> will not work and only about half of the e-mail providers I have used over time would have worked with such an assumption.
In such cases you either need the e-mail client to send the correct e-mail address or to support <userinput>-fields (https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat#User_input_fields), which K-9 doesn't do either.
there is the syntax
<username>%EMAILADDRESS%</username>. This would be enough in most cases as account-specific configs are rare.
@nxmndr Are they now? In automx2, I specifically added the ability of mapping email addresses to configurable LDAP attributes (i.e. the login name), because people need that feature. This requirement is not as rare as you may think, in particular in larger organisations with uniform sign-on for multiple systems.
Also we shouldn't encourage Microsoft like behavior on standards - where you do it halfway and halfway your way..
The standards says clearly that emailaddress part on https is REQUIRED
* 1.1. https://autoconfig.%EMAILDOMAIN%/mail/config-
v1.1.xml?emailaddress=%EMAILADDRESS% **(Required)**
* 1.2. https://%EMAILDOMAIN%/.well-known/autoconfig/mail/config-
v1.1.xml **(Optional)**
* 1.3. http://autoconfig.%EMAILDOMAIN%/mail/config-v1.1.xml
**(Optional)**
@oddlama I have experienced this to and created a issue there: https://github.com/stalwartlabs/mail-server/issues/1042 @nxmndr for Evolution there is also a problem: gnome-evolution#2941
Using https first with the correct e-mail address will solve the Stalwart mailserver issues. If https fails (for whatever reason) you can try http with the current implementation as is.
When will the version 9 be deployed to android. The autoconfig works for v9 but not v8 which is currently the default in android store via https://autoconfig.example.com/mail/config-v1.1.xml
v9 is currently rolling out at 10%, we expect to raise that over the course of the week. Note that v9 still does not pass the email address, but we've fixed an issue on what sources are used.
Note that v9 still does not pass the email address,
Very disappointing to hear that!!!
By doing so you create unnecessary hurdle to end-users, and render the app not user friendly. This will really not help adoption of Thunderbird on Android!!!
At the least, why not simply warn the end-user about the privacy concern and let them decide to either consent to pass their email address for auto config or to configure manually as alternative. This should be a user choice not a dev choice. Instead of forcing end-users into an unsatisfying solution, and raising an understandable and avoidable known error created by an arbitrary decision from the dev team!
It works fine with Thunderbird Desktop app so it make no sense not to provide the feature in Thunderbird-Android if not K9Mail!!!
A lot of suppliers and organisations out there rely on the auto-config feature with email address to identify the correct user and its parameters, such as username when it differs from the email address... among other per user customised settings...
Until a better more secure standard solution (for addressing privacy concern) can be put it place, why to brake intentionally what already exists, and what has worked for so many years and is still working today in so many other email client software including Thunderbird.
It really make no sense to brake intentionally feature that works and are needed today. It does not prevent to implement better solution later on, over time to address privacy concerns.
Note that v9 still does not pass the email address,
Very disappointing to hear that!!!
True
It works fine with Thunderbird Desktop app so it make no sense not to provide the feature in Thunderbird-Android if not K9Mail!!!
This is very much True ! and this one is the most concerning as it looks like both products are going in different ways.... different ideology which is a bit scary.
I been using and recommending thunderbird from early version (even before it was called thunderbird) and most of the times it was well received - with exception of pasting excel/word into mail that would totally break formatting and offend only those that knew outlook would do better job at it. Even when core features wasn't that great plugin/addon system came to help.
This is very strange how same product can have different ideology on different platforms.
but we've fixed an issue on what sources are used.
I hope that this mean it will pass email but not in v9 version.
but we've fixed an issue on what sources are used.
The only fix here is that it will prefer an autoconfig from the provider/domain before falling back to ispdb, see the changelog.
I know you are all very passionate about this issue, it shows in how much effort you are making in responding and countering the arguments. Please do note that Thunderbird Desktop and Android come from completely different histories, and have different form factors. You might be reading too much into it if you feel there being deviation is concerning and ideological.
I'm sure we'll revisit this at some point and see how we can let the user make that choice, and when we do we'll likely do it both across desktop and mobile. It just isn't a priority right now. If we were completely opposed to finding a solution, or see our current choice as final, we would have closed the issue. But as you can see it is still open.
Until then, I believe all the necessary information is on the table here. I'd appreciate if we could restrict commenting to actual development work. A good next step might be some mockups on different ways to present this to the user, and/or some qualified legal research on if sending the email address requires an opt-in or not.
Mockups? Legal research? I'll assume this is your entry for the "Tell me you have way too much time and no interest in getting a trivial feature done" contest. Remarkable. 😆
I'm not here to favor or disfavor any technical aspect of this reported issue, but to bring clarity to what's productive here in github. Not all issues can be at the very top of development priority. And when that is true then we have these possible useful steps, offer code which will solve the problem, help design or scope it out as @kewisch has previously mentioned, or add data to help better understand the issue.
We all understand disappointment, but expressing disappointment or outrage here doesn't fit one of those buckets, and doesn't help the goal of making the product better.
Okay, probably the following code change will solve the problem:
thunderbird-android/feature/autodiscovery/service/src/main/kotlin/app/k9mail/autodiscovery/service/RealAutoDiscoveryRegistry.kt, line 20, change the value of includeEmailAddress from "false" to "true".
Maybe you don't recognize the problem, because your test case already uses "true" instead of "false" an thus hides the problem in the production code? See feature/autodiscovery/autoconfig/src/test/kotlin/app/k9mail/autodiscovery/autoconfig/ProviderAutoconfigUrlProviderTest.kt, lines 16 and 32.
Additional data to help better understand the issue?
I think we have given enough explanations why the current implementation breaks auto configuration whenever the POP/IMAP login is not identical to the e-mail address or the localpart.
If legal research is necessary, it has probably been done for Thunderbird for Desktop and the results did not prevent a proper implementation there, so they won't prevent a proper implementation in Thunderbird for Android. If legal research hasn't been necessary for Thunderbird for Desktop, why should it be necessary for Thunderbird for Android?
Thanks for focusing on the implementation side, much appreciated! I agree that if it just a matter of flipping the default it would be that one line change. For desktop it was apparently considered to be ok back then given the privacy policy mentions "During this process, your email domain may be sent to Mozilla's configuration database, and your email address may be disclosed to your network administrators".
However, the legal landscape changes over time, and I think we need to evaluate this for both apps. Not to say we can't include the email address, but rather to determine if we need to ask for consent and how.
To be clear for those doubting: I'd love to find a way to include the email address to not break the systems that expect it, but I'd like to make sure we do it responsibly. To set expectations, I can have the legal aspects reviewed, but I can't promise when that would be given the other priorities.
As we are probably either developers or email administrators here, a valid contribution to new legal research is rather difficult.
In the meantime, it might be an option to submit the email address but drop HTTP support (which could also be done with a switch in the code), as there should be no privacy concerns with an HTTPS connection.
Dropping HTTP support of course also “breaks” the RfC, but for administrators of affected systems, enabling HTTPS should be much easier than switching the authentication mechanism to login = email address or localpart (which potentially affects all users of the respective service).
I understand that the topic may seem rather unimportant, but for average users of affected e-mail services it is decisive as to whether they set up Thunderbird/K9 or not.