Consistently getting BadAuthentication

[stevearc]

The same effect as #24, though possibly a different cause. Every time I do gpsoauth.perform_master_login() I receive a BadAuthentication error. This happens for a normal password protected account as well as 2FA using an app password. This is also happening to @kaddkaka , so slightly less chance that it's just something weird with my account. Has Google changed something in their API recently?

OS: linux Python: 3.10.4 pip freeze:

certifi==2022.9.14
cffi==1.15.1
charset-normalizer==2.1.1
cryptography==38.0.1
future==0.18.2
gkeepapi==0.14.2
gpsoauth==1.0.2
idna==3.4
jaraco.classes==3.2.2
jeepney==0.8.0
keyring==23.9.3
keyrings.alt==4.2.0
more-itertools==8.14.0
pycparser==2.21
pycryptodomex==3.15.0
requests==2.28.1
SecretStorage==3.3.3
urllib3==1.26.12

[simon-weber]

A test login just worked for me. I also tried that version of urllib3.

[stevearc]

Hmmm...maybe something odd is going on with both repro accounts then. If you don't mind, I'll leave this issue open for a bit longer and try to do some more investigation.

[jnperry]

i didn't have time to look into the cause, but i had this exact same problem when i was trying to use the library on ubuntu. however, it worked fine in windows. can you try a different platform?

[simon-weber]

Were they two different machines? Sometimes Google will do a mandatory mfa step when it sees a new device, which I think results in BadAuth (I can't remember).

[kaddkaka]

I have only tried this from my Ubuntu 22.04 machine, and I just get the BadAuthentication. Not sure what I can do to investigate further.

[simon-weber]

There's no logging of the responses or anything (since they might be sensitive) but you could hack that into your local code and hopefully get more information on why the request is failing. The code is pretty simple: https://github.com/simon-weber/gpsoauth/blob/master/gpsoauth/init.py

[jnperry]

sorry, it was the same machine. i use my google account regularly on both linux and windows on that computer, so not sure. i only mentioned it because i saw another post on another project (sorry, i can't find it anymore) that mentioned several people were having trouble with this auth on the latest ubuntu

[stevearc]

I tried again on my non-work laptop with an account that does not have 2-factor enabled, and the login worked. Before when I was testing it I was traveling, so it's entirely possible that Google was blocking access because the traffic looked suspicious. @kaddkaka I think your best bet is Simon's suggestion to add more logging to gpsoauth and see if you can find more information. Initially I thought this could be a gpsoauth issue since it was happening to me as well (it's worked without issue for a long time), but now it seems more likely that it's some account-specific problem.

[markx]

Same issue when I tried to login with app password (my account has 2FA enabled).

---

Update: OK it failed when I was using the Python3.9 that came with MacOS.
However, it worked then after I use Python3.10 installed via homebrew! Not sure what's the root cause here though.

[Shonke]

Same issue when I tried to login with app password (my account has 2FA enabled).

Update: OK it failed when I was using the Python3.9 that came with MacOS. However, it worked then after I use Python3.10 installed via homebrew! Not sure what's the root cause here though.

Unfortunately, it doesn't work for me

[leikoilja]

We are facing the same problem with the package that relies on gpsoauth (related to https://github.com/leikoilja/ha-google-home/issues/599).

Seems that I was just able to replicate the issue. Authentication works fine on the Mac M1 machine with OpenSSL 1.1.1j 16 Feb 2021 (openssl version), but on Ubuntu 22.04 with OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022) I am hit with BadAuthentication. @simon-weber, do you think you can try the same? I just spun the VM with ubuntu 22.04, checked openssl version (verify with python -c "import ssl; print(ssl.OPENSSL_VERSION)" that it's OpenSSL 3.0.2) and was able to replicate the issue. ping @KapJI

[simon-weber]

Cool, I can repro with openssl 3. Switching from EncryptedPasswd to Passwd works on the old openssl but not the new one, so my guess is urllib3 is doing something different with the tls that Google doesn't like. This happened once before in https://github.com/urllib3/urllib3/issues/2101.

[Tony450]

Does this script work as of May, 2023? Or Google has changed something that makes this script not working? I have tried the steps that are posted in this issue and none of them have worked for me

[simon-weber]

It worked for me just now on openssl 1.

[engdorm]

so what did you do? did you import something else? installed something? or set some configuration?

[Tony450]

In my case it finally worked with OpenSSL 1. I used Kali Linux 2021.2 (in which OpenSSL 1 is not changed by OpenSSL 3 and has Python 3.9.2 by default). I tried to change from OpenSSL 3 to OpenSSL 1 in another virtual machine (the last version of Kali) but none of the guides published out there worked for me. It was easier to just install an older distribution that came with OpenSSL 1.

[artickl]

Still the same:

• urllib3==1.26.16 with OpenSSL 1 working fine
• urllib3-2.0.3 with OpenSSL 1 giving no attribute 'DEFAULT_CIPHERS'
• urllib3==1.26.16 with OpenSSL 3 giving {'Error': 'BadAuthentication'}
• urllib3-2.0.3 with OpenSSL 3 giving no attribute 'DEFAULT_CIPHERS'

So, maybe it's time to fix DEFAULT_CIPHERS issue https://github.com/simon-weber/gpsoauth/issues/52?

[miccico]

Hi All!

I'd like to hijack this issue as i had the BadAuthentication problem myself on Linux but not on Windows. As Google in their infinite wisdom decided to kill the integration to custom shopping list providers I had a huge motivation to get my sync running so i sunk a few hours into that problem.

What I noticed is that on linux the EncryptedPasswd string was terminated with a \n while on windows it wasn't.

Easy but ugly fix: within init.py modify _perform_auth_request and just strip it off if it is there

   session.headers = {
        "User-Agent": USER_AGENT,
        "Content-type": "application/x-www-form-urlencoded",
    }                        
    if data["EncryptedPasswd"].endswith("\n"):
        data["EncryptedPasswd"]=data["EncryptedPasswd"][:-1]
        print("RemovedNewline")
    res = session.post(AUTH_URL, data=data, verify=True)

Done that and it just works every time now. I hope that helps some of you

best regards

Michael

[simon-weber]

Can someone with the openssl v3 issue try this and see if it helps?

[artickl]

@simon-weber / @miccico,

Sorry, I checked on both versions by adding a new test into pytest "test_48.py" (I didn't commit anything, because I did not see a value in it):

gpsoauth$ cat tests/test_48.py
"""Tests for issue 48: 'Consistently getting BadAuthentication #48'"""

import gpsoauth

def test_master_key() -> None:
    email = '[email protected]'
    password = 'my-password'
    android_id = '0123456789abcdef'

    master_response = gpsoauth.perform_master_login(email, password, android_id)
    print(master_response)

    assert master_response == "{'Error': 'BadAuthentication'}

Works well as is for urllib3==1.26.5 with OpenSSL 1.1.1:

(gpsoauth-py3.9) artickl@penguin:~/git/github/gpsoauth$ cat /etc/debian_version 
11.7
(gpsoauth-py3.9) artickl@penguin:~/git/github/gpsoauth$ dpkg --list | grep -e "openssl " -e "python3-urllib3"
ii  openssl                           1.1.1n-0+deb11u5               amd64        Secure Sockets Layer toolkit - cryptographic utility
ii  python3-urllib3                   1.26.5-1~exp1                  all          HTTP library with thread-safe connection pooling for Python3

gpsoauth$ pytest
=================================== test session starts ===================================
platform linux -- Python 3.9.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: ..../gpsoauth
collected 3 items                                                                         

tests/test_48.py F                                                                  [ 33%]
tests/test_all.py ..                                                                [100%]
================================= short test summary info =================================
FAILED tests/test_48.py::test_master_key - assert {'Error': 'NeedsBrowser', 'ErrorDetail...

But still failing with {'Error': 'BadAuthentication'} on urllib3==1.26.5 with OpenSSL 3.0.2:

gpsoauth$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"

gpsoauth$ dpkg --list | grep -e "openssl " -e "python3-urllib3"
ii  openssl                               3.0.2-0ubuntu1.10                       amd64        Secure Sockets Layer toolkit - cryptographic utility
ii  python3-openssl                       21.0.0-1                                all          Python 3 wrapper around the OpenSSL library
ii  python3-urllib3                       1.26.5-1~exp1                           all          HTTP library with thread-safe connection pooling for Python3

gpsoauth$ pytest 
=================================== test session starts ===================================
platform linux -- Python 3.10.6, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: ..../gpsoauth
collected 3 items                                                                         

tests/test_48.py F                                                                  [ 33%]
tests/test_all.py ..                                                                [100%]
==================================== warnings summary =====================================
tests/test_48.py::test_master_key
  /home/lenovo/git/github-personal/gpsoauth/gpsoauth/__init__.py:75: DeprecationWarning: ssl.SSLContext() without protocol argument is deprecated.
    context = SSLContext()

tests/test_48.py::test_master_key
  /home/lenovo/git/github-personal/gpsoauth/gpsoauth/__init__.py:75: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
    context = SSLContext()

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================= short test summary info =================================
FAILED tests/test_48.py::test_master_key - assert {'Error': 'BadAuthentication'} == "{'Error': 'BadAuthentication'}"
========================= 1 failed, 2 passed, 2 warnings in 0.40s =========================

Tests are giving the same result on Jun 1 commit without any additional changes:

gpsoauth$ git log
commit b22c57a236eb0bd63517cb4c1bf35d3fca35e2c7 (HEAD -> master, origin/master, origin/HEAD)
Author: Simon Weber <[email protected]>
Date:   Thu Jun 1 21:18:38 2023 -0400

    appease linters

And with changes based on @miccico suggestion give the same BadAuthentication result:

gpsoauth$ git diff gpsoauth/__init__.py
diff --git a/gpsoauth/__init__.py b/gpsoauth/__init__.py
index a9b2378..bacb4ed 100644
--- a/gpsoauth/__init__.py
+++ b/gpsoauth/__init__.py
@@ -94,6 +94,11 @@ def _perform_auth_request(
         }
     )
 
+    print(data["EncryptedPasswd"])
+    if data["EncryptedPasswd"].decode().endswith(tuple("\n")):
+        data["EncryptedPasswd"]=data["EncryptedPasswd"][:-1]
+        print("RemovedNewline")
+
     res = session.post(AUTH_URL, data=data, verify=True)
 
     return google.parse_auth_response(res.text)

[simon-weber]

Thanks for testing it out! It looks like that fixes a separate issue from the openssl one.

[ELind77]

Hi all,

I'm having the same issue. It's a bummer because there are a bunch of downstream projects getting hit with the same problem as well and this is clearly a project used by a lot of other libraries.

@simon-weber would it be possible to get a summary of the state of the world on this bug? What you think the root cause is and any possible fixes? I'm running this in Home Assistant so fixes that can be done in an environment where I have a shared OS with other dependencies would also be appreciated.

Eric

[simon-weber]

My guess is that this is something like https://github.com/urllib3/urllib3/issues/2101, where a) Google is picky about tls params and b) using openssl v3 shifts them to something Google doesn't like. If someone wants to test that, they can set up a debugging https proxy and compare a working request to a non-working one.

As a workaround you can use a pre-v3 openssl version.

[jkitching]

Here's my solution for the moment:

gpsoauth_login.Dockerfile

FROM alpine:3.16
RUN apk upgrade --update-cache --available && \
    apk add python3 py3-pip && \
    pip install gpsoauth 'urllib3<=1.25.11'
RUN echo $'import gpsoauth; import json\n\
ret = gpsoauth.perform_master_login(input(), input(), input())\n\
print(json.dumps(ret))' > /glogin.py
ENTRYPOINT ["python3", "/glogin.py"]

Usage:

$ docker build -t gpsoauth_login -f gpsoauth_login.Dockerfile .
$ docker run -it --rm gpsoauth_login
<stdin: username>
<stdin: password>
<stdin: android_id>
{'SID': 'BAD_COOKIE', 'LSID': 'BAD_COOKIE', 'Token': ... }

Save your token somewhere, and get your application to use that instead of calling perform_master_login directly.

Here's an example of saving the token directly into your keyring:

(Python) https://gist.github.com/jkitching/57baead775f6773848d2e334cc950999 (Bash) https://gist.github.com/jkitching/1e6bab9e3fd935aca2169528cd7eb497

[VNRARA]

Here's my solution for the moment:

gpsoauth_login.Dockerfile

FROM alpine:3.16
RUN apk upgrade --update-cache --available && \
    apk add python3 py3-pip && \
    pip install gpsoauth 'urllib3<=1.25.11'
RUN echo $'import gpsoauth; import json\n\
ret = gpsoauth.perform_master_login(input(), input(), input())\n\
print(json.dumps(ret))' > /glogin.py
ENTRYPOINT ["python3", "/glogin.py"]

Usage:

$ docker build -t gpsoauth_login -f gpsoauth_login.Dockerfile .
$ docker run -it --rm gpsoauth_login
<stdin: username>
<stdin: password>
<stdin: android_id>
{'SID': 'BAD_COOKIE', 'LSID': 'BAD_COOKIE', 'Token': ... }

Save your token somewhere, and get your application to use that instead of calling perform_master_login directly.

Here's an example of saving the token directly into your keyring:

(Python) https://gist.github.com/jkitching/57baead775f6773848d2e334cc950999 (Bash) https://gist.github.com/jkitching/1e6bab9e3fd935aca2169528cd7eb497

Can't get it to work without androidid. If only the script would pause and let me do the URL thing.

[Cris70]

Here's my solution for the moment:

gpsoauth_login.Dockerfile

FROM alpine:3.16
RUN apk upgrade --update-cache --available && \
    apk add python3 py3-pip && \
    pip install gpsoauth 'urllib3<=1.25.11'
RUN echo $'import gpsoauth; import json\n\
ret = gpsoauth.perform_master_login(input(), input(), input())\n\
print(json.dumps(ret))' > /glogin.py
ENTRYPOINT ["python3", "/glogin.py"]

Usage:

$ docker build -t gpsoauth_login -f gpsoauth_login.Dockerfile .
$ docker run -it --rm gpsoauth_login
<stdin: username>
<stdin: password>
<stdin: android_id>
{'SID': 'BAD_COOKIE', 'LSID': 'BAD_COOKIE', 'Token': ... }

Save your token somewhere, and get your application to use that instead of calling perform_master_login directly.

Here's an example of saving the token directly into your keyring:

(Python) https://gist.github.com/jkitching/57baead775f6773848d2e334cc950999 (Bash) https://gist.github.com/jkitching/1e6bab9e3fd935aca2169528cd7eb497

@jkitching your solution is brilliant, but it does not work (anymore?) unfortunately.

Instead of the token, I get this: {"Error": "NeedsBrowser", "Url": "https://accounts.google.com/signin/continue?sarp=1&scc=1&continue=https://accounts.google.com/o/android/auth?hl%3Den_us%26xoauth_display_name%3DAndroid%2BLogin%2BService%26source%3DAndroid%2BLogin&plt=VERY_LONG_TOKEN_REDACTED", "ErrorDetail": "To access your account, you must sign in on the web. Touch Next to start browser sign-in."}

I tried to make it work by opening the url with a browser, it seems to work as it asks me to confirm my login details, but then I get a static page telling me to "wait a moment", which doesn't lead anywhere (i.e. it waits forever).

Any suggestions?

[jkitching]

@Cris70 I can't seem to reproduce on my end... it still happily hands me a valid token.

Are you using 2FA? Perhaps you need to set up an App Password as suggested here? https://github.com/PiotrMachowski/Home-Assistant-custom-components-Google-Keep/issues/3

[Cris70]

@jkitching I am using 2FA and I am using an app password. No joy unfortuantely. I tried generating another app password, but the same happened.

p.s. If you have read my previous reply, that was meant for another thread. I am writing on multiple issues and I am messing things up a bit 😅

[KapJI]

I spent some time debugging it and the only difference in working and broken environments was ffdhe in supported groups in TLS Client Hello packet. But I haven't found a way to change that using SSLContext.

E.g. in Python 3.8 it works with OpenSSL 1.1.1o but in Python 3.11 which adds ffdhe with OpenSSL 3.0.12 it fails with BadAuthentication.

[simon-weber]

Nice find! That seems like something we should be able to configure.