ACME 2.2.0 query_registration raises TypeError: frozendict(payload='...', protected='...', signature='...') is not JSON serializable

[himiranov]

I use acme library directly in my Python application, and after upgrading to the last acme version I started getting error when calling query_registration method with existing registration. Account MUST use EAB (ExternalAccountBinding)

My operating system is (include version):

Ubuntu 20.04.1 LTS (64-bit)

I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):

I use acme library directly in my Python application:

Python version 3.10

acme==2.2.0
cryptography==39.0.0
josepy==1.13.0

I ran this command and it produced this output:

example.py content:

import os
from typing import Optional

from acme.client import ClientNetwork, ClientV2
from acme.messages import RegistrationResource, Directory
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from josepy import JWKRSA

SSL_COM_URL = 'https://acme.ssl.com/sslcom-dv-rsa'


def load_jwk_account_key(
    file_path: str
) -> Optional[JWKRSA]:
    private_key, private_key_pem = None, None
    if os.path.exists(file_path):
        with open(file_path, 'rb') as f:
            private_key_pem = f.read()

        if private_key_pem:
            private_key = serialization.load_pem_private_key(
                private_key_pem,
                password=None,
                backend=default_backend()
            )

    if not private_key:
        return None
    return JWKRSA(key=private_key)


def main():
    jwk_account_key = load_jwk_account_key('account.pem')
    if not jwk_account_key:
        print('Cant load jwk_account_key')
        return
    net = ClientNetwork(jwk_account_key, user_agent='python-acme')
    directory = Directory.from_json(net.get(SSL_COM_URL).json())
    client_acme = ClientV2(directory, net=net)

    registration_data = {
        "body":{
            "contact":[
                "mailto:[email protected]"
            ],
            "status":"valid",
            "termsOfServiceAgreed": True,
            "externalAccountBinding":{
                "payload":"...",
                "protected":"...",
                "signature":"..."
            }
        },
        "uri":"https://acme.ssl.com/ejbca/acme/sslcom-dv-rsa/acct/..."
    }

    regr = RegistrationResource.from_json(registration_data)
    # error happens here
    regr = client_acme.query_registration(regr)


main()

account.pem content

-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----

Certbot's behavior differed from what I expected because:

regr should be returned or unauthorized error should be raised

Here is a traceback log showing the issue:

Traceback (most recent call last):
  File "/home/erik/workspace/evo_ssl/evo_ssl/lib/example2.py", line 64, in <module>
    main()
  File "/home/erik/workspace/evo_ssl/evo_ssl/lib/example2.py", line 60, in main
    regr = client_acme.query_registration(regr)
  File "/home/erik/workspace/evo_ssl/venv/lib/python3.10/site-packages/acme/client.py", line 79, in query_registration
    self.net.account = self._get_v2_account(regr, True)
  File "/home/erik/workspace/evo_ssl/venv/lib/python3.10/site-packages/acme/client.py", line 109, in _get_v2_account
    response = self._post(self.directory['newAccount'], only_existing_reg)
  File "/home/erik/workspace/evo_ssl/venv/lib/python3.10/site-packages/acme/client.py", line 338, in _post
    return self.net.post(*args, **kwargs)
  File "/home/erik/workspace/evo_ssl/venv/lib/python3.10/site-packages/acme/client.py", line 711, in post
    return self._post_once(*args, **kwargs)
  File "/home/erik/workspace/evo_ssl/venv/lib/python3.10/site-packages/acme/client.py", line 721, in _post_once
    data = self._wrap_in_jws(obj, self._get_nonce(url, new_nonce_url), url)
  File "/home/erik/workspace/evo_ssl/venv/lib/python3.10/site-packages/acme/client.py", line 520, in _wrap_in_jws
    jobj = obj.json_dumps(indent=2).encode() if obj else b''
  File "/home/erik/workspace/evo_ssl/venv/lib/python3.10/site-packages/josepy/interfaces.py", line 186, in json_dumps
    return json.dumps(self, default=self.json_dump_default, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.10/json/encoder.py", line 439, in _iterencode
    yield from _iterencode(o, _current_indent_level)
  File "/usr/lib/python3.10/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/home/erik/workspace/evo_ssl/venv/lib/python3.10/site-packages/josepy/interfaces.py", line 213, in json_dump_default
    raise TypeError(repr(python_object) + ' is not JSON serializable')
TypeError: frozendict(payload='...', protected='...', signature='...') is not JSON serializable

Account.pem & registration.json

You can register free account on ssl.com or use my account that I can send you in private message.

[alexzorin]

Hi,

So, I think there is a legitimate bug here. acme.message.Registration loses the ability to be serialized back to JSON if it contains an EAB and goes through a deserialization→serialization cycle. 👍

A couple of questions:

1. In what version of acme did this code work? acme==1.32.0/josepy==1.13.0 seems to fail to perform this serialization as well.
2. Is EAB actually required for querying an account? RFC8555 only mentions it for newAccount requests where the actual binding of the EAB credentials occurs. Once an account is created, my understanding is that the EAB credentials are not needed again. I've posted a question about this FWIW.
3. If you really need this, you should be able to construct the objects as follows:

#!/usr/bin/env python
from acme.messages import ExternalAccountBinding, RegistrationResource, Registration
from cryptography.hazmat.primitives.asymmetric.rsa import generate_private_key
from josepy import JWKRSA

directory = {"newAccount": "http://example.com"} # actual ACME directory

key = JWKRSA(key=generate_private_key(65537, 2048))
eab = ExternalAccountBinding.from_data(
    directory=directory, hmac_key="foobar", kid="lala", account_public_key=key.public_key()
)
regrb = Registration.from_data(email="[email protected]", external_account_binding=eab)
regr = RegistrationResource(body=regrb, uri="http://url/to/acme/account/1")

print(regr.json_dumps()) # a.k.a query_registration

In your original program, I think you should be able to avoid the issue by just dropping EAB from the body, since it is likely not needed anyway for that query:

regr = regr.update(body=regr.body.update(external_account_binding=None))

or even re-setting the EAB field using ExternalAccountBinding.from_datalike in my example.

[himiranov]

Hi,

Thank you for the quick reply.

My application is quite old, so I haven't updated Python and ACME versions for a long time and I used:

Python 3.7

acme==1.3.0
josepy==1.3.0

I'm not really sure about EAB requirement for querying an account, but I can check it with my application or simply use Registration.from_data which I actually did and it worked for me. However I think RegistrationResource.from_json looks a bit better :slightly_smiling_face:

[github-actions[bot]]

We've made a lot of changes to Certbot since this issue was opened. If you still have this issue with an up-to-date version of Certbot, can you please add a comment letting us know? This helps us to better see what issues are still affecting our users. If there is no activity in the next 30 days, this issue will be automatically closed.

[github-actions[bot]]

This issue has been closed due to lack of activity, but if you think it should be reopened, please open a new issue with a link to this one and we'll take a look.