CCM mode doesn't check message length

[geitda]

The notes for CCM explain the tradeoff between nonce size and maximum message length, but the programming doesn't enforce it. See at https://github.com/Legrandin/pycryptodome/blob/a6b6ecd8959d155eeec46db664c6817359d50ac7/lib/Crypto/Cipher/_mode_ccm.py#L72 for the details. If n is the nonce length, and as CCM requires 7 <= n <= 13, then q is the counter length, specifically q = 15 - n. Message lengths m < 2^8q always work correctly and are interoperable. Message lengths 2^8q <= m < 2^8q+4-16 appear to encipher correctly as there are no errors, but are not interoperable as they are illegal (per the CCM spec) for any given q. Notably, these decode correctly with pycryptodome itself, as the error is "symmetric" for both encrypt_and_digest and decrypt_and_verify. Message lengths m > 2^8q+4-16 will raise an OverflowError as the underlying CTR mode cipher will wrap. The '+4' comes from fact that a single count in CTR mode can encipher 16 bytes (the block size), so the total number of bytes is 16 times larger (or 4 bits). And the -16 is because 16 bytes (one block) of the CTR keystream is needed for the tag, so the message maximum is one block less. Not that it makes much difference as this, again, is outside the CCM spec. I would expect a ValueError "Message is too long for given nonce length" or similar that should raise as soon as 2^8q bytes is "reached," whether that be from the pre-declared length being too large (that is, raise immediately on the AES.new call with the passed msg_len too big), or if one or more encrypt calls makes the total message length exceed the limit. You can test the overflow case easily

>>> from Crypto.Cipher import AES
>>> key = AES.get_random_bytes(16)
>>> nonce = AES.get_random_bytes(13) # that is, q == 2
>>> message = b'\x00' * (64 * 1024 * 16) # 64K blocks of 16 bytes
>>> ciphertext, tag = AES.new(key, AES.MODE_CCM, nonce=nonce).encrypt_and_digest(message)
Traceback (most recent call last):
  File "<pyshell#5>", line 1, in <module>
    ciphertext, tag = AES.new(key, AES.MODE_CCM, nonce=nonce).encrypt_and_digest(message)
  File "C:\Python311\Lib\site-packages\Crypto\Cipher\_mode_ccm.py", line 575, in encrypt_and_digest
    return self.encrypt(plaintext, output=output), self.digest()
  File "C:\Python311\Lib\site-packages\Crypto\Cipher\_mode_ccm.py", line 373, in encrypt
    return self._cipher.encrypt(plaintext, output=output)
  File "C:\Python311\Lib\site-packages\Crypto\Cipher\_mode_ctr.py", line 206, in encrypt
    raise OverflowError("The counter has wrapped around in"
OverflowError: The counter has wrapped around in CTR mode

No other correct implementation will accept the non-interoperable output from pycryptodome, so I don't think there's any need to test it directly.