ValueError: Insufficient data - ... bytes requested but only 18358 available on AWS Lambda/Amazon Linux

[robincsamuel]

I'm using a package, that uses asn1crypto, with chalice and everything works fine on my local machine (Mac), an AWS EC2 (Ubuntu) and AC2 (Amazon linux, x86_64).

However, an asn1crypto method (Asn1Value.load) is throwing an error, similar to https://github.com/wbond/asn1crypto/issues/85, when I execute as an AWS Lambda function.

Stack trace:

Traceback (most recent call last):
  File "/var/task/pyas2lib/as2.py", line 591, in parse
    self.enc_alg, decrypted_content = decrypt_message(
  File "/var/task/pyas2lib/cms.py", line 193, in decrypt_message
    cms_content = cms.ContentInfo.load(encrypted_data)
  File "/var/task/asn1crypto/core.py", line 230, in load
    value, _ = _parse_build(encoded_data, spec=spec, spec_params=kwargs, strict=strict)
  File "/var/task/asn1crypto/core.py", line 5668, in _parse_build
    info, new_pointer = _parse(encoded_data, encoded_len, pointer)
  File "/var/task/asn1crypto/parser.py", line 225, in _parse
    raise ValueError(_INSUFFICIENT_DATA_MESSAGE % (contents_end, data_len))
ValueError: Insufficient data - 1545633062365291768294061961431016176374627348951649321802178575314893183602912632065505301327437350584106059005008117218689175730975503838612273058029938831908239103060349018132643124869528847276954337830257591395451447158305993254068662164490556331379749698778432190 bytes requested but only 18358 available

Reference: https://github.com/abhishek-ram/pyas2-lib/issues/48

Any suggestions? Thanks :)

[MatthiasValvekens]

First thought: perhaps you're decrypting with the wrong key? This looks suspiciously like something that could happen when trying to treat garbage data as DER. If the cipher is unauthenticated and unpadded, using the wrong key wouldn't necessarily trigger an error in the decryption process, it'd just spit out meaningless bytes.

[robincsamuel]

@MatthiasValvekens I’ll verify the keys and share the findings.

But, I have the same code running on different servers and they all use the same keys (as I deployed all these for testing purpose); and it works fine on all other servers.

Also, I’m using a package, pyas2lib, and that package is using asn1crypto. The pyas2lib package throws an error if I pass the wrong key. I have passed wrong keys in the past and got a proper error like invalid signature.

[robincsamuel]

@MatthiasValvekens I've verified the keys & payload and it's exactly the same.

I connected my local application to the same database the lambda uses so that it loads the exact same keys and that worked fine. The payload was also the same. I've also logged the keys & payload from the lambda and that too looks fine.

The only difference I see is the platform, AWS Lambda.

[geitda]

Something is corrupting this data before it makes it to asn1crypto: the number of bytes requested (154563......432190) looks almost like ASN.1 data:

BF BD 28 11 06 09 2A EF BF BD 48 EF BF BD EF BF BD 0D 01 07 03 EF BF BD EF BF BD 28 02 30 EF BF
BD 27 EF BF BD 02 01 00 31 EF BF BD 02 EF BF BD 30 EF BF BD 02 EF BF BD 02 01 00 30 EF BF BD EF
BF BD 30 EF BF BD EF BF BD 31 0B 30 09 06 03 55 04 06 13 02 55 53 31 0B 30 09 06 03 55 04 08 0C
02 4D 4E 31 0B 30 09 06 03 55 04 07 0C 02 BE

Note all the repeated EF BF BD strings: this is UTF-8 for U+FFFD i.e. "REPLACEMENT CHARACTER" Looks like some non-ASCII bytes (highest bit set) are getting mangled. There are two un-mangled sections (towards the end) that are totally valid ASN.1:

31 0B 30 09 06 03 55 04 06 13 02 55 53 is

SET (1 elem)
  SEQUENCE (2 elem)
    OBJECT IDENTIFIER 2.5.4.6 countryName (X.520 DN component)
    PrintableString US

and 31 0B 30 09 06 03 55 04 08 0C 02 4D 4E is

SET (1 elem)
  SEQUENCE (2 elem)
    OBJECT IDENTIFIER 2.5.4.8 stateOrProvinceName (X.520 DN component)
    UTF8String MN

These two sections are all in ASCII range, thus they're un-mangled. So I'm pretty confident that asn1crypto itself is not to blame, but some odd bytes -> str -> bytes round tripping (with encoding errors set to use the replacement character) is going on in pyas2. I have no clue what AWS Lambda is doing differently than your local install, but this is at least a place to start.