pyasn1
pyasn1 copied to clipboard
etingof
DER decoder is missing components
In version 0.0.11a (yes, I know, old), this works nicely:
>>> import pyasn1.codec.der.decoder
>>> pyasn1.codec.der.decoder.decode('0o\xa0"\x06\n+\x06\x01\x04\x01\x827\x14\x02\x03\xa0\x14\x0c\x12000167512@00056427\xa0I\x06\x03U\x05\x05\xa0B\x16@2.16.528.1.1003.1.3.5.5.2-1-000167512-Z-00056427-01.010-00000000')
This yields the result:
(Sequence().setComponentByPosition(0, ObjectIdentifier('1.3.6.1.4.1.311.20.2.3')).setComponentByPosition(1, UTF8String('000167512@00056427')).setComponentByPosition(2, ObjectIdentifier('2.5.5.5')).setComponentByPosition(3, IA5String('2.16.528.1.1003.1.3.5.5.2-1-000167512-Z-00056427-01.010-00000000')), '')
Note that there are FOUR components in the sequence.
In versions 0.1.7 and 0.4.3, the exact same calls above yield the result:
[<SequenceOf value object at 0x7f23e8c9e110 componentType=None tagSet=<TagSet object at 0x7f23e8c98dd0 tags 0:32:16> subtypeSpec=<ConstraintsIntersection object at 0x7f23e8c76310> sizeSpec=<ConstraintsIntersection object at 0x7f23e8c76350> payload [<ObjectIdentifier value object at 0x7f23e8c9e090 tagSet <TagSet object at 0x7f23e8c98f10 tags 0:0:6-128:32:0> payload [1.3.6.1.4.1.311.20.2.3]>, <ObjectIdentifier value object at 0x7f23e8c9e0d0 tagSet <TagSet object at 0x7f23e8c98f50 tags 0:0:6-128:32:0> payload [2.5.5.5]>]>, '']
Notice that there are now only TWO components, both of the type ObjectIdentifier. The UTF8String type objects have disappeared from the result. I see no way of finding them in this result.
Is this a bug or am I missing something regarding the absent UTF8String objects in the result?
Sorry for not looking into this earlier!
I think this has something to do with either ambiguous or incorrect encoding. Do you know ASN.1 structure you are trying to decode into? Especially its tagging.
Without schema being given, the latest pyasn1 tries to interpret the inner components of the SEQUENCE as EXPLICITly tagged. While doing so it appears that those UTF8 strings are piggybacked inside the EXPLICIT tag container past OBJECT IDENTIFIER. This is something that DER does not allow IIRC thus UTF8 strings get dropped.
But with schema containing proper tags pyasn1 decoder may get back on track, that's why I am asking. ;-)
Here is how I think I figured what's going on (search for "lost"):
2018-07-26 10:44:40,889 pyasn1: running pyasn1 0.4.5, debug flags all
2018-07-26 10:44:40,891 pyasn1: debug category 'all' enabled
2018-07-26 10:44:40,892 pyasn1: decoder called at scope with state 0, working with up to 113 octets of substrate:
00000: 30 6F A0 22 06 0A 2B 06 01 04 01 82 37 14 02 03
00016: A0 14 0C 12 30 30 30 31 36 37 35 31 32 40 30 30
00032: 30 35 36 34 32 37 A0 49 06 03 55 05 05 A0 42 16
00048: 40 32 2E 31 36 2E 35 32 38 2E 31 2E 31 30 30 33
00064: 2E 31 2E 33 2E 35 2E 35 2E 32 2D 31 2D 30 30 30
00080: 31 36 37 35 31 32 2D 5A 2D 30 30 30 35 36 34 32
00096: 37 2D 30 31 2E 30 31 30 2D 30 30 30 30 30 30 30
00112: 30
2018-07-26 10:44:40,894 pyasn1: tag decoded into <TagSet object at 0x10d016898 tags 0:32:16>, decoding length
2018-07-26 10:44:40,895 pyasn1: value length decoded into 111, payload substrate is:
00000: A0 22 06 0A 2B 06 01 04 01 82 37 14 02 03 A0 14
00016: 0C 12 30 30 30 31 36 37 35 31 32 40 30 30 30 35
00032: 36 34 32 37 A0 49 06 03 55 05 05 A0 42 16 40 32
00048: 2E 31 36 2E 35 32 38 2E 31 2E 31 30 30 33 2E 31
00064: 2E 33 2E 35 2E 35 2E 32 2D 31 2D 30 30 30 31 36
00080: 37 35 31 32 2D 5A 2D 30 30 30 35 36 34 32 37 2D
00096: 30 31 2E 30 31 30 2D 30 30 30 30 30 30 30 30
2018-07-26 10:44:40,898 pyasn1: codec SequenceOrSequenceOfDecoder chosen by a built-in type, decoding value
2018-07-26 10:44:40,900 pyasn1: decoder called at scope NoneType with state 0, working with up to 111 octets of substrate:
00000: A0 22 06 0A 2B 06 01 04 01 82 37 14 02 03 A0 14
00016: 0C 12 30 30 30 31 36 37 35 31 32 40 30 30 30 35
00032: 36 34 32 37 A0 49 06 03 55 05 05 A0 42 16 40 32
00048: 2E 31 36 2E 35 32 38 2E 31 2E 31 30 30 33 2E 31
00064: 2E 33 2E 35 2E 35 2E 32 2D 31 2D 30 30 30 31 36
00080: 37 35 31 32 2D 5A 2D 30 30 30 35 36 34 32 37 2D
00096: 30 31 2E 30 31 30 2D 30 30 30 30 30 30 30 30
2018-07-26 10:44:40,901 pyasn1: tag decoded into <TagSet object at 0x10d016940 tags 128:32:0>, decoding length
2018-07-26 10:44:40,902 pyasn1: value length decoded into 34, payload substrate is:
00000: 06 0A 2B 06 01 04 01 82 37 14 02 03 A0 14 0C 12
00016: 30 30 30 31 36 37 35 31 32 40 30 30 30 35 36 34
00032: 32 37
2018-07-26 10:44:40,902 pyasn1: codec <none> chosen by a built-in type, decoding as explicit tag
2018-07-26 10:44:40,903 pyasn1: codec ExplicitTagDecoder chosen, decoding value
2018-07-26 10:44:40,904 pyasn1: decoder called at scope NoneType.? with state 0, working with up to 34 octets of substrate:
00000: 06 0A 2B 06 01 04 01 82 37 14 02 03 A0 14 0C 12
00016: 30 30 30 31 36 37 35 31 32 40 30 30 30 35 36 34
00032: 32 37
2018-07-26 10:44:40,905 pyasn1: tag decoded into <TagSet object at 0x10d016a20 tags 0:0:6-128:32:0>, decoding length
2018-07-26 10:44:40,906 pyasn1: value length decoded into 10, payload substrate is:
00000: 2B 06 01 04 01 82 37 14 02 03
2018-07-26 10:44:40,907 pyasn1: codec ObjectIdentifierDecoder chosen by a built-in type, decoding value
2018-07-26 10:44:40,909 pyasn1: codec ObjectIdentifierDecoder yields type ObjectIdentifier, value:
1.3.6.1.4.1.311.20.2.3
...remaining substrate is:
00000: A0 14 0C 12 30 30 30 31 36 37 35 31 32 40 30 30
00016: 30 35 36 34 32 37
2018-07-26 10:44:40,910 pyasn1: decoder left scope NoneType.?, call completed
2018-07-26 10:44:40,910 pyasn1: explicit tag container carries 22 octets of trailing payload (will be lost!):
00000: A0 14 0C 12 30 30 30 31 36 37 35 31 32 40 30 30
00016: 30 35 36 34 32 37
2018-07-26 10:44:40,911 pyasn1: codec ExplicitTagDecoder yields type ObjectIdentifier, value:
1.3.6.1.4.1.311.20.2.3
...remaining substrate is:
00000: A0 49 06 03 55 05 05 A0 42 16 40 32 2E 31 36 2E
00016: 35 32 38 2E 31 2E 31 30 30 33 2E 31 2E 33 2E 35
00032: 2E 35 2E 32 2D 31 2D 30 30 30 31 36 37 35 31 32
00048: 2D 5A 2D 30 30 30 35 36 34 32 37 2D 30 31 2E 30
00064: 31 30 2D 30 30 30 30 30 30 30 30
2018-07-26 10:44:40,911 pyasn1: decoder left scope NoneType, call completed
2018-07-26 10:44:40,912 pyasn1: decoder called at scope NoneType with state 0, working with up to 75 octets of substrate:
00000: A0 49 06 03 55 05 05 A0 42 16 40 32 2E 31 36 2E
00016: 35 32 38 2E 31 2E 31 30 30 33 2E 31 2E 33 2E 35
00032: 2E 35 2E 32 2D 31 2D 30 30 30 31 36 37 35 31 32
00048: 2D 5A 2D 30 30 30 35 36 34 32 37 2D 30 31 2E 30
00064: 31 30 2D 30 30 30 30 30 30 30 30
2018-07-26 10:44:40,912 pyasn1: tag decoded into <TagSet object at 0x10d016940 tags 128:32:0>, decoding length
2018-07-26 10:44:40,912 pyasn1: value length decoded into 73, payload substrate is:
00000: 06 03 55 05 05 A0 42 16 40 32 2E 31 36 2E 35 32
00016: 38 2E 31 2E 31 30 30 33 2E 31 2E 33 2E 35 2E 35
00032: 2E 32 2D 31 2D 30 30 30 31 36 37 35 31 32 2D 5A
00048: 2D 30 30 30 35 36 34 32 37 2D 30 31 2E 30 31 30
00064: 2D 30 30 30 30 30 30 30 30
2018-07-26 10:44:40,913 pyasn1: codec <none> chosen by a built-in type, decoding as explicit tag
2018-07-26 10:44:40,913 pyasn1: codec ExplicitTagDecoder chosen, decoding value
2018-07-26 10:44:40,913 pyasn1: decoder called at scope NoneType.? with state 0, working with up to 73 octets of substrate:
00000: 06 03 55 05 05 A0 42 16 40 32 2E 31 36 2E 35 32
00016: 38 2E 31 2E 31 30 30 33 2E 31 2E 33 2E 35 2E 35
00032: 2E 32 2D 31 2D 30 30 30 31 36 37 35 31 32 2D 5A
00048: 2D 30 30 30 35 36 34 32 37 2D 30 31 2E 30 31 30
00064: 2D 30 30 30 30 30 30 30 30
2018-07-26 10:44:40,914 pyasn1: tag decoded into <TagSet object at 0x10d016a90 tags 0:0:6-128:32:0>, decoding length
2018-07-26 10:44:40,914 pyasn1: value length decoded into 3, payload substrate is:
00000: 55 05 05
2018-07-26 10:44:40,914 pyasn1: codec ObjectIdentifierDecoder chosen by a built-in type, decoding value
2018-07-26 10:44:40,915 pyasn1: codec ObjectIdentifierDecoder yields type ObjectIdentifier, value:
2.5.5.5
...remaining substrate is:
00000: A0 42 16 40 32 2E 31 36 2E 35 32 38 2E 31 2E 31
00016: 30 30 33 2E 31 2E 33 2E 35 2E 35 2E 32 2D 31 2D
00032: 30 30 30 31 36 37 35 31 32 2D 5A 2D 30 30 30 35
00048: 36 34 32 37 2D 30 31 2E 30 31 30 2D 30 30 30 30
00064: 30 30 30 30
2018-07-26 10:44:40,915 pyasn1: decoder left scope NoneType.?, call completed
2018-07-26 10:44:40,915 pyasn1: explicit tag container carries 68 octets of trailing payload (will be lost!):
00000: A0 42 16 40 32 2E 31 36 2E 35 32 38 2E 31 2E 31
00016: 30 30 33 2E 31 2E 33 2E 35 2E 35 2E 32 2D 31 2D
00032: 30 30 30 31 36 37 35 31 32 2D 5A 2D 30 30 30 35
00048: 36 34 32 37 2D 30 31 2E 30 31 30 2D 30 30 30 30
00064: 30 30 30 30
2018-07-26 10:44:40,915 pyasn1: codec ExplicitTagDecoder yields type ObjectIdentifier, value:
2.5.5.5
...remaining substrate is: <none>
2018-07-26 10:44:40,916 pyasn1: decoder left scope NoneType, call completed
2018-07-26 10:44:40,917 pyasn1: codec SequenceOrSequenceOfDecoder yields type SequenceOf, value:
SequenceOf:
1.3.6.1.4.1.311.20.2.3 2.5.5.5
...remaining substrate is: <none>
2018-07-26 10:44:40,917 pyasn1: decoder left scope , call completed
SequenceOf:
1.3.6.1.4.1.311.20.2.3 2.5.5.5
edit: please note, this is for the encoder but I'm guessing there is some overlap in functioning. If not I'll make another issue.
I also noticed a similar problem with optional named values in some pyasn1_modules definitions which use optional parameters.
The Null operator (which is defined as an optional any type) disappears in DigestAlgorithmIdentifiers. It has to be there as it is defined as an Any type even though it is optional, if defined it is expected to be there.
An example with the SHA256 digest identifier, thoug any valid oid works of course:
from pyasn1_modules.rfc5652 import DigestAlgorithmIdentifier
from pyasn1.type import univ
from pyasn1.codec.der import encoder
digest_algorithm = DigestAlgorithmIdentifier()
digest_algorithm['algorithm'] = univ.ObjectIdentifier("2.16.840.1.101.3.4.2.1")
digest_algorithm['parameters'] = univ.Null()
encoder.encode(digest_algorithm)
Result is in version 0.4.4: b'0\x05\x06\x03*\x03\x04' Result is in version 0.2.3: b"0\x07\x06\x03*\x03\x04\x05\x00"
The 0.2.3 result is the only correct one.
So the optional parameter is missing.
Reply to self:
digest_algorithm['parameters'] = univ.Null('')
solves my problem