Calling dump on a SignedData object took 30 seconds

[peteris-zealid]

I tried really hard to make a reproducible example, but failed at that. While investigating I did write some interesting code.

The problem seems to be related to a specific certificate. In most cases the dump function works fine.

While investigating I noticed that the dump method was called around 200000 times. The recursion depth was not big it peaked at depth of 11 which is completely normal. I could not figure out from reading the code where exactly the other dump calls were coming from but I suspect it is from x509.py

I changed the dump code to cache the results it returns and that solved the problem. I understand that cacheing is not there for two reasons 1) the objects are mutable and dump can return different results 2) cacheing would create unneeded memory consumption where in most cases it is not needed.

Since there is (as far as I can tell) a trade off between memory consumption and performance I would suggest to have some sort of flag that would allow cacheing the result of the dump function.

Here is some code I wrote to try to reproduce the problem. It worked well but slight manual editing was needed to make it construct the python object. Only removing the parts that did not de-serialize was needed. So the python object was created, but the slow serialization was not reproduced. asn_to_py.patch reconstructed.py.txt

Here is the particular CMS. cms.txt

[joernheissler]

Hi, when running your reconstructed.py, it takes about 40ms to load the python code, 75ms to build ci and 0.04ms to dump it.

Also, the result is only 14151 bytes, your cms.txt has 15511 bytes. When loading+dumping cms.txt I get back the original result (also fast).

My system: Python 3.11.2 (Debian), asn1crypto from git.

Could you provide more info about your system (OS, Python version, asn1crypto version) and the needed steps to reproduce your issue? So far I'm not convinced that there is an issue.

[peteris-zealid]

Yes, I said I also failed to reproduce the issue locally in particular I cannot - in a reasonable timeframe - have the same setup as where the problem is, but through ad-hoc debugging I narrowed it down to asn1crypto dump function.

From what I read in the source code loading and dumping will have significantly different logic than creating an object and dumping. But creating would be a lot of manual labour that I am not willing to do. Note that reconstruct.py dump does not perfectly match the cms as given in cms.txt

Versions where the problem exists: Ubuntu 20.04.3 LTS asn1crypto Version: 1.5.1 pyHanko Version: 0.18.0.dev1

That being said I do not think the problem is with the os, because I did explicitly check that the dump function gets called too many times.

The asn1crypto is used through pyHanko.

I understand that this ticket is not very actionable as a bug report without clear steps to reproduce the problem. I was hoping that someone with reasonably good understanding of asn1crypto internals would magically know where the problem is. Since this is not the case you can close this one and I will open a new one when I have time to investigate in-depth.

Alternatively you can take this to be feature request and consider cacheing anyway.

[joernheissler]

dump already uses caching, the force parameter turns it off. I've had a quick look at all 9 different dump functions and I didn't see any issue with those. Might have missed something though.

Dump being called 200_000 times is very strange. You could record the class and optionally the ID of the object being dumped and then determine which class/object is dumped how often.

[MatthiasValvekens]

Seconding @joernheissler's PoV; both pyHanko and pyhanko-certvalidator actually make calls to .dump(), so IMHO it's more likely to be a piece of validation logic on that end going haywire.

Hypothesis: 200000 is a lot, but given that .dump() is defined only on a few universal classes with many subclasses in asn1crypto, I could imagine hitting that number by (re)encoding a particularly large CRL, for example (even so, the reencoding might not even be the bottleneck). The fact that this happens at all is probably a bug, though, or at least a bad fallback. PyHanko's revinfo fetcher is sometimes a bit too clever for its own good.

If you can find a way to reliably reproduce the slowness with the pyHanko API, feel free to start a discussion over there. I may be able to assist in narrowing down the root cause once you give me a way to reproduce the problem. (EDIT: if enabling retroactive_revinfo=True in the validation context solves the problem, then definitely open a discussion; that means it's a new manifestation of an old class of bugs)

TL;DR: asn1crypto is probably not the culprit.

[peteris-zealid]

I managed to reproduce locally and it seems the problem is not with pyHanko. But pyhanko will be needed to reproduce the problem.

Attaching zip with two files. reproduce_long_dump.zip

One very interesting thing I noticed is that when the signature_value variable is made shorter (by removing the * 16 multiplier) the whole thing magically becomes fast.

[peteris-zealid]

Also calling dump for the second time will take twice as long as as the first time.

[MatthiasValvekens]

Here's a simpler reproducer that sidesteps pyHanko (pardon the wall of text):

import binascii, time

from asn1crypto import tsp, cms
payload = "30823c9306092a864886f70d010702a0823c8430823c80020101310f300d06096086480165030402010500300b06092a864886f70d010701a08220b4308207453082052da00302010202147b4642045c3345a743376794cb831373767e9be0300d06092a864886f70d01010b05003081ab310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d31123010060355040a13095a65616c69442041423120301e060355040313175a65616c694420526f6f74204341203230323120535447311530130603550461130c5345353536393732343238383020170d3231313032323133353632305a180f32303531313032323133353632305a3081ab310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d31123010060355040a13095a65616c69442041423120301e060355040313175a65616c694420526f6f74204341203230323120535447311530130603550461130c53453535363937323432383830820222300d06092a864886f70d01010105000382020f003082020a0282020100cc4188a8ad07680aabc1dea474a46d0014a108b630bb66be6e6e3652952b5a6f7180987e9efa581f54a47cfdfa988227e59f88a305a726412a8084c55c947dee0da3bf56bf8ffb45dd438886cad6154d8fd3d591aad15b35492e136afb4f29ed1472455a5a55fa850151ef5a372c3cb6009da671126b4bb8b33a62ac74207a70675157e0dc40bf1e416ff6e65e19296d5229a7d06fe5db2e248a131f02f9e8e047611563c955dc18176746bad3a7f430af6bcf2f0a167ec218b444dd95f11ec8fd89e659ce2572c03488c65939c744378f2961412343d518ca68a36ebd90b8a7ce8fed8b60b354b77e47292723747aebae94178bfdb61b8cdb62e897745b9591fcee6036cf9fb79b4edf94d1f6c866b08d1d7b369894a4c18f7c123463d2ccc331e4eb6d2363c4ad0961abb6858f86c51d9b5fb0f48a4847331698251434bc4407601fbce0154b0a0c7620f9b4581fd5f3445278ac67372c6c8d7e74a37d7178e87b42b6873ad277911515fb072cbed6f7ee3f1d0eb72a18db004483f44887ea8dc7abe7d4d54d70249e9dfbcba5493759b4dc847445684d61b6ee870e4848c6d5812dddb776265e710176273400b974e455e6bc8c6bccc9ead7659cf3781054cac520464a6210909c01cb1422043a721424762336947f57850a6264cb386f5b0937e4a9c5f7a68e5bf768f052ae78217f5ecd7603ba812b8491744cee8790330203010001a382015b30820157300e0603551d0f0101ff040403020106300f0603551d130101ff040530030101ff308201130603551d200482010a30820106308184060704008bec4001023079302d06082b06010505070201162168747470733a2f2f7777772e7a65616c69642e636f6d2f7265706f7369746f7279304806082b06010505070202303c0c3a436572746966696361746520686173206265656e20697373756564206163636f7264696e6720746f205143502d6e2d7173636420706f6c696379307d060604008f7a01023073302d06082b06010505070201162168747470733a2f2f7777772e7a65616c69642e636f6d2f7265706f7369746f7279304206082b0601050507020230360c34436572746966696361746520686173206265656e20697373756564206163636f7264696e6720746f204e43502b20706f6c696379301d0603551d0e04160414a756e4c5d810ac17c2aab0fab8720173faf215c9300d06092a864886f70d01010b05000382020100bb1e8b90a1023567a2189bc646bf71ffd61b0d2c5a5cddb5baaf252ee79b21a1529c1289818236bc12cc8a061fd354ed1f997a8063a217ac0b5caf820b2a5c6247e2ec1b6dca7b0355b00caf0920fba0281cec2365a50597c3fa52d246bd7c6bc4d4766bb4756897ec290037a5c115a2d71788a472557f0175fc6a64c1eb8afb77101e08f9820ae2f2faef22f3892f5d08e6ba4b4e732e8ad58d4fee65bb21ce797e35bc161122ebf0c39bb2d949423520375cff60255504ec8c1bec7941c244452f2eec13c47a47d43c7b2eb4700377db508104cf582ef136557e4a0a26b365a32a8484b5138347aff12432231353adedcace5ffe7abcc3472d0a9434428e93971ea666d3a5836750237855e4fcc7a12603f3bfcfeaeb7289d81c7f58e6c4801e02f9b8fe617de476a578095649925a4eb9c9ceac3c692032adf8d9e2763c129d66c9801e4fac28ac25aa2478a582932741ddb415862c0c60419fbe1f17a9e84465dc2cfcf2b7f79b0c71159297efeaf2904d2d2d2fb0ba8a61d21fe5047da50e2fc5a2b9bd3a4a1dd32c1a4956e738f8a9ae9b9452aed96115d5c4652b807f434938cd6cb94023d131790388a3dbdb7376011294290c7edd7b176c27cfb783fbd7fdc0dcbac11abc546351e8bd8c9a7039894a081daac0878678e84cccdaba6d929f6df554d515f69924923ba00864d780b55c8412cb98375966ac77ab1ff7308207db308205c3a00302010202145532fd05749577c52736ee8170af75ffd858ad8c300d06092a864886f70d01010c05003081ab310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d31123010060355040a13095a65616c69442041423120301e060355040313175a65616c694420526f6f74204341203230323120535447311530130603550461130c534535353639373234323838301e170d3232303332313133313232315a170d3337303332313133313232315a3081ae310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312330210603550403131a5a65616c69442049737375696e6720434120323032312053544730820222300d06092a864886f70d01010105000382020f003082020a0282020100ddedcd7dffd28b928c74c91fb6c94ba3d31db0662f2ec67c9f7199796b965a0267be2822929295c8f7d2d3a5df4068f62481b5e567ead47cf95fccffe8d42c6d4fc96d4cf6a8f846a31d6f9482a4c7ef45053319479eba5163d408339deccf53bc2ffb12cb92cd7fa64eaa645d60d35f94a2fc3c44aa9e519811c02ac6f5ad43984ea7880a308291c4f1c0c7cde77c0074b079ba0a93422919c8465065a6fc7c2aa5d45067c18c95c4fe9d85a36b32e40ad4750ab3b9ddbc719becb4c782dfa389ea97bbb1125a837f31deb229b47d0d99e59c8bc0720ae7bc217aa6f0b06a00f0019180adbf67487c97e8d02b78ba6dbcfad1b0f30135270d70865d62e509038d633ae2f1cfae179b44db4c1c0c1de2f5b4fba0b1a166994cc040973e1d7158c8735c87c72edcfee4387451307df16658a64eb65ee0227fb813118b512a4931d8a4276090584bfaacc27e3c69127ae532b3b0b66443844bec30cda4e243a9f78b9854c90fdc1eabfe1d6b7ddb52ef6109cef7e06fe66741784d3f4ede0984a2631831d8eac4b22d6f3bbed95cc681354ad19194c2c2980e59da5f4ae527290aa1cb7159c99681e51a7fc59ad483675bbc92faf8bd9996507e22cbb7844ec9bce0ead1164100fba20d9d5101ac673e97e88c8feafa2dd8eef706494328dfa4426da0288ac3e148cf8db5ba8f7aa8aba7e5877218e627c8916597977ce8f823210203010001a38201f0308201ec300e0603551d0f0101ff04040302010630120603551d130101ff040830060101ff020100303e06082b0601050507010104323030302e06082b06010505073001862268747470733a2f2f6f6373702d74657374322e7374672d7a65616c69642e636f6d2f301f0603551d23041830168014a756e4c5d810ac17c2aab0fab8720173faf215c9308201130603551d200482010a30820106308184060704008bec4001023079302d06082b06010505070201162168747470733a2f2f7777772e7a65616c69642e636f6d2f7265706f7369746f7279304806082b06010505070202303c0c3a436572746966696361746520686173206265656e20697373756564206163636f7264696e6720746f205143502d6e2d7173636420706f6c696379307d060604008f7a01023073302d06082b06010505070201162168747470733a2f2f7777772e7a65616c69642e636f6d2f7265706f7369746f7279304206082b0601050507020230360c34436572746966696361746520686173206265656e20697373756564206163636f7264696e6720746f204e43502b20706f6c696379302f06082b06010505070103042330213008060604008e4601013008060604008e460104300b060604008e46010302010a301d0603551d0e041604148aa3ec02733e9d1b5142efd2b3968536d67b213e300d06092a864886f70d01010c0500038202010084924eeba2ff0e88914b5efeb75ae53efa3ddfb02da4ff6e3bb637af19517451a13ab55eb257c6b9e667c8db76b82b269b84961d356d4b1f4a3074405fb62f18735ca20c5e77990c27867633614a9718bf784e7b265e8e12e4772649da54eaa4742b4025522d99af8d344561188fa5cea6f1c23467b3924dab05617e873503f37a115c9dc408a887b8af7a46745bf24807122683dee4b3ae9366470c0be7e279bc8a4319cba2333ef1b2478fbaaa58d4d885551b97e9108b4ec033e1924654c1023688f34b2e4d4407e728d5ad2aa51fe1f92976af8a3870bfa5b3a721715d144fefd915a56d694f5898a991e8fed0f6a9e35d327df4eea287d58d87707d566967e4fa53364a20d26514d101ea2ac7b2dbbdb8c7545e75830fef2c4ca15c9aa3abf6eb5e649c92fbd9a2f63c39d80bd81237ce57dbf68af10d3657a73c398bba270ef69e971ca32dd6fb0ff4b19767003515e345285584d0b30b2c84cd112c1859eb17cccad71af4cd5ed8c07051d06451cc6e9e187cc395b908972394668e9488326a80fa25c01dc85801844b1ad2b9aa2015629b539b03f4ad45ef5042d149ce6e4e49f7dd1d0d744a9fcf983eea0d134085579eecc84384214b6566d16ff691002985ff60660d2fc839692e5c33cdcf0e1b6dffb931d99cc85cc4999d549d7f4d9d7483c237494703c659854dd0a8fbaa1ad77b30246f1f6b59181735e861308208c2308206aaa00302010202143785006c9815a67bd4c87d6f4b62460666efcc3b300d06092a864886f70d01010b05003081ae310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312330210603550403131a5a65616c69442049737375696e67204341203230323120535447301e170d3233303332383131333031305a170d3235303332383131333031305a307f311a301806035504051311504e4f4c542d3338343132303931303631310b3009060355040613024c54311330110603550404130a4d75737465726d616e6e3110300e060355042a1307486172746d7574312d302b06035504031324486172746d7574204d75737465726d616e6e20504e4f4c542d333834313230393130363130820222300d06092a864886f70d01010105000382020f003082020a0282020100abd1ed654eaf7980cd5b7fc88ac01d7343a9f7ac232f0f9dbdcc60c30e7364430713ed5e363fac08d2f4ac8e6bb1a28fbcc401bd0872d921c72675afab4d19a9dc7880535050b5026fb93b264523d1287dd41e2cdbb540294e24e1dc2d9e4a706d370a9089734864ddf4334026bfc4fe7dbb10bb15bd73dd82a606ca75ecb67f71d01bc24bd4afbabdc83ceabf83bb24806c457711d120bcb67fcba0fbc9fb3e2735697ffdb32ba4b6a833858ed2a40e05986b6cc7f8e3fc5aa398338936efdde57833c71ca80987ef2be096070ec48039d26c71751d8c74e3e4810d34ff28b530d22a8cf798f64fddff2316410284813047562abb589d202644da4075c4db2c59b9ea110495cbef89216929fded4905dfc8648ae07c3143e00be10e750beaf72c6ab4602ddf8c3ee778110f69006cb3382c79dafc862ae039a51777773f40c9c63b24b02563b99318d4b6fbed6c27e069bfd3b712987bb6e33138b3d9c331f2c3b1a16bd6855bbbab1f542c8273903d9ee8bc68a651c2c907e3799a9277d4c847731f5d6c178bb7136968f517332d37ab67b9bc2bd8bbfc18b621bff45b02ed1908689e68826433fcf6217c4f1759516703dc45b6aa008b01f8eafe7de68d4705595b1cb963038c1c654f09163b34e01a5ce180687ca2128517f69e383e60b67094d8f035d63eac634e18a53f6ba40618084628280b43093bb98d1a1fb693770203010001a382030430820300300e0603551d0f0101ff040403020640300c0603551d130101ff0402300030818806082b06010505070101047c307a302e06082b06010505073001862268747470733a2f2f6f6373702d74657374322e7374672d7a65616c69642e636f6d2f304806082b06010505073002863c68747470733a2f2f7777772e7374672d7a65616c69642e636f6d2f7265706f7369746f72792f5a65616c69445f526f6f745f43412e6465722e637274301f0603551d230418301680148aa3ec02733e9d1b5142efd2b3968536d67b213e308201130603551d200482010a30820106308184060704008bec4001023079302d06082b06010505070201162168747470733a2f2f7777772e7a65616c69642e636f6d2f7265706f7369746f7279304806082b06010505070202303c0c3a436572746966696361746520686173206265656e20697373756564206163636f7264696e6720746f205143502d6e2d7173636420706f6c696379307d060604008f7a01023073302d06082b06010505070201162168747470733a2f2f7777772e7a65616c69642e636f6d2f7265706f7369746f7279304206082b0601050507020230360c34436572746966696361746520686173206265656e20697373756564206163636f7264696e6720746f204e43502b20706f6c6963793081fd06082b060105050701030481f03081ed3008060604008e460101306006082b06010505070b023054060704008bec49010130498647687474703a2f2f6f69642d696e666f2e636f6d2f6367692d62696e2f646973706c61793f6f69643d302e342e302e3139343132312e312e3126616374696f6e3d646973706c61793060060604008e46010530563054164e68747470733a2f2f7777772e7a65616c69642e636f6d2f68756266732f5a65616c6944253230504b49253230446973636c6f7375726525323053746174656d656e7425323076253230332e7064661302454e3013060604008e4601063009060704008e460106013008060604008e460104301d0603551d0e04160414f9cbf7e2834ec073c2c1bf957c44e53babaecb1b300d06092a864886f70d01010b050003820201003c5abebb2503fac5546a261ae2bde0f3347f3d0d8c974434055fb3a3184f60d6527c0b0bf9819084f495dd0ca74d04eed1a86cb6dd1552534c1f60172c5290984dc60751c2cd12526056f67e19a3f5e8dddd6b084f11f45bf034033957ab72b5b10b95f316f4202e7b0a7cfdb8f43abd9fa432c2721b258c08057b4f1bca32ee366ef3219ed2c9be4cd75a2480f395685cfbc4882b6e1fc2c586f0a0e1499cb3e0e8a77860af1e283cdf362e62066852a44564d9549d57ce07f554af0fed8ff943f1809ce804d78e3311755db2ff350b00e9b49cff86ba8f049bccd71d3d32e65c56a320405825982e46d17e62053e673e674abda69fd7eabe60284fc873acdf25461b47c7b184a7d5673a626dd3405bc88eade3488d72b5c8cb248f19afcb5474175192997f35884663ec1a801100caf4cfdcb3fa2094673f40bdfd26df59b29c221f93534da4a75c1b795386229cb0710d72553080e0ae927a1d057b36b372472770ddee04e36e9a9be12722927a90f3548aacab6a4ee186bda24043b5df177f82e95a81f96c71e98afac5ce6ab9e386844960832da4448f2a0f85acf70658194af90150f4653cb564dea2bc88bac592a921a001f675adae6c6cb30b89a39dce4c74a5e9be3ce252bbf219f9482dbc0e6e9b7b3dd8c0549e2b6dfc9b91964301b1967bdc009dcfdc1bb34d078e26b4e350fea12d9345086c245e675d35733e308208c2308206aaa00302010202143785006c9815a67bd4c87d6f4b62460666efcc3b300d06092a864886f70d01010b05003081ae310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312330210603550403131a5a65616c69442049737375696e67204341203230323120535447301e170d3233303332383131333031305a170d3235303332383131333031305a307f311a301806035504051311504e4f4c542d3338343132303931303631310b3009060355040613024c54311330110603550404130a4d75737465726d616e6e3110300e060355042a1307486172746d7574312d302b06035504031324486172746d7574204d75737465726d616e6e20504e4f4c542d333834313230393130363130820222300d06092a864886f70d01010105000382020f003082020a0282020100abd1ed654eaf7980cd5b7fc88ac01d7343a9f7ac232f0f9dbdcc60c30e7364430713ed5e363fac08d2f4ac8e6bb1a28fbcc401bd0872d921c72675afab4d19a9dc7880535050b5026fb93b264523d1287dd41e2cdbb540294e24e1dc2d9e4a706d370a9089734864ddf4334026bfc4fe7dbb10bb15bd73dd82a606ca75ecb67f71d01bc24bd4afbabdc83ceabf83bb24806c457711d120bcb67fcba0fbc9fb3e2735697ffdb32ba4b6a833858ed2a40e05986b6cc7f8e3fc5aa398338936efdde57833c71ca80987ef2be096070ec48039d26c71751d8c74e3e4810d34ff28b530d22a8cf798f64fddff2316410284813047562abb589d202644da4075c4db2c59b9ea110495cbef89216929fded4905dfc8648ae07c3143e00be10e750beaf72c6ab4602ddf8c3ee778110f69006cb3382c79dafc862ae039a51777773f40c9c63b24b02563b99318d4b6fbed6c27e069bfd3b712987bb6e33138b3d9c331f2c3b1a16bd6855bbbab1f542c8273903d9ee8bc68a651c2c907e3799a9277d4c847731f5d6c178bb7136968f517332d37ab67b9bc2bd8bbfc18b621bff45b02ed1908689e68826433fcf6217c4f1759516703dc45b6aa008b01f8eafe7de68d4705595b1cb963038c1c654f09163b34e01a5ce180687ca2128517f69e383e60b67094d8f035d63eac634e18a53f6ba40618084628280b43093bb98d1a1fb693770203010001a382030430820300300e0603551d0f0101ff040403020640300c0603551d130101ff0402300030818806082b06010505070101047c307a302e06082b06010505073001862268747470733a2f2f6f6373702d74657374322e7374672d7a65616c69642e636f6d2f304806082b06010505073002863c68747470733a2f2f7777772e7374672d7a65616c69642e636f6d2f7265706f7369746f72792f5a65616c69445f526f6f745f43412e6465722e637274301f0603551d230418301680148aa3ec02733e9d1b5142efd2b3968536d67b213e308201130603551d200482010a30820106308184060704008bec4001023079302d06082b06010505070201162168747470733a2f2f7777772e7a65616c69642e636f6d2f7265706f7369746f7279304806082b06010505070202303c0c3a436572746966696361746520686173206265656e20697373756564206163636f7264696e6720746f205143502d6e2d7173636420706f6c696379307d060604008f7a01023073302d06082b06010505070201162168747470733a2f2f7777772e7a65616c69642e636f6d2f7265706f7369746f7279304206082b0601050507020230360c34436572746966696361746520686173206265656e20697373756564206163636f7264696e6720746f204e43502b20706f6c6963793081fd06082b060105050701030481f03081ed3008060604008e460101306006082b06010505070b023054060704008bec49010130498647687474703a2f2f6f69642d696e666f2e636f6d2f6367692d62696e2f646973706c61793f6f69643d302e342e302e3139343132312e312e3126616374696f6e3d646973706c61793060060604008e46010530563054164e68747470733a2f2f7777772e7a65616c69642e636f6d2f68756266732f5a65616c6944253230504b49253230446973636c6f7375726525323053746174656d656e7425323076253230332e7064661302454e3013060604008e4601063009060704008e460106013008060604008e460104301d0603551d0e04160414f9cbf7e2834ec073c2c1bf957c44e53babaecb1b300d06092a864886f70d01010b050003820201003c5abebb2503fac5546a261ae2bde0f3347f3d0d8c974434055fb3a3184f60d6527c0b0bf9819084f495dd0ca74d04eed1a86cb6dd1552534c1f60172c5290984dc60751c2cd12526056f67e19a3f5e8dddd6b084f11f45bf034033957ab72b5b10b95f316f4202e7b0a7cfdb8f43abd9fa432c2721b258c08057b4f1bca32ee366ef3219ed2c9be4cd75a2480f395685cfbc4882b6e1fc2c586f0a0e1499cb3e0e8a77860af1e283cdf362e62066852a44564d9549d57ce07f554af0fed8ff943f1809ce804d78e3311755db2ff350b00e9b49cff86ba8f049bccd71d3d32e65c56a320405825982e46d17e62053e673e674abda69fd7eabe60284fc873acdf25461b47c7b184a7d5673a626dd3405bc88eade3488d72b5c8cb248f19afcb5474175192997f35884663ec1a801100caf4cfdcb3fa2094673f40bdfd26df59b29c221f93534da4a75c1b795386229cb0710d72553080e0ae927a1d057b36b372472770ddee04e36e9a9be12722927a90f3548aacab6a4ee186bda24043b5df177f82e95a81f96c71e98afac5ce6ab9e386844960832da4448f2a0f85acf70658194af90150f4653cb564dea2bc88bac592a921a001f675adae6c6cb30b89a39dce4c74a5e9be3ce252bbf219f9482dbc0e6e9b7b3dd8c0549e2b6dfc9b91964301b1967bdc009dcfdc1bb34d078e26b4e350fea12d9345086c245e675d35733e31821ba330821b9f0201013081c73081ae310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312330210603550403131a5a65616c69442049737375696e6720434120323032312053544702143785006c9815a67bd4c87d6f4b62460666efcc3b300d06096086480165030402010500a082015a301806092a864886f70d010903310b06092a864886f70d010701302f06092a864886f70d0109043122042061616161616161616262626262626262636363636363636364646464646464643082010b060b2a864886f70d010910022f3181fb3081f83081f53081f20420b3461312b5491d545107deceed6b7df3c21d97b3faec3236611c124fa560684e3081cd3081b4a481b13081ae310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312330210603550403131a5a65616c69442049737375696e6720434120323032312053544702143785006c9815a67bd4c87d6f4b62460666efcc3b300d06092a864886f70d01010b0500048202006161616161616161626262626262626263636363636363636464646464646464616161616161616162626262626262626363636363636363646464646464646461616161616161616262626262626262636363636363636364646464646464646161616161616161626262626262626263636363636363636464646464646464616161616161616162626262626262626363636363636363646464646464646461616161616161616262626262626262636363636363636364646464646464646161616161616161626262626262626263636363636363636464646464646464616161616161616162626262626262626363636363636363646464646464646461616161616161616262626262626262636363636363636364646464646464646161616161616161626262626262626263636363636363636464646464646464616161616161616162626262626262626363636363636363646464646464646461616161616161616262626262626262636363636363636364646464646464646161616161616161626262626262626263636363636363636464646464646464616161616161616162626262626262626363636363636363646464646464646461616161616161616262626262626262636363636363636364646464646464646161616161616161626262626262626263636363636363636464646464646464a182174e3082174a060b2a864886f70d010910020e318217393082173506092a864886f70d010702a082172630821722020103310f300d0609608648016503040201050030820126060b2a864886f70d0109100104a0820115048201113082010d02010106062a03040506073031300d060960864801650304020105000420e055e588dca83c643b0e2f2315acac58259aeec8596a6d52ac3ccf603bee90a302070098f7e102c875180f32303233303431393131333331345a020801e8ad745d2238f9a081a8a481a53081a2310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142311730150603550403130e5a65616c69442054534120434e54a0821281308205a43082038ca003020102021473be5b5decc753d202dc3596ccf2b5402d6b701b300d06092a864886f70d01010b05003081b2310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312730250603550403131e5a65616c694420416476616e6365642049737375696e6720434120434e54301e170d3231313230333132343632305a170d3236313230333132343632305a3081a2310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142311730150603550403130e5a65616c69442054534120434e5430820122300d06092a864886f70d01010105000382010f003082010a02820101008476ed0e8d56208a8516804c3b33f8fe221da7106b2261037e1a83b267bed700a6dbb03c12048916a4a13251d6950a44b3cb79471147494cebcf138b18c8ec7fb4b55cf95663c3e45265f04c592eb85e3395465672530ef1ee6448b557d4abd180609cdcf886fe75e42d13289b2e617307e8f97a8b0f391f41826ffbd3572af0a9eb91fbc11fde869810d37bc58f3de0cd2b065a005f7e9724971587ae738cab5d39d39c7e3563957fa0ae7c3731d3a575100bd7d0672662d26cc76474d6b91d206f67ab532e135c3c0535e067f0dac7f90ceae9bd1be614d8435218ba83925f3c9878791bb05355b6204d5fb45626220fdcc764e89c19bdf4df3bca5f4a66430203010001a381bf3081bc30160603551d250101ff040c300a06082b06010505070308300e0603551d0f0101ff0404030206c0300c0603551d130101ff04023000304406082b0601050507010104383036303406082b06010505073002862868747470733a2f2f6f6373702d7369676e696e672d746573742e7374672d7a65616c69642e636f6d301f0603551d23041830168014954e553afd381fd7207026396ba0b93ee460411c301d0603551d0e041604144a1e92691b81481942561c32b0e628dcce732025300d06092a864886f70d01010b05000382020100858e4d76f025215aeda86b0f6d0cb5673fbbada8d6a919b6f969b435ab0b235dc99770b62b2591bd564239155135f0642294ca2de50b7c27856097a3c76add41575e23bb97e366b6a1308dd3a5d089f09781a8a161e4a7b5105e317f0009bf84986f210b4c88c16cdd9537f2f3ef6e34642a7941e04f356addc37b2975e380b710740187e9c5a4fb49c281b96910ff3be21b51542068d4a80f32cc39cf14faa8d08e65a407a2f3dbdc750602a9fcd00990b12aa8773ae37a1dea30a2d71fd239583833da233c34d977fe7d4a7a50505410b7741dc621e10ae81ed4d7efbba5ca5a7ec8e55ba76264fefb03c53c199580ed090f5c21b47e4d974d5e921b48d3cb28c0817d8fb24de2104714ec4b7d0b7160463650868c95a19ee3dc9574b8b17c18188ba5481086e272bdb6d0be5702edce9ccec6207daf38fb0007707a073e8780e29be9ffa01216ae6330d39d35ebd21f122181346236ce5fd539ea4f97627b909ea99bb8d25198130f49bfcac9d12f59e9c0ba8a11e6c376cc39aba5a7ff8c671f02f2848d26c84d6fa399b3069bea489b7d4a8ecd490d42acf8b28a9ae53489a17e9981f00d48a30b90b8f7f97a3d2470a74c2f2953ace8e267e79f0f1d1b1b1495f106011b28e8655bd1131b3b13ea9f93ab8577f450a5ea54ac485dc319aae0dca3eea09281b140a569f25627adc7f18b8419045b809b0c74359b62040e3082069f30820487a003020102021423de92e0db20dc42cbe801614bd7269c01b4ab10300d06092a864886f70d01010c05003081af310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d31123010060355040a13095a65616c6944204142312430220603550403131b5a65616c694420416476616e63656420526f6f7420434120434e54311530130603550461130c534535353639373234323838301e170d3231313230333037323235325a170d3336313230333037323235325a3081b2310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312730250603550403131e5a65616c694420416476616e6365642049737375696e6720434120434e5430820222300d06092a864886f70d01010105000382020f003082020a0282020100aa0b96cf1fc146c60f07a9d0f92ba41c6d37ccadc2e4c82c6edafac0aff77253ae7a64797d8854106f2034d04340d2c313bca467f2fb5ecd8866e87f0cde7e4b2d4e191e2b4578687b91c989586ee6372167c87430f7baebe14582a4a90e1e5fc9b84f0dea67bf6c7c3d4354d1b2ddd9b4d5dcb8d3bb3caa8918dcb178ef4c24b422ec6eef0c5b9e4d1ce1aaeef7052847d3f20628f3b38006bb09e6a4a9073ae10a57918728cf96f936ab1d77105d576c6260db47d0fd19fb2e73d7af6532f611c15e8e47eeeaebe8ab4fb2b883f197db33ff36bf64c384ca285554b3f6e2f6e230a76a937afbf73c24fe500d30fe0fdf01aaa792d14afcd614572613c818dcdccf0a9c75372282671c37b2920a413c7678e6de2d30390c92b29a495cb4a39788d9b52e0560752b9084063e64d6208a2f916d1920505d976797f808db1e22b6291aad2a91ff73629e9a4c49bfcc02557cf26df2b453620ae0eb6890b0a13f256a0dcf2702fc2a14e8d5e7661c004fa0f5accffb68baddda14ff7113de6c22e365ada33bc6ba838f5cb05501afda62799286d708b8672c5d6ef3be952c0023c708e53ac1bfaa17188ae4c91fdfcdfa8946e94f1ea6f021200f22683b2c5947ea67ff3a6f06bed0649172d018583045c172a7e7fa1be4c76d9fd0f837ef8528d31fde636ebbaa53f4186f1ed8414eeffcb0271b405b3ca52deb3ee4a96ae25bb10203010001a381ad3081aa300e0603551d0f0101ff04040302010630120603551d130101ff040830060101ff020100304406082b0601050507010104383036303406082b06010505073001862868747470733a2f2f6f6373702d7369676e696e672d746573742e7374672d7a65616c69642e636f6d301f0603551d230418301680141462f0388199e34abf9bf17cd1ddc20f3f750d0a301d0603551d0e04160414954e553afd381fd7207026396ba0b93ee460411c300d06092a864886f70d01010c05000382020100783b5872b16d1db4e92e750e7fa2bb403ee5b28d567c7b1d013a23b0ecaf8b4c279fdefc7d51f9bd8ad89d486a48f309525b4097a39f0c049d1f5ed29eeaf79ef34ba7135c3da981b7b6b5a75c27b7611e21cd8a0c6416ac38afe299db0698bcf5fdd8355a13cf6623fc22d9a9353c516b934874143178ce844bc1cd3403bbc72ed2e6d0e910e358450f79717f9086b5c18ca0525522aae354d491140ad85c575f94fc1a48500e60530f371d7acb54bb5bb16b203e74548398102d541e03ffbd1136a7caf61264559749b78fbc0d27691f35cce22e4d8e13ec9050decf970e0d67a3d6d1bbd3b2a68b238c9da4d68350e66368c5874a02ef1f8e91d293a10373929f78a55ba5d5e00710e7b7cb3e7dbeb1c41bbc77be8921ee18fc5599f09567d11b68431fb2ac0e0567cbf260c651ba843ebf61dfdafa7ab16ddff68273b4a5348303fbd7c42c3754dd8a11a96239a57559980b75572f31c93bd2ede19194da08e145b69aeb35877c6ca348bcdb980a6e1cc4e8d315461bb286b4c01958b53195e7b28f31932281e01594690ff21a69b1c5afed22b32f034df66b58e7e2a598cfabe3798228c7642f3ac51b3873a57781224bc7b9f603fef353b0aa50d92f6b37ceffe53a8c97ff81aca459c6b31f9d5dbea1075fb27560d97a4fb13609be6e03b400272ece29993bdc4b7858180196a191ac50b943ff56603217a4c5ffd987308206323082041aa003020102021469035111ca3231192d72b63eaffff792c259dedb300d06092a864886f70d01010b05003081af310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d31123010060355040a13095a65616c6944204142312430220603550403131b5a65616c694420416476616e63656420526f6f7420434120434e54311530130603550461130c5345353536393732343238383020170d3231313230313134323934355a180f32303531313230313134323934355a3081af310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d31123010060355040a13095a65616c6944204142312430220603550403131b5a65616c694420416476616e63656420526f6f7420434120434e54311530130603550461130c53453535363937323432383830820222300d06092a864886f70d01010105000382020f003082020a0282020100a9ebca9ddcd703482fe9ea2033e571ae9909e3f6d40d69798a6e5652b8be5cf62f0ce6806411e343c1052c6d314c988f1573e06866d8b2b1b94b9b43856d39be414d8b1cbe7f7cf3b9f21503a16d2e8761f4f3440c521b46ace03fc510c1ac175d62d91a2085f4f89f52798fb7876aa7b7d5876c589c97515f032de9a018a72293b8c0a912971b1ff96d0a4f27e456c5d79d0e00443e9f426db80880a6ce0b04cfd5f2b6d2aca956ec1814ce6d30acac5746350d91e408d674272a76edc8bf8423a27afc4857843a76218fca57345ac0e603b155c78889fcaea4b51354a21fdee27409fc4cac7f2b13fb513dda01d3499c46102caee6ada779e03fec0b772991c43271acdadbae87d9ebd283d0ed3b39a7800cf28cc406df7fbb6b492ea0f89d2270fac18e15abaaa0155625ae31a17d542535fec910765db9e8ce699b3acccc7ee7fea1fdca3871ac4e1e477c9523b8b28cfa5e6572a3def19aa0e56ba55bfadcad9ec50507f183c66ec3f0cde3c3cafa95145bdeec514de5e7a2f53947c7a0cce67dbce79047ce6afe30fa8330213fe4211b81bc683388326a288bc0468284f0a5b06308a15dcdc97749beabff8c688b1ae77320cc2a279d78a15f4e11bf312fff7769db83e270680d6055ec080134b80fcbc4f2b45a5f496c38a1029f4804618d4f6d6d3a6187589df59e957c78082d791c3bc22946d4c082acfa006b6eeb0203010001a3423040300e0603551d0f0101ff040403020106300f0603551d130101ff040530030101ff301d0603551d0e041604141462f0388199e34abf9bf17cd1ddc20f3f750d0a300d06092a864886f70d01010b050003820201009f557540358b32f20cc7c8fc951a4687ba51b53ad837a09d19a9409b63deafcc4422ef384f5f79dd11246653287cec76f826fe970cb5becf40efc9bcda97efb264b28a8cae7cdf62978d3a01e6e744e56c29d13129b6ffe09db400abd2b4da16e7ab7f0da1cbeea77c8f8ec4c67e284e5e256062ff1a604c8e1bdf25235a44952b73c85d6f498f148a298d6347dd8fcadd8c964cc17a9c8a44097f3e081e88d10d0ffd751bbc964c884e04f94be84b3adc930546b21683e004cea7201e5f1b22b6cbb2534226a4f27d06f0c65a16ffa8f9735a0f919472f4e62a77769bb1cb4aa7eb88a062096640fc95e2cfe59baf87ec16129f70f87671343f66423bb0380b968006375ed455606181d0a909c73a5306be2a1ee604cc270e7da54c92ed3c8133226ace827a6564987823b4569c6bfa791c050fdc38dc7567655a3b4b7e4377bbcba7ad0488a899614c403d536d52a2de7fcd2c3bcc877361bf36e955dbb243a2b9902db849895af53336c476ab748723bca5c01f87a04bfcdfcb2509fc1851134fd0e0a0b6678ca064578e1dc3fcf4f89e490b6dc7834484521d7600b969ae4331d173cabb137eaf99b68569f493ce1e137aff4136caedf550bd7686faab632de521f9f20d54d6dfe1de6c5e18cd2a640233a2a80a7b59baa09da9eb2fdf607ea397367771b1db7231b8c6ce40b43d29b86e9f5903b933b76f3c54a7bcc7323182035b308203570201013081cb3081b2310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312730250603550403131e5a65616c694420416476616e6365642049737375696e6720434120434e54021473be5b5decc753d202dc3596ccf2b5402d6b701b300d06096086480165030402010500a0820160301a06092a864886f70d010903310d060b2a864886f70d0109100104302f06092a864886f70d010904312204204584ee2cbb02cf7a6a299f4603fa16428bbbe3f0351f666538da938f784d4cf43082010f060b2a864886f70d010910022f3181ff3081fc3081f93081f604200b2fb5b074648a2e3a7cc5679a971ff651b49804cd44295a4c0a8aa94e61dfb53081d13081b8a481b53081b2310b3009060355040613025345310e300c060355041113053131313536311c301a06035504091313426f7820333433372c2053746f636b686f6c6d3121301f06092a864886f70d0109011612737570706f7274407a65616c69642e636f6d311530130603550461130c53453535363937323432383831123010060355040a13095a65616c6944204142312730250603550403131e5a65616c694420416476616e6365642049737375696e6720434120434e54021473be5b5decc753d202dc3596ccf2b5402d6b701b300d06092a864886f70d0101010500048201004431019ba357c38cedddcc5e2d64651ea2bded28e4fa09f09707edc0d3257b25492b1d7949b1988cf707853b789f531da3eece589996bd584edd0ae21342cc0fb487cdeca4fa911f699bd96ab1f67352e1bf2351ddb2377903feed42c93b835f480b466970d01d5ea18e8193bc03b72bbcb756aff134a3b3ebd9c6fcee41103e689d6a4b4b08a668778303e2a8538b04b5d9cea67c508f41cadbe0ba75810e7151130db156f73de9355ba2cf93cabca8bc4f878ca6007361a916b512101e293fe7ae87d086907abbee1f7ef474899e006dbd8e88cc631e08bc58c6f999b5d295703c98d57f0b3a958463111784e270673365dd10f291b1bba06745d7d706d630"

value = cms.ContentInfo.load(binascii.unhexlify(payload))

start = time.time()
value.dump(force=True)
end = time.time()
print(start, end, end - start)

This takes 20 seconds on my machine, whereas @peteris-zealid's example only takes about 7. Sure, I used force=True to ignore cached values, but a payload of ~15-odd KiB should not take this long to dump anyhow.

EDIT: hang on, the process is suddenly massively faster (<0.01s) if I omit the tsp import. I suspect one of the types that is only registered in that module may be the culprit...

[MatthiasValvekens]

I haven't been able to pin down exactly why, but for some reason .dump() gets called on SetOfSigningCertificatesV2 an ungodly number of times. If I replace the reference to ESSCertIDv2 with Any to stop the parser there, it's called twice (matching the number of times this attribute occurs throughout the entire payload).

It's not the full story, but I hope it's a useful start.

[joernheissler]

Thanks, that's very useful. I can reproduce the issue and I'm quite puzzled so far.

value.native can be used to reproduce the issue too.

[joernheissler]

I might be a step closer: Sequence and SequenceOf each got a _mutated flag that gets set, but never reset. When adding self._mutated = False to the end of def _set_contents, dump(force=True) becomes fast (0.02s instead of 36s), without changing the result. Test suite works too.

But a code change like this is risky, I didn't think it through yet. The original input doesn't match the output. Perhaps the original input isn't properly encoded. I'll look into this. But loading+dumping the modified output is still slow (without above code change), and not sure yet why.

[wbond]

Hoping to take a look at this soon. Sounds like an interesting issue to debug.

[wbond]

Yeah, I ran line profiler on above and saw core.Asn1Value.dump() called 9.7M times…