Performance comparison for asn1 decode of SNMP

[philmayers]

This is not really an issue - it's more by way of informational/comparison, given the focus on performance.

We use the ancient https://github.com/jpwarren/libsnmp for a local/bespoke high-throughput SNMP poller. I've recently tried using pysnmp, which had lower performance, apparently due to pyasn1 - see https://github.com/etingof/pysnmp/issues/44

I recently came across asn1crypto and decided to try and build a simple SNMP encoder/decoder and compare performance. It seems that asn1crypto is faster than pysnmp/pyasn1 but still slower than the libsnmp code.

This is in many ways an unfair comparison as libsnmp is a very constrained application and can avoid being completely general - but I thought it was worth raising here in case it reveals any opportunities for optimisation.

The test script I use can be found here:

https://gist.github.com/philmayers/67b9300d8fb7282481a1a6af5ed45818

Some quick results from my box

cpython2.7

timeit for libsnmp 1000 iterations per-call 0.366773ms
timeit for pysnmp 1000 iterations per-call 1.913801ms
timeit for asn1crypto 1000 iterations per-call 0.784892ms

pypy 5.4.0 (much larger iteraction count to let JIT kick in)

timeit for libsnmp 100000 iterations per-call 0.026978ms
timeit for pysnmp 100000 iterations per-call 0.192198ms
timeit for asn1crypto 100000 iterations per-call 0.066650ms

Feel free to close this if the info is not of interest - as noted, comparing a very constrained libsnmp to the much more general encoding in asn1crypto is possibly not reasonable/useful.

Thanks for your work on the project - the code is great quality and I really like the API

[wbond]

A lineprofiler run could help here to see which part of asn1crypto is most expensive for this use case.

I peeked at the script. One aspect is that with asn1crypto you are converting the whole structure (recursively) to native Python data types. Only grabbing the two values of interest would likely make a difference. That may be the case with libsnmp also, I’m not sure.

From the profiling I have done (which was a while ago), one of the most expensive parts was the generic parsing of the raw ASN.1 data into class, tag, length, payload. Unfortunately I don’t foresee much room for improvement there while still handling generic ASN.1 structure and being pure Python. I imagine all of the try/catch blocks with the nice exception handling probably add a bit to the overhead.

In terms of improving performance, the most likely improvement for large chunks of data would be to create an optional C ext that would do the parsing of the raw data and construct the Python object, reducing the amount of byte data manipulation being done in pure Python.

From what I recall, one of the biggest performance differences between pyasn1 and asn1crypto is handling of bit strings. pyasn1 uses shift operations to shift each bit off of the byte string to construct a tuple of 1s and 0s. However, many uses of bit string in crypto don’t ever access is a bit-wise fashion, but just use the full byte string.

[eerimoq]

@philmayers I recently got intrested in ASN.1 and wrote a Python package called asn1tools with a ASN.1 specification parser and a BER codec. There are bugs, there are design flaws, there are missing functionality, but it can actually parse and decode your SNMP performance example pretty quickly. As more features are added the performace will likely degrade, but hopefully just slightly.

Anyway, below are the performance numbers from TravisCI executing this script (based on your script):

Decoding the message 3000 times took:

PACKAGE      SECONDS
asn1tools    0.523167
libsnmp      1.131108
asn1crypto   2.722282
pyasn1       9.171269

The decoded data is a Python dictionary:

{
    "version": 0,
    "community": bytearray('public'),
    "data": {
        "set-request": {
            "request-id": 60,
            "error-status": 0,
            "error-index": 0,
            "variable-bindings": [
                ...
            ]
        }
    }
}