ASN.1/BER correctness

[mtodd]

Working on #142 revealed to me the need for better understanding, testability, and regression testing for our ASN.1/BER practices.

I discovered the PROTOS project which is designed to test LDAP servers for robustness, but gives us a library of useful sample data to compare ourselves against: https://www.ee.oulu.fi/research/ouspg/PROTOS_Test-Suite_c06-ldapv3

I'd like to see how we might integrate those samples into our automated testing, or explore other options to improve reliability/correctness. This is motivated by a need to ensure proper data integrity in production use while allowing for major refactoring to be undertaken without introducing regressions.

cc @jch @schaary

[jch]

:metal:

[mtodd]

https://github.com/ruby-ldap/ruby-net-ldap/pull/172 is an example of some fairly significant issues with our implementation of BER encoding: basic integer encoding is broken according to the spec.

[mtodd]

Some relevant publications: http://www.oss.com/asn1/resources/books-whitepapers-pubs/asn1-books.html

[mtodd]

I've been collecting resources and references to useful examples, technical documents, implementations, et al. Here's what I've found:

---

The Net::LDAP Perl library depends on Convert::ASN1 library (see below), and uses the LDAP ASN.1 spec to manage encoding/decoding messages, something I'd like to consider for this library: https://github.com/perl-ldap/perl-ldap/blob/master/lib/Net/LDAP/ASN.pm

The Convert::ASN1 Perl library, used by the Net::LDAP Perl library, is a fount of useful implementation examples and data:

• docs: http://search.cpan.org/~gbarr/Convert-ASN1-0.27/lib/Convert/ASN1.pod
• encoding: https://github.com/gbarr/perl-Convert-ASN1/blob/master/lib/Convert/ASN1/_encode.pm
• decoding: https://github.com/gbarr/perl-Convert-ASN1/blob/master/lib/Convert/ASN1/_decode.pm
• encoding/decoding utils: https://github.com/gbarr/perl-Convert-ASN1/blob/master/lib/Convert/ASN1.pm
• tests: https://github.com/gbarr/perl-Convert-ASN1/blob/master/t/00prim.t
• bigint tests: https://github.com/gbarr/perl-Convert-ASN1/blob/master/t/06bigint.t

The Encoding::BER Perl library also has relevant implementation and examples, but is not in use by the Net::LDAP Perl library:

• docs: http://search.cpan.org/~jaw/Encoding-BER-1.00/lib/Encoding/BER.pm
• https://github.com/jaw0/Encoding-BER/blob/master/lib/Encoding/BER.pm

The OpenSSL library uses the ASN.1 DER encoding rule and implements ASN.1 encoding/decoding libraries. It might also support BER encoding/decoding; need to investigate further; it could be used as a replacement for our own BER implementation:

• https://github.com/openssl/openssl/blob/master/crypto/asn1/a_int.c
• Ruby shim docs: http://ruby-doc.org/stdlib-1.9.3/libdoc/openssl/rdoc/OpenSSL/ASN1.html
• Ruby shim: https://github.com/ruby/ruby/blob/trunk/ext/openssl/ossl_asn1.c
• Ruby shim tests: https://github.com/ruby/ruby/blob/trunk/test/openssl/test_asn1.rb

A general discussion about ASN.1 encoding/decoding in Ruby that is interesting: https://groups.google.com/forum/#!topic/comp.lang.ruby/PyrTDhmV4-E

---

ApacheDS has a by-the-book implementation of the spec (unlike other servers) which surfaced problems in the Integer encoding in https://github.com/ruby-ldap/ruby-net-ldap/pull/172. Relevant encoding/decoding implementations (with examples for reference):

• decoding: https://github.com/apache/directory-shared/blob/trunk/asn1/ber/src/main/java/org/apache/directory/api/asn1/ber/tlv/IntegerDecoder.java
• encoding: https://github.com/apache/directory-shared/blob/trunk/asn1/ber/src/main/java/org/apache/directory/api/asn1/ber/tlv/BerValue.java#L295-L392

OpenLDAP is more lax with its implementation of the spec (not enforcing negative message ID values, or treating any integer above zero as TRUE) but it provides a good reference of implementation:

• decoding: https://github.com/openldap/openldap/blob/master/libraries/liblber/decode.c#L698
• encoding: https://github.com/openldap/openldap/blob/master/libraries/liblber/encode.c

---

Vijay Mukhi documented some BER basics here: http://www.vijaymukhi.com/vmis/ber.htm

Dr. Lillian Cassel wrote a bit about BER basics here: http://www.csc.villanova.edu/~cassel/netbook/ber/ber.html

A Layman's Guide to a Subset of ASN.1, BER, and DER is a document from the RSA Laboratories and includes some specific examples of encoded values.

The ITU-T X.690 specification (PDF): http://www.itu.int/ITU-T/studygroups/com17/languages/X.690-0207.pdf