Question: what is the result of `decode codec cs` if `cs` contains uncomplete data ?

[Jo-Blade]

Hello, I'm interested into using your library to write a simple LDAP client using eio for the network stack.

As I understand, a LDAP packet is an ASN.1 BER encoded "SEQUENCE", so it should be convenient to parse it using this library.

According to the documentation, the Asn.decode function takes a "string" as input and not a bytes input stream. So, I will have to read n bytes from the socket and convert it into a string before trying to decode. The problem is, I don't know in advance the size of the LDAP packet so I think there is a risk I try to decode an unfinished BER "SEQUENCE". I don't know to handle this case, and I don't want to write a function to calculate the size of the sequence because I would have to rewrite parts of what this lib is already doing.

Do you have any advice to help me ?

Thank you

[hannesm]

Thanks for opening this issue. Indeed, there are no streaming parsers available (yet?).

From trying out what you describe, if you only provide a partial ASN.1 string, you get back Error (`Parse "Unexpected EOF ..." -- so your code could handle this error case by reading more data from the network stream.

FWIW, the code I tested it with:

let projections_of asn =
    let c = Asn.codec Asn.der asn in
    (Asn.decode c, Asn.encode c)

let asn =
    Asn.S.(sequence3
             (required ~label:"version" int)
             (required ~label:"sequence" int)
             (required ~label:"name" utf8_string))

let dec, enc = projections_of asn

Ohex.encode (enc (1, 2, "Foo"))
(* returns "300b0201010201020c03466f6f" *)

dec (Ohex.decode "300b0201010201020c0346");;
(* returns:
Error
 (`Parse
    "Unexpected EOF (msg String.sub / Bytes.sub): 300b020101020102 0c0346")
*)

Hope this helps. Of course, this package could provide a streaming parser (and encoder) -- but you can build that also on top of this library. If you dislike the error, we can discuss about more fine-grained errors returned by the decoding parts (so you get back a `Partial error instead of the general `Parse of string). What do you think?

[Jo-Blade]

Thank you for your very quick response.

I think handling the "EOF" error as you suggest suits well my need.

But reading strings to see which error happened is far from ideal (the strings values are not documented, and we have no warranty that these strings won't be changed in the future...). So I really like your proposition to have more precise errors. Do you think we can do it without breaking too much the other packages that rely on asn1-combinators ?

[Jo-Blade]

From the other side, handling streams as input could be a cool feature, but I have no idea how much work it would be, as I'm not really into parsers, and if it will really be so useful.

I've seen, for example, Angstrom, which is a parser lib that can handle streams. However, it looks like they had to do duplicated work for all async IO libraries to make it work (Async, Lwt, and there is an Eio branch pending). This is maybe out of the scope of this project.

[hannesm]

I think there'll be some API breakage, but that's fine. I can propose a PR in the following days.

In respect to streaming parsers, if you take a look at e.g. jsont/xmlm, such an API could be provided independent of the IO layer. But I don't have much time for this at the moment. If there's a pressing need (e.g. someone wants to parse asn1 values that exceed the memory), I'd need to take some time to figure out how to do it properly.

[hannesm]

I prepared this branch https://github.com/hannesm/ocaml-asn1-combinators/tree/error-eof for you to try out. Let me know whether this suits your needs.

[hannesm]

Dear @Jo-Blade, did you have time to test the branch I prepared? Would it solve the issue for you?

[Jo-Blade]

Dear @Jo-Blade, did you have time to test the branch I prepared? Would it solve the issue for you?

Hi, sorry for the late response. I was very busy the last few weeks, and to be honest, I hadn't written any code before opening this issue, so I had to write it first.

I can now confirm that your merge request successfully solved my issue !

You can find my LDAP client prototype here if you're interested: https://git.inpt.fr/pisentt/ocaml-ldap-client

Big thanks again for your lib; it was really a pleasure to use it for this project. Everything works perfectly well. My only problem is an infinite loop in the lib/messages.ml with the fixpoint function when I try to fully match the LDAP RFC (see comments). I will investigate more and maybe open another issue.