Speed up parsing

[PMunch]

Combparser turns out to be fairly slow. This is probably caused by it's very recursive nature and lot's of string slicing (which creates a new copy of the string). As a means to fix this either combparser need to be sped up by improving how it handles strings internally, or a new parser needs to be written/implemented.

[zevv]

Well, too bad: I implemented the parser in NPeg today, but the compiler does not agree when running at compile time:

Error: VM problem: too many registers required

[zevv]

Ok, mistake: I was running an old Nim version, but this has recently been fixed on devel. Parser works at compile time.

[PMunch]

But only for devel? Have you tried the parser with any more complicated protobuf files?

[zevv]

Quoting PMunch (2019-11-13 10:36:13)

But only for devel?

Only for devel - will not work for 1.0.2 or earlier because of VM limitations.

Have you tried the parser with any more complicated protobuf files?

Well, it parses stuff I feed it, but I'm not sure if I'm properly parsing into your AST yet - that will probably require a bit more love and attention to get right. Not sure if it makes sense to spend the time on that now given that the Nim VM also has a hard time with running this parser in the VM....

-- :wq ^X^Cy^K^X^C^C^C^C

[madasebrof]

I love the concept and usage of this library!

But I noticed this issue, so I thought I'd post here. I was trying to use this to decode a fairly large byte array (51x512), but the results are unusable.

It takes 2ms to read data off the socket into a string, then another 90ms (!) to parse.

Executing task server in .../protobuf_tester.nimble
Server: started. Listening to new connections on port 8888...
Server: client connected
2.60ms - got raw message = 1048727 bytes - 93.29ms - parsed protobuf 
2.08ms - got raw message = 1048727 bytes - 91.03ms - parsed protobuf 
1.95ms - got raw message = 1048727 bytes - 92.02ms - parsed protobuf 
1.72ms - got raw message = 1048727 bytes - 92.05ms - parsed protobuf 
1.95ms - got raw message = 1048727 bytes - 91.40ms - parsed protobuf 

[PMunch]

Do you have a test case?

This should really be it's own issue though. The original issue is about the speed of parsing the protoc files and not about the runtime speed.

[PMunch]

And @zevv, do you still have that NPEG parser lying around? Probably works with the latest Nim versions so this could finally be implemented.

[zevv]

Quoting PMunch (2021-10-29 14:15:19)

And @zevv, do you still have that NPEG parser lying around? Probably works with the latest Nim versions so this could finally be implemented.

Apart from an ancient vim undo file at /home/ico/.vim_undo/%home%ico%sandbox%prjs%npeg%protobuf.nim, nothing to be found on my machine :(

[PMunch]

And nothing useful in that file? Bummer..