protocyt
protocyt copied to clipboard
Evgenus
Unacceptable performance
I am migrating to python 3 and I should upgrade my dependencies. I was using cprotobuf in python 2. it was great but unfortunately it does not support python 3. I tested some of python 3 compatible protobuf implementations and this was one of them.
I have a single ~100kb .proto file.
protocyt first made a ~3.6mb .pyx file from it in a couple of minutes
then made a ~68.6mb .c file from it in about ten minutes.
at last compiled it to a ~16mb '.so' file. this last compile took an hour! In this compile step, it used over 10 Gb memory.
My system is a Core i7 mac with 16 Gb memory and SSD hard drive. I can't imagine how long it would take on a low-end system. I bet it won't even compile!
It get's even worse. I couldn't find any documentation in Github of this project, so I should do a trial-and-error. I just re-run the application and guess what? it deleted the .so file and started the whole procedure from the beginning!
Do I really have to wait 1.5 hours for each execution of my code? I really liked coding style of your project and really want to use it, but please do something about it.
@masihyeganeh, I'm just curious, was encoding/decoding super-fast or was it somewhere between slow and moderate?
@tailhook I'm not sure if got your question, If you mean after creation of .so file and in use, I can't answer that. As I said, because of lack of a documentation I couldn't even make it work and after run for second time, it started the whole process from the beginning so I could not test it's performance.
Your proto file is really big. You might not get much feedback because of that. Unless you have a specific reason to use protos you should flip to a more appropriate serialization
Sent from my iPhone
On May 21, 2015, at 6:58 PM, Masih Yeganeh [email protected] wrote:
I am migrating to python 3 and I should upgrade my dependencies. I was using cprotobuf in python 2. it was great but unfortunately it does not support python 3. I tested some of python 3 compatible protobuf implementations and this was one of them.
I have a single ~100kb .proto file. protocyt first made a ~3.6mb .pyx file from it in a couple of minutes then made a ~68.6mb .c file from it in about ten minutes. at last compiled it to a ~16mb '.so' file. this last compile took an hour! In this compile step, it used over 10 Gb memory.
My system is a Core i7 mac with 16 Gb memory and SSD hard drive. I can't imagine how long it would take on a low-end system. I bet it won't even compile!
It get's even worse. I couldn't find any documentation in Github of this project, so I should do a trial-and-error. I just re-run the application and guess what? it deleted the .so file and started the whole procedure from the beginning!
Do I really have to wait 1.5 hours for each execution of my code? I really liked coding style of your project and really want to use it, but please do something about it.
— Reply to this email directly or view it on GitHub.
