Feature request - increase performance by using Hiredis for packing commands into RESP

[zalmane]

Version: What redis-py and what redis version is the issue happening on? all

Platform: What platform / version? (For example Python 3.5.1 on Windows 7 / Ubuntu 15.10 / Azure) all

Description: Description of your issue, stack traces from errors and code that reproduces the issue Currently, if hiredis-py is installed, redis-py uses Hiredis for parsing the response. It seems that when sending large HSETs with many fields, a major CPU bottleneck is packing the commands into RESP. Using Hiredis bindings for packing the command can yield a 15%-20% performance gain over the python implementation

[WisdomPill]

Are you open to help doing this? I think this would be very welcome :)

[utkarshgupta137]

I've already created a PR with the command packer (and key slot) written in C: https://github.com/redis/redis-py/pull/2118. I don't believe hiredis-py has existing bindings for command packing.

[chayim]

I'm generally pro increasing our use of hiredis to get better performance. However - I don't like the (understandable) compilation requirement this places on the library.

If we maintain both paths, so that users with their non-compiler use cases can proceed - I see no reason not to further embrace hiredis.

[utkarshgupta137]

@chayim I've not removed the ability to use the package without the C extension. In fact, I've added tests to ensure that everything works fine with or without the C extensions.

[zalmane]

Would it make sense to add the C extension into hiredis-py and then rely on it as an optional dependency like the other methods are used?

[chayim]

Should close - this is available already.