mysql2redis icon indicating copy to clipboard operation
mysql2redis copied to clipboard

Published 20 hours ago • verified publisher icon jackeylu

performance improvement: add redis Connection pool

Open dawnbreaks opened this issue 11 years ago • 6 comments

Hi, all I think there is no need to make a tcp connection to redis server everytime you execute a udf function, a connection pool would be great for performance.

anybody could provide a patch for this enhancement? Or give me your opinions.

dawnbreaks avatar Jun 06 '13 06:06 dawnbreaks

+1 i'm agree with you - opening and closing connexions on redis it's a perf killer for both mysql & redis

ichiriac avatar Aug 09 '13 12:08 ichiriac

Well, as my opinion, I think MySQL2Redis is not a good choice to dump data from MySQL to Redis very frequently.

And, the hot pot of performance may be not in the redis server.

If we can prepare all the data firstly then dump them into redis with one or N (could be any number less than 1024) tcp connection from udp to redis, the performance could be improved a lot.

I am busy not for my job, but I can try to fix it in my part-time, thanks for your suggestion.

jackeylu avatar Aug 10 '13 15:08 jackeylu

Hi, jackey UDP maybe a better good choice, but that 's not best. An SQL may update thousands of rows in a table, in such a situation you need a queue to cache all the rows and a backend thread to poll the queue and update the redis asynchronously.

I have implemented a similar project based on this project. See https://github.com/dawnbreaks/mysql2redis

Thanks & Best Regards.

dawnbreaks avatar Aug 12 '13 00:08 dawnbreaks

hi dawnbreaks, thank you for your information.

I am so glad that you do a better improvement for this project.

May I have a little suggestion? Can you provide a function that the data in a MySQL server can dump into more than one redis server?

The "More" may be 2 or 3, sometimes, we may have some backup of redis. but, people can sync data between redises with the mechanism of redis.

jackeylu avatar Aug 12 '13 23:08 jackeylu

hi, jackey Why you need a udf to dump data from mysql to redis? If you use redis as a cache server only , redis should only cache the hot data. If the dataset is small, you could clean the data in redis every midnight; if a large dataset, you need a salve server to backup the master redis.

Also you could consider abandon mysql and use redis as a main storage server.

Best Regards!

dawnbreaks avatar Aug 13 '13 02:08 dawnbreaks

dawnbreaks, you are right, The solution of MySQL2Redis is odd, but it is really occurred. Why I created such a project? That is came from an odd requirement of one of my web site projects.

We was unable to change our architecture at that time, so MySQL2Redis such a temp solution came.

jackeylu avatar Aug 13 '13 12:08 jackeylu