redisql
redisql copied to clipboard
DGKSK8LIFE
use redis pipeline to write CopyToString
To solve issue #45
before data pipelining
> go test -v -db mysql -rows 10000
Preparing Test...
=== RUN TestCopyToString
--- PASS: TestCopyToString (11.00s)
=== RUN TestCopyToList
--- PASS: TestCopyToList (1.25s)
=== RUN TestCopyToHash
--- PASS: TestCopyToHash (1.25s)
PASS
ok github.com/DGKSK8LIFE/redisql 18.056s
after data pipelining
> go test -v -db mysql -rows 10000
Preparing Test...
=== RUN TestCopyToString
--- PASS: TestCopyToString (1.31s)
=== RUN TestCopyToList
--- PASS: TestCopyToList (1.05s)
=== RUN TestCopyToHash
--- PASS: TestCopyToHash (1.15s)
PASS
ok github.com/DGKSK8LIFE/redisql 7.865s
Should we replicate the pipelining for lists and hashes? I think we could try to exec the pipeline after it reads through all the rows rather than just column.
I made the pipeline executed after we scanned through all rows rather than just columns.
pipeline executed after column scan (10k rows)
go test -v -db mysql -rows 10000
Preparing Test...
=== RUN TestCopyToString
--- PASS: TestCopyToString (1.31s)
=== RUN TestCopyToList
--- PASS: TestCopyToList (1.05s)
=== RUN TestCopyToHash
--- PASS: TestCopyToHash (1.15s)
PASS
ok github.com/DGKSK8LIFE/redisql 7.865s
pipeline executed after row scan (10k rows)
go test -v -db mysql -rows 10000
Preparing Test...
=== RUN TestCopyToString
--- PASS: TestCopyToString (0.39s)
=== RUN TestCopyToList
--- PASS: TestCopyToList (0.25s)
=== RUN TestCopyToHash
--- PASS: TestCopyToHash (0.25s)
PASS
ok github.com/DGKSK8LIFE/redisql 5.085s
pipeline executed after column scan (50k rows)
go test -v -db mysql -rows 50000
Preparing Test...
=== RUN TestCopyToString
--- PASS: TestCopyToString (7.76s)
=== RUN TestCopyToList
--- PASS: TestCopyToList (6.75s)
=== RUN TestCopyToHash
--- PASS: TestCopyToHash (6.88s)
PASS
ok github.com/DGKSK8LIFE/redisql 42.491s
pipeline executed after row scan (50k rows)
go test -v -db mysql -rows 50000
Preparing Test...
=== RUN TestCopyToString
--- PASS: TestCopyToString (2.68s)
=== RUN TestCopyToList
--- PASS: TestCopyToList (1.67s)
=== RUN TestCopyToHash
--- PASS: TestCopyToHash (1.30s)
PASS
ok github.com/DGKSK8LIFE/redisql 25.817s
haven't test on postgres yet, will update you later
Should we replicate the pipelining for lists and hashes? I think we could try to exec the pipeline after it reads through all the rows rather than just column.
Agreed. Yeah, bulk insertion into Redis is better overall (IMO we should do pipelining for all datatypes). Try to implement it similarly across all data types and optimize it accordingly if possible. Really great contributions so far 👍🏻. The goal all in all is <5 seconds per function for 1million+ rows. Please join the Discord so that we can communicate more efficiently, thanks.
