tidis
tidis copied to clipboard
yongman
Does tidis support pipeline operation?
Hi, I want to know whether tidis support pipeline operation?
I use the following command and encountered the problem.
cat redis_command_zset_310100.log | redis-cli -h 10.125.233.215 -p 5379 --pipe
Error reading from the server: Connection reset by peer
@jasonkoo it is supposed to support pipeline request, and the error may be caused by unsupported commands?
@yongman From our testing, it does not support.
Without pipeline, it worked as expected. head -n 2 redis_command_zset_310100_clean.log | redis-cli -h 10.125.233.215 -p 5379 (integer) 0 (0.60s) (integer) 0 (1.05s)
With pipeline, it responded with Error. head -n 2 redis_command_zset_310100_clean.log | redis-cli -h 10.125.233.215 -p 5379 --pipe All data transferred. Waiting for the last reply... Error reading from the server: Connection reset by peer
Any idea about this?
@yongman For you reference
head -n 2 redis_command_zset_310100_clean.log zadd gis:staging:zset:eta:quadkey:15_27427_13447:weekly:1h:3:13 60 '"15_27427_13447"' 155 '"15_27428_13447"' 163 '"15_27429_13447"' 950 '"15_27430_13447"' 800 '"15_27431_13447"' 800 '"15_27432_13447"' 578 '"15_27423_13446"' zadd gis:staging:zset:eta:quadkey:15_27428_13447:weekly:1h:3:13 149 '"15_27427_13447"' 60 '"15_27428_13447"' 60 '"15_27429_13447"' 902 '"15_27430_13447"' 760 '"15_27431_13447"' 760 '"15_27432_13447"' 773 '"15_27423_13446"'
@jasonkoo tidis just add echo command support, because redis-cli --pipe will send an echo command at last.
You can try it again.
@yongman Thanks for the quick reply and fix.
I updated my local code and rebuilt the tidis server.
-
Tried with the following command
head -n 2 redis_command_zset_310100_clean.log | redis-cli -h 10.125.233.215 -p 5379 --pipe
Still I got the following error.
The client side All data transferred. Waiting for the last reply... No replies for 30 seconds: exiting. errors: 1, replies: 0
The server side time="2018-11-28T11:55:04+08:00" level=error msg="bad resp line terminator"
-
Tried with the following command cat redis_command_zset_310100_clean.log | redis-cli -h 10.125.233.215 -p 5379 --pipe
I got the following error.
The client side All data transferred. Waiting for the last reply... Error reading from the server: Connection reset by peer
The server side time="2018-11-28T11:58:56+08:00" level=error msg="bad resp line terminator"
Anything wrong with my operation?
@jasonkoo you should convert command to RESP protocol as follows:
*16
$4
ZADD
$58
gis:staging:zset:eta:quadkey:15_27428_13447:weekly:1h:3:13
$2
60
$16
"15_27428_13447"
$2
60
$16
"15_27429_13447"
$3
149
$16
"15_27427_13447"
$3
760
$16
"15_27431_13447"
$3
760
$16
"15_27432_13447"
$3
773
$16
"15_27423_13446"
$3
902
$16
"15_27430_13447"
*16
$4
ZADD
$58
gis:staging:zset:eta:quadkey:15_27427_13447:weekly:1h:3:13
$2
60
$16
"15_27427_13447"
$3
155
$16
"15_27428_13447"
$3
163
$16
"15_27429_13447"
$3
578
$16
"15_27423_13446"
$3
800
$16
"15_27431_13447"
$3
800
$16
"15_27432_13447"
$3
950
$16
"15_27430_13447"
then you can send to tidis using redis-cli --pipe
@yongman Thanks for the guide.
I tried with RESP protocol. It is partially successful.
cat test.txt
*3
$3
SET
$4
name
$6
nobody
I tried with cat test.txt | redis-cli -h 10.125.233.215 -p 5379 --pipe and I successfully inserted the data. But I got the following errors.
From the client side,
All data transferred. Waiting for the last reply...
No replies for 30 seconds: exiting.
errors: 1, replies: 1
From the server side,
time="2018-11-28T19:10:57+08:00" level=error msg="short resp line"
BTW, the version of redis-cli I am using is 4.0.10.
Am I still missing something? Thank you.
@yongman I am using the latest the code from master branch. The problem still exists.
@jasonkoo use xxd command to open test.txt file and check, you properly treat \n as \r\n.
xxd aof
00000000: 2a33 0a24 330a 7365 740a 2434 0a6e 616d *3.$3.set.$4.nam
00000010: 650a 2436 0a6e 6f62 6f64 790a e.$6.nobody.
The right case should be
xxd aof1
00000000: 2a33 0d0a 2433 0d0a 5345 540d 0a24 330d *3..$3..SET..$3.
00000010: 0a6b 6579 0d0a 2435 0d0a 7661 6c75 650d .key..$5..value.
00000020: 0a
Before check tidis, you can test the case in redis server first.
@yongman I checked with my file and I am sure each line ends with \r\n.
cat test.txt
*3
$3
SET
$4
name
$6
shstes
xxd test.txt
0000000: 2a33 0d0a 2433 0d0a 5345 540d 0a24 340d *3..$3..SET..$4.
0000010: 0a6e 616d 650d 0a24 360d 0a73 6873 7465 .name..$6..shste
0000020: 730d 0a s..
cat test.txt | redis-cli -h 10.125.233.215 -p 5379 --pipe
All data transferred. Waiting for the last reply...
No replies for 30 seconds: exiting.
errors: 1, replies: 1
cat logs/stderr.log
time="2018-11-29T16:34:34+08:00" level=info msg="server started"
time="2018-11-29T16:34:34+08:00" level=info msg="[pd] create pd client with endpoints [10.125.233.215:2379]"
time="2018-11-29T16:34:34+08:00" level=info msg="[pd] leader switches to: http://10.125.233.215:2379, previous: "
time="2018-11-29T16:34:34+08:00" level=info msg="[pd] init cluster id 6579393768691328756"
time="2018-11-29T16:34:34+08:00" level=info msg="server listen in :5379"
time="2018-11-29T16:34:34+08:00" level=info msg="Async tasks started for async deletion"
time="2018-11-29T16:37:51+08:00" level=error msg="short resp line"
@jasonkoo oh, I think your tidis server is not be built with the newest commit. The short resp line error is caused by redis-cli --pipe will send duplicated \r\n at last and tidis handle this different.
https://github.com/yongman/tidis/commit/154009add1f401771b246a95630bc64f1132c90a this commit will ignore the duplicated \r\n as redis does committed a day ago. Please make sure you have the latest code.