go-redis-server
go-redis-server copied to clipboard
Memory leak when parse large request
func readArgument(r *bufio.Reader) ([]byte, error) {
line, err := r.ReadString('\n')
if err != nil {
return nil, malformed("$<argumentLength>", line)
}
var argSize int
if _, err := fmt.Sscanf(line, "$%d\r", &argSize); err != nil {
return nil, malformed("$<argumentSize>", line)
}
// I think int is safe here as the max length of request
// should be less then max int value?
data, err := ioutil.ReadAll(io.LimitReader(r, int64(argSize)))
if err != nil {
return nil, err
}
if len(data) != argSize {
return nil, malformedLength(argSize, len(data))
}
// Now check for trailing CR
if b, err := r.ReadByte(); err != nil || b != '\r' {
return nil, malformedMissingCRLF()
}
// And LF
if b, err := r.ReadByte(); err != nil || b != '\n' {
return nil, malformedMissingCRLF()
}
return data, nil
}
ioutil.ReadAll
will allocate at least 512bytes memery even if we don't need it, when prase a large request (for instance, MGET key1 key2 ... keyn
), the memory costs will increase fast ( n * 512bytes).
Force gc will trigger every 2 min, and gc is also a goroutine. if memory increase very fast, even reach the phyiscal memory, gc will not invoke immediately.