go-redis-server icon indicating copy to clipboard operation
go-redis-server copied to clipboard

Memory leak when parse large request

Open lfyzjck opened this issue 10 years ago • 0 comments

func readArgument(r *bufio.Reader) ([]byte, error) {

    line, err := r.ReadString('\n')
    if err != nil {
        return nil, malformed("$<argumentLength>", line)
    }
    var argSize int
    if _, err := fmt.Sscanf(line, "$%d\r", &argSize); err != nil {
        return nil, malformed("$<argumentSize>", line)
    }

    // I think int is safe here as the max length of request
    // should be less then max int value?
    data, err := ioutil.ReadAll(io.LimitReader(r, int64(argSize)))
    if err != nil {
        return nil, err
    }

    if len(data) != argSize {
        return nil, malformedLength(argSize, len(data))
    }

    // Now check for trailing CR
    if b, err := r.ReadByte(); err != nil || b != '\r' {
        return nil, malformedMissingCRLF()
    }

    // And LF
    if b, err := r.ReadByte(); err != nil || b != '\n' {
        return nil, malformedMissingCRLF()
    }

    return data, nil
}

ioutil.ReadAll will allocate at least 512bytes memery even if we don't need it, when prase a large request (for instance, MGET key1 key2 ... keyn), the memory costs will increase fast ( n * 512bytes).

Force gc will trigger every 2 min, and gc is also a goroutine. if memory increase very fast, even reach the phyiscal memory, gc will not invoke immediately.

lfyzjck avatar Dec 17 '14 05:12 lfyzjck