Investigate pooling of Gob encoders/decoders

[bep]

See

https://www.reddit.com/r/golang/comments/4ix4gj/oh_gob/

[asdine]

I have tried to improve the Encode part with this:

// Package gob contains a codec to encode and decode entities in Gob format
package gob

import (
    "bytes"
    "encoding/gob"
    "sync"
)

// Codec serializing objects using the gob package.
// See https://golang.org/pkg/encoding/gob/
var Codec = newCodec()

type encBuffer struct {
    enc  *gob.Encoder
    buff *bytes.Buffer
}

func newCodec() *gobCodec {
    var c gobCodec
    c.b.New = func() interface{} {
        return &bytes.Buffer{}
    }
    c.e.New = func() interface{} {
        buffer := c.b.Get().(*bytes.Buffer)
        buffer.Reset()
        return &encBuffer{
            enc:  gob.NewEncoder(buffer),
            buff: buffer,
        }
    }

    return &c
}

type gobCodec struct {
    b sync.Pool
    e sync.Pool
}

func (c *gobCodec) Encode(v interface{}) ([]byte, error) {
    e := c.e.Get().(*encBuffer)
    err := e.enc.Encode(v)
    if err != nil {
        return nil, err
    }
    return e.buff.Bytes(), nil
}

func (c *gobCodec) Decode(b []byte, v interface{}) error {
    r := bytes.NewReader(b)
    dec := gob.NewDecoder(r)
    return dec.Decode(v)
}

but the performance is the same as without the benchmark

The Decode method is harder to move to sync.Pool because the decoder is stateful.

[bep]

OK, I managed to delete my last comment.

In the above I see:

• Get from the pool, but no Put. That should get about the same performance, no surprise there.
• You also need to make a copy of e.buff.Bytes() or you will get some surprises down the road.

[bep]

But it doesn't look like this is possible -- the encoding part of Gob is also very much stateful -- it keeps track of what typeinfo has been sent etc.

But what would be a improvement would be a streaming interface for collections.

[asdine]

This is a great idea but i can't imagine how it would be used

[bep]

This is a great idea but i can't imagine how it would be used

Me neither ... I will investigate further later, let us keep this issue open.

[bep]

Seems the "statefulness of Gob" has been discussed:

https://groups.google.com/forum/#!topic/golang-dev/Znc7ExcE1gw

[asdine]

Talking about streams, it may be interesting to have something like:

db.SetReader("mybucket", "mykey", r.Body)
db.GetReader("mybucket", "mykey", w)

GetReader would be less interesting though since bucket.Get returns a []byte

[pdf]

When I looked at storing gob-encoded data a while back, the primary driver was storage efficiency rather than perf - encoding the type information with every record actually makes gob significantly less space-efficient than JSON. The only sensible way to do it though is to prime the encoders and decoders by requiring users to register stored types before accessing the database.

Should be doable, but not sure if it's worth it overall.