guava icon indicating copy to clipboard operation
guava copied to clipboard

BloomFilter Serialization

Open matebase opened this issue 2 years ago • 1 comments

BloomFilter size keeps increasing when serializing and deserializing with kryo ?

Below is the relevant code

import com.esotericsoftware.kryo.kryo5.Kryo;
import com.esotericsoftware.kryo.kryo5.io.Input;
import com.esotericsoftware.kryo.kryo5.io.Output;
import com.esotericsoftware.kryo.kryo5.objenesis.strategy.StdInstantiatorStrategy;
import com.esotericsoftware.kryo.kryo5.util.DefaultInstantiatorStrategy;
import com.esotericsoftware.kryo.kryo5.util.Pool;
import com.google.common.hash.BloomFilter;
import com.google.common.hash.Funnels;

protected byte[] kryoSerialize(Object obj) {
    Kryo kryo = kryoPool.obtain();
    try {
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        Output output = new Output(byteArrayOutputStream);
        kryo.writeObject(output, obj);
        output.close();
        return byteArrayOutputStream.toByteArray();
    } catch (Exception e) {
        return null;
    } finally {
        kryoPool.free(kryo);
    }
}

protected <T> T kryoDeserialize(byte[] bytes, Class<T> clazz) {
    Kryo kryo = kryoPool.obtain();
    try {
        kryo.setInstantiatorStrategy(new DefaultInstantiatorStrategy(new StdInstantiatorStrategy()));
        ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
        Input input = new Input(byteArrayInputStream);
        input.close();
        return (T) kryo.readObject(input, clazz);
    } catch (Exception e) {
        return null;
    } finally {
        kryoPool.free(kryo);
    }
}

Pool<Kryo> kryoPool = new Pool<Kryo>(true, false, 8) {
    @Override
    protected Kryo create() {
        Kryo kryo = new Kryo();
        kryo.setReferences(true);
        kryo.setRegistrationRequired(false);
        return kryo;
    }
};

matebase avatar Jun 02 '22 09:06 matebase

Could you be clearer about how you're detecting this size increase? I see you've included adapter code to plug into Kryo, but nothing that specifically reproduces an issue. I also suggest that your minimum reproducible example use normal Java serialization (Object(Input|Output)Stream), rather than Kryo. If you can't reproduce the problem with normal serialization, then the issue would appear to be with Kryo rather than with BloomFilter.

amalloy avatar Jun 10 '22 20:06 amalloy