differential-datalog icon indicating copy to clipboard operation
differential-datalog copied to clipboard

Fast serialization of sql records

Open mihaibudiu opened this issue 3 years ago • 8 comments

This commit introduces a more efficient way to move data from/to Java when using "SqlRecords" - these are structs with possibly nullable base types as fields. This is implemented by serializing the objects manually into byte[] and transferring the byte[] in a single JNI call, then deserializing it into a DDlogRecord. The reverse path is also supported.

This is a subset of what the flatbufs offers, but the flatbufs do not generate APIs to manipulate records, but rather collections.

Currently the SqlRecord lives in the src project, while the serialization support is in DDlogAPI. This is arguably wrong, we should choose one of the two locations. But this PR is here mostly to solicit an initial review.

We also need to support more base types and probably Tuples as well (needed for primary-key based operation).

The normal way to use the SqlRecord is shown in the test createREcordFromRelation:

        Translator t = this.createInputTables(false);
        DDlogProgram ddprogram = t.getDDlogProgram();
        DDlogRelationDeclaration decl = ddprogram.getRelation("Rt2");
        Assert.assertNotNull(decl);
        DDlogType type = decl.getType();
        DDlogTStruct ts = t.resolveType(type).to(DDlogTStruct.class);
        SqlRecord rec = ts.createEmptyRecord();
        rec.add(2L);

mihaibudiu avatar Nov 13 '20 01:11 mihaibudiu

@lalithsuresh : please take a look at this API. Note that using the DDlogAPI helpers requires the ddlog dynamic library to be already loaded, and this can happen only once. So the order of the tests is important.

mihaibudiu avatar Nov 13 '20 01:11 mihaibudiu

Thanks @mbudiu-vmw. I'll give this a spin.

lalithsuresh avatar Nov 13 '20 01:11 lalithsuresh

@lalithsuresh, @mbudiu-vmw , shall we close this PR for now?

ryzhyk avatar Nov 17 '20 18:11 ryzhyk

I expect this is still useful

mihaibudiu avatar Nov 17 '20 18:11 mihaibudiu

If you're ok maintaining this code, we can merge it.

ryzhyk avatar Nov 17 '20 19:11 ryzhyk

Let's first wait for Lalith's comments. Then, if he indeed is using it, it has to be generalized to handle more types. We may also consider moving it to another C file rather than having it in ddlogapi, but it is simpler to have only one of these.

mihaibudiu avatar Nov 17 '20 19:11 mihaibudiu

If @lalithsuresh is not using this we can perhaps keep it as a draft PR for a while until we figure out whether it will be ever used.

mihaibudiu avatar Nov 17 '20 19:11 mihaibudiu

I think we should keep it around as a draft PR for now. We'll get more data when I try out #817 in DCM (larger tables, each with more columns etc).

lalithsuresh avatar Nov 17 '20 19:11 lalithsuresh