serialize or deserilaze the Roaring64NavigableMap

[panda0120]

bug reproduce: when the size of Roaring64NavigableMap is bigger than 32, and some values are smaller than Integer.MAX , and some values are bigger than Integer.MAX, this bigger ones will change to unknown numbers.

the table's ddl like this: ` CREATE TABLE default.index_bitmap64_mergetree (

`index_name` String ,

`index_val` String ,

`cst_bitmap` AggregateFunction(groupBitmap,

UInt64) ) ENGINE = MergeTree PARTITION BY index_name PRIMARY KEY (index_name, index_val) ORDER BY (index_name, index_val) SETTINGS index_granularity = 8192 `

the test class like this: ` import com.clickhouse.client.ClickHouseDataType; import com.clickhouse.client.data.ClickHouseBitmap; import com.clickhouse.jdbc.ClickHouseDataSource; import org.junit.Assert; import org.roaringbitmap.longlong.Roaring64NavigableMap;

import java.io.IOException; import java.sql.Connection; import java.sql.PreparedStatement; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Statement; import java.util.Properties;

public class JdbcDemoTest { static long[] SOURCE_DATA = new long[]{123356317L,164762885L,200762117L,224309989L,327028679L,329495408L,356325249L,392030441L,531586461L,544972938L,562184558L,574223351L,627899306L,681918149L,682107622L,683384650L,706089828L,755775908L,790859820L,798600351L,902777797L,1097633181L,1157136484L,1194580949L,1241717591L,1259292504L,1269453760L,1346239972L,1346646629L,1434537481L,1460662828L,1536566697L,1560002982L,1593131717L,1621740170L,1822436116L,1848496543L,9158129537L,9251136255L,9503335505L};

public static void main(String[] args) throws SQLException, IOException {
    String url = "jdbc:clickhouse://xxx:8123";
    ClickHouseDataSource dataSource = officialJdbcDataSource(url);
    truncateBitMap64(dataSource);

    insertBitMap64(dataSource);

    searchBitmap64(dataSource);
}

public static ClickHouseDataSource officialJdbcDataSource(String url) throws SQLException {
    Properties properties = new Properties();
    return new ClickHouseDataSource(url, properties);
}



public static void searchBitmap64(ClickHouseDataSource dataSource) throws IOException {
    String sql = "" +
            " select " +
            "   index_name," +
            "   index_val," +
            "   length(bitmapToArray(cst_bitmap)) as bitmap_count, " +
            "   cst_bitmap " +
            " from index_bitmap64_mergetree " +
            " order by index_name, index_val desc limit 1";
    System.out.println(sql);
    try (Connection connection = dataSource.getConnection("default", "");
         Statement statement = connection.createStatement();
         ResultSet resultSet = statement.executeQuery(sql)) {
        while (resultSet.next()) {
            String indexName = resultSet.getString(1);
            String indexVal = resultSet.getString(2);
            int bitmapCount = resultSet.getInt(3);
            Roaring64NavigableMap rnm = (Roaring64NavigableMap) resultSet.getObject(4, ClickHouseBitmap.class).unwrap();
            Assert.assertArrayEquals(SOURCE_DATA, rnm.toArray());
            System.out.println("-----------------------------------------------------");
        }
    } catch (SQLException e) {
        e.printStackTrace();
    }

}


public static void insertBitMap64(ClickHouseDataSource dataSource) {
    Roaring64NavigableMap rnm = new Roaring64NavigableMap();
    rnm.add(SOURCE_DATA);
    try (Connection connection = dataSource.getConnection("default", "");
        PreparedStatement preparedStatement = connection.prepareStatement("insert into index_bitmap64_mergetree values(?,?,?)")) {
        preparedStatement.setString(1, "gdr");
        preparedStatement.setString(2,"gdr");
        preparedStatement.setObject(3, ClickHouseBitmap.wrap(rnm, ClickHouseDataType.UInt64));
        preparedStatement.execute();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

public static void truncateBitMap64(ClickHouseDataSource dataSource) {
    try (Connection connection = dataSource.getConnection("default", "");
         PreparedStatement preparedStatement = connection.prepareStatement("truncate table index_bitmap64_mergetree")) {
        preparedStatement.execute();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

}

`

[zhicwu]

Hi @panda0120, sorry I probably accidentally closed #1031. I think it's better to fix the issue in RoaringBitmap Java lib, instead of adding a workaround in JDBC driver. Will RoaringBitmap/RoaringBitmap#572 be merged soon? In case there's concern or the change is specific for ClickHouse, we can create a patched version of RoaringBitmap at here.

[panda0120]

Hi @panda0120, sorry I probably accidentally closed #1031. I think it's better to fix the issue in RoaringBitmap Java lib, instead of adding a workaround in JDBC driver. Will RoaringBitmap/RoaringBitmap#572 be merged soon? In case there's concern or the change is specific for ClickHouse, we can create a patched version of RoaringBitmap at here.

Merge the commit #572 can't avoid to modify the deserialize method of JDBC driver. In addition, the commit #572 need more code to be compatible with c, as the jdbc driver has done. Otherwise, it's not realy compatible with c.

My OS is windows, now there is a exception when i run the test cases of RoaringBitmap. I cant't resolve it. So maybe i will try to run the test cases on linux.