Index Dynamic Fields

[hirik]

Hi, Is it possible to query Map inside an object,

private final Long id; private final Map<Long, String> keyValue;

How to convert the below statement to IndexedCollection Query, please help me to figure this out, TIA

id > 10 && keyValue.get(2L).equals("test")

[npgall]

Yes. You can find somewhat relevant examples for that here and here.

Let me know if you need more info though.

[hirik]

Hey, this is very similar to what I'm looking for. I'm looking to solve a case where there are two levels of objects and I need to query referring the first and second level.

Example class Event { private final Long id; private final Map<String, Object> dynamicInput; }

id == 10 && dynamicInput.get("type").equals("realtime") or id == 10 && dynamicInput.get("target").equals(100_000L)

From the above examples, I can derive only the following condition. If I missed anything please let me know. TIA

id == 10 && dynamicInput.values().contains("realtime")

[hirik]

@npgall I found a way to solve this. But it is taking much time than before

My Sample code

public static void main(String[] args)
    {
        final long iterations = 1_000;
        final long entries = 1_000_000;

        final IndexedCollection<Entry> indexedCollection = new ConcurrentIndexedCollection<>();
        indexedCollection.addIndex(NavigableIndex.onAttribute(Entry.TIME));
        indexedCollection.addIndex(HashIndex.onAttribute(Entry.FIELDS));

        long time = System.currentTimeMillis();
        for(long index = 0; index < entries; index++)
        {
            final List<Field> field = Arrays.asList(new Field(1L, "test"+index), new Field(2L, "test"));
            indexedCollection.add(new Entry(System.currentTimeMillis(), field));
        }
        System.out.println("Time Taken to Insert: "+ (System.currentTimeMillis() - time));

        int count = 0;
        final And<Entry> query = and(lessThan(Entry.TIME, System.currentTimeMillis()), equal(Entry.FIELDS, new Field(1L, "test1")), equal(Entry.FIELDS, new Field(2L, "test")));
        time = System.currentTimeMillis();
        for(long index = 0; index < iterations; index++)
        {
            count += indexedCollection.retrieve(query).size();
        }
        System.out.println("Time Taken : "+ (System.currentTimeMillis() - time)+ "  Count : "+count);
    }

public class Entry
{
    private final Long time;
    private final List<Field> fields;

    Entry(Long time, List<Field> fields)
    {
        this.time = time;
        this.fields = fields;
    }

    public long getTime()
    {
        return time;
    }

    static final Attribute<Entry, Field> FIELDS = new MultiValueAttribute<>("fields")
    {
        public Iterable<Field> getValues(Entry data, QueryOptions queryOptions) { return data.fields; }
    };

    static final Attribute<Entry, Long> TIME = new SimpleAttribute<>("time")
    {
        public Long getValue(Entry data, QueryOptions queryOptions) { return data.time; }
    };

    public List<Field> getFields()
    {
        return fields;
    }
}

public class Field
{
    private final Long fieldId;
    private final Object value;
    private int hashCode = 0;
    public Field(Long fieldId, Object value)
    {
        this.fieldId = fieldId;
        this.value = value;
    }

    public Long getFieldId()
    {
        return fieldId;
    }

    public Object getValue()
    {
        return value;
    }

    @Override
    public boolean equals(Object obj)
    {
        if(obj instanceof Field)
        {
            return this == obj || this.hashCode() == obj.hashCode();
        }
        else
        {
            return false;
        }
    }

    @Override
    public int hashCode()
    {
        if(hashCode == 0)
        {
            final int prime = 31;
            int result = 1;
            result = prime * result + fieldId.hashCode();
            result = prime * result + value.hashCode();
            this.hashCode = result;
        }
        return hashCode;
    }
}

Result :

Time Taken to Insert: 4870 Time Taken : 475 Count : 1000

If I missed anything please let me know. btw thanks for this wonderful library!!!

[npgall]

I took a look at that code, but it's quite a complex snippet. If you could isolate the problem to a simpler example it would help.

Another thing which comes to mind, is that if you are trying to store unstructured data in a collection using dynamic maps or lists: the computation of map/list.hashCode() and map/list.equals() on those maps/lists can be a performance killer. And CQEngine will call those methods intensively during query evaluation.

There is some support to optimize that use case. Take a look at the documentation for MapEntity and PrimaryKeyedMapEntity for details.

[sdvdxl]

This issue may be easier to resolve if it supports automatic registration through conditional extraction parameters. For example, if you have a map, you may need to dynamically set the filter condition select * from map where a='1' and b=1 and xx='xx', expect to be able to automatically extract a, b, xx parameters automatically registered to the parser. Otherwise, you can only register yourself, as in the following example:

SQLParser parser = SQLParser.forPojo(Map.class); 
parser.registerAttribute(QueryFactory.mapAttribute("a",String.class)); 
// other attribute

If parser.setAutoRegisterAttribute(true), then auto register attribute a,b,xx. Maybe also need parser.setExceptionWhenTypeNotMatch,or parser.setAutoConvertTypeWhenNotMatch

中文说明：

如果支持通过条件提取参数自动注册，这个问题可能会比较容易解决。 比如有一个map，可能需要动态设置过滤条件 select * from map where a='1' and b=1 and xx='xx'，期望能够自动提取 a，b，xx参数自动注册到parser里面 否则的话只能自行注册