HugeCollections-OLD

HugeCollections-OLD copied to clipboard

Hi everyone,

i am having an issue with my HugeHashMap and Database persist. In fact, when i persist my objects from the HugeHashMap i don't get what i was expecting.

I have an Instance that contains an id, a word context as a String, some positions as integers,.. And instead of having the values i got this for each Instance in HugeHashMap: id="table.Instance[ id=000171fe-15a2-46fd-97eb-957d42940f8a ]" Auto="0" Context="" position1="0" position2="0" position3="0" position4="0" position5="0" foreignId=""

Here is the code where i manage the HugeHashMap:

public void generateInstances(String path) throws IOException, ParseException, InvalidTokenOffsetsException{
        HugeConfig config = HugeConfig.DEFAULT.clone();
        HugeHashMap<String, Instance> instanceArray = new HugeHashMap<String, Instance>(config,String.class,Instance.class);
        HugeHashMap<Integer,TermeinstancedocumentPK> tidArray = new HugeHashMap<Integer,TermeinstancedocumentPK>(config,Integer.class,TermeinstancedocumentPK.class);
        ArrayList<Context> contexesArray = new ArrayList<>();
        Integer idtid=1;
        DocumentIndexer lucene = new DocumentIndexer(path);
        Project p = this.getProject();        
        for(Notion n : p.getNotionCollection()){
            for (Term t : n.getTermCollection()) {
                for (Variable v : t.getVariableCollection()) {
                    contexesArray = lucene.getContextes(v.getName(),90);
                    for(Context c : contexesArray){
                        boolean instanceAlreadyExists=false;
                        String rightContext = c.getRightContext();
                        String[] contextRight = rightContext.split("\n");
                        StringJoiner joiner = new StringJoiner(" | ");
                        for(String s: contextRight){
                            joiner.add(s);
                        }
                        rightContext = joiner.toString();
                        String leftContext = c.getLeftContext();
                        int leftContextSizeBefore=leftContext.length();
                        String[] contextLeft = leftContext.split("\n");
                        StringJoiner joiner2 = new StringJoiner(" | ");
                        for(String s: contextLeft){
                            joiner2.add(s);
                        }
                        leftContext = joiner2.toString();
                        int leftContextSizeAfter=leftContext.length();
                        int relativeLeftContextSize = leftContextSizeAfter-leftContextSizeBefore;
                        String context = leftContext + c.getTerm() + rightContext;
                        Instance i = new Instance();
                        if(instanceArray.containsKey(context)){
                                instanceAlreadyExists=true;
                                i=instanceArray.get(context);
                        }
                        if(!instanceAlreadyExists){
                            String idinst = UUID.randomUUID().toString();
                            i.setId(idinst);
                            i.setContexte(context);
                            i.setPositionBeginTermeInstance(c.getStartTerm()+relativeLeftContextSize);
                            i.setPositionEndTermeInstance(c.getEndTerm()+relativeLeftContextSize);
                            i.setPositionBeginTermeTxtBrut(c.getStartTerm());
                            i.setPositionEndTermeTxtBrut(c.getEndTerm());
                            i.setContexteSize(c.getContext().length());
                            i.setProjectId(p);
                            if((p.getNbWordMinForAuto()!=-1 &&(this.getNbWord(v.getName())>p.getNbWordMinForAuto()))
                                | (p.getNbCharMinForAuto()!=-1 && (this.getNbChar(v.getName())>p.getNbCharMinForAuto()))){
                                i.setAuto(true);
                            }
                            TermeinstancedocumentPK tidPK = new TermeinstancedocumentPK(t.getId(),c.getUuid(), idinst,c.getIndexStart());
                            instanceArray.put(context,i);
                            tidArray.put(idtid,tidPK);
                            idtid++;
                        }
                        else{
                            if((p.getNbWordMinForAuto()!=-1 &&(this.getNbWord(v.getName())>p.getNbWordMinForAuto()))
                                | (p.getNbCharMinForAuto()!=-1 && (this.getNbChar(v.getName())>p.getNbCharMinForAuto()))){
                                instanceArray.get(context).setAuto(true);
                            }
                            TermeinstancedocumentPK tidPK = new TermeinstancedocumentPK(t.getId(),c.getUuid(), i.getId(),c.getIndexStart());
                            tidArray.put(idtid,tidPK);
                            idtid++;
                        }
                    }
                contexesArray.clear();
                }
            }
        }
        lucene.close();
        this.addInstances(instanceArray);
        this.addTIDs(tidArray);
        instanceArray.clear();
        tidArray.clear();
    }

//the method to persist is like that
public void addInstances(HugeHashMap<String,Instance> ints){
        this.open();
        em.getTransaction().begin();
        for(Instance inst : ints.values()){
            em.persist(inst); 
        }
        em.getTransaction().commit();
        this.close();
    }

Do you have any idea to solve this problem?

Thanks

SharedHashMap had a bug in it's iterator, perhaps HugehashMap does as well. We will check this.

On 20 August 2014 19:54, Mignastor [email protected] wrote:

Hi everyone,

i am having an issue with my HugeHashMap and Database persist. In fact, when i persist my objects from the HugeHashMap i don't get what i was expecting.

I have an Instance that contains an id, a word context as a String, some positions as integers,.. And instead of having the values i got this for each Instance in HugeHashMap: id="table.Instance[ id=000171fe-15a2-46fd-97eb-957d42940f8a ]" Auto="0" Context="" position1="0" position2="0" position3="0" position4="0" position5="0" foreignId=""

Here is the code where i manage the HugeHashMap:

public void generateInstances(String path) throws IOException, ParseException, InvalidTokenOffsetsException{ HugeConfig config = HugeConfig.DEFAULT.clone(); HugeHashMap<String, Instance> instanceArray = new HugeHashMap<String, Instance>(config,String.class,Instance.class); HugeHashMap<Integer,TermeinstancedocumentPK> tidArray = new HugeHashMap<Integer,TermeinstancedocumentPK>(config,Integer.class,TermeinstancedocumentPK.class); ArrayList<Context> contexesArray = new ArrayList<>(); Integer idtid=1; DocumentIndexer lucene = new DocumentIndexer(path); Project p = this.getProject(); for(Notion n : p.getNotionCollection()){ for (Term t : n.getTermCollection()) { for (Variable v : t.getVariableCollection()) { contexesArray = lucene.getContextes(v.getName(),90); for(Context c : contexesArray){ boolean instanceAlreadyExists=false; String rightContext = c.getRightContext(); String[] contextRight = rightContext.split("\n"); StringJoiner joiner = new StringJoiner(" | "); for(String s: contextRight){ joiner.add(s); } rightContext = joiner.toString(); String leftContext = c.getLeftContext(); int leftContextSizeBefore=leftContext.length(); String[] contextLeft = leftContext.split("\n"); StringJoiner joiner2 = new StringJoiner(" | "); for(String s: contextLeft){ joiner2.add(s); } leftContext = joiner2.toString(); int leftContextSizeAfter=leftContext.length(); int relativeLeftContextSize = leftContextSizeAfter-leftContextSizeBefore; String context = leftContext + c.getTerm() + rightContext; Instance i = new Instance(); if(instanceArray.containsKey(context)){ instanceAlreadyExists=true; i=instanceArray.get(context); } if(!instanceAlreadyExists){ String idinst = UUID.randomUUID().toString(); i.setId(idinst); i.setContexte(context); i.setPositionBeginTermeInstance(c.getStartTerm()+relativeLeftContextSize); i.setPositionEndTermeInstance(c.getEndTerm()+relativeLeftContextSize); i.setPositionBeginTermeTxtBrut(c.getStartTerm()); i.setPositionEndTermeTxtBrut(c.getEndTerm()); i.setContexteSize(c.getContext().length()); i.setProjectId(p); if((p.getNbWordMinForAuto()!=-1 &&(this.getNbWord(v.getName())>p.getNbWordMinForAuto())) | (p.getNbCharMinForAuto()!=-1 && (this.getNbChar(v.getName())>p.getNbCharMinForAuto()))){ i.setAuto(true); } TermeinstancedocumentPK tidPK = new TermeinstancedocumentPK(t.getId(),c.getUuid(), idinst,c.getIndexStart()); instanceArray.put(context,i); tidArray.put(idtid,tidPK); idtid++; } else{ if((p.getNbWordMinForAuto()!=-1 &&(this.getNbWord(v.getName())>p.getNbWordMinForAuto())) | (p.getNbCharMinForAuto()!=-1 && (this.getNbChar(v.getName())>p.getNbCharMinForAuto()))){ instanceArray.get(context).setAuto(true); } TermeinstancedocumentPK tidPK = new TermeinstancedocumentPK(t.getId(),c.getUuid(), i.getId(),c.getIndexStart()); tidArray.put(idtid,tidPK); idtid++; } } contexesArray.clear(); } } } lucene.close(); this.addInstances(instanceArray); this.addTIDs(tidArray); instanceArray.clear(); tidArray.clear(); } //the method to persist is like thatpublic void addInstances(HugeHashMap<String,Instance> ints){ this.open(); em.getTransaction().begin(); for(Instance inst : ints.values()){ em.persist(inst); } em.getTransaction().commit(); this.close(); }

Do you have any idea to solve this problem?

Thanks

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/HugeCollections/issues/41.

Hi,

Thanks for your quick answer.

I made some tests and it appears that i can't iterate throught the HugeHashMap. To solve this problem i had to put an Integer as first parameter and do like this:

HugeHashMap<Integer, HashMap<String,String>>...

int size=ints.size();
for(int i=1;i<size;i++){
     HashMap<String,String> instanceMap=ints.get(i);
            ...

Also i had to replace JPA items by an HashMap<String,String> with each Key, value i need to create the JPA Object. Then, i create all objects when i iterate the HugeHashMap and persist them. In fact, JPA Objects are not restored correctly from HugeHashMap by the get function and that is why i had wrong values. Do you have any idea why or is it a problem with JPA?

Thanks

Thanks for your feedback regarding the iterator, we will take a look to see if we can fix this issue, just as an aside we are no longer doing any new development on HugeHashHap, we have moved our development focus to ChronicleMap, which ( apart from resizing capability should offer you most of the functionality that you get with HugeHashMap ), it should also not have the iterator issue that you identified above.

Regarding your JPA issue, HugeHashMap stores the keys and values off heap, so the JPA object will undergo serialization, The problems with JPA maybe down to serialization.

we have raised the following JIRA to look at the iterator issue see https://higherfrequencytrading.atlassian.net/browse/HCOLL-126