vor-knowledge-graph

vor-knowledge-graph copied to clipboard

New link [Set cover problem] HAS => [25th] ['Graz', 'Austria'] Adding : Set cover problem ===> ['Graz', 'Austria'] New link [Set cover problem] HAS => [Graz] New link [Set cover problem] HAS => [Austria] ['doi'] Adding : Set cover problem ===> ['doi'] New link [Set cover problem] HAS => [doi] ['related', 'media', 'Set', 'Setcover', 'links', 'cover', 'problem', 'Commons', 'coverproblem'] Adding : Set cover problem ===> ['related', 'media', 'Set', 'Setcover', 'links', 'cover', 'problem', 'Commons', 'coverproblem'] New link [Set cover problem] HAS => [related] New link [Set cover problem] HAS => [media] New link [Set cover problem] HAS => [Set] New link [Set cover problem] HAS => [Setcover] New link [Set cover problem] HAS => [links] New link [Set cover problem] HAS => [cover] New link [Set cover problem] HAS => [problem] New link [Set cover problem] HAS => [Commons] New link [Set cover problem] HAS => [coverproblem] ['Solutions', 'Set', 'Benchmarks', 'WinnerDetermination', 'NPoptimization', 'Cover', 'Covering', 'HiddenOptimum', 'optimization', 'SetCover', 'CoveringSet', 'Hidden', 'Determinationcompendium', 'Winner', 'Minimum', 'MinimumSet', 'Optimum', 'Determination', 'compendium', 'problems'] Adding : Set cover problem ===> ['Solutions', 'Set', 'Benchmarks', 'WinnerDetermination', 'NPoptimization', 'Cover', 'Covering', 'HiddenOptimum', 'optimization', 'SetCover', 'CoveringSet', 'Hidden', 'Determinationcompendium', 'Winner', 'Minimum', 'MinimumSet', 'Optimum', 'Determination', 'compendium', 'problems'] New link [Set cover problem] HAS => [Solutions] New link [Set cover problem] HAS => [Set] New link [Set cover problem] HAS => [Benchmarks] New link [Set cover problem] HAS => [WinnerDetermination] New link [Set cover problem] HAS => [NPoptimization] New link [Set cover problem] HAS => [Cover] New link [Set cover problem] HAS => [Covering] New link [Set cover problem] HAS => [HiddenOptimum] New link [Set cover problem] HAS => [optimization] New link [Set cover problem] HAS => [SetCover] New link [Set cover problem] HAS => [CoveringSet] New link [Set cover problem] HAS => [Hidden] New link [Set cover problem] HAS => [Determinationcompendium] New link [Set cover problem] HAS => [Winner] New link [Set cover problem] HAS => [Minimum] New link [Set cover problem] HAS => [MinimumSet] New link [Set cover problem] HAS => [Optimum] New link [Set cover problem] HAS => [Determination] New link [Set cover problem] HAS => [compendium] New link [Set cover problem] HAS => [problems] Set cover problem processed with 534 nodes. 34 wiki documents processed so far... Traceback (most recent call last): File "build_knowledge.py", line 113, in for topic,sentence in iter_topic(crawl_collection,args['start']): File "build_knowledge.py", line 41, in iter_topic for wiki in crawl_collection.query({'downloaded': True},field=None,skip=start): File "/home/mldl/ub16_prj/vor-knowledge-graph/pylib/knowledge/datasource.py", line 21, in query for n in query: File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1189, in next if len(self.__data) or self._refresh(): File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1126, in _refresh self.__send_message(g) File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 978, in __send_message codec_options=self.__codec_options) File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1067, in _unpack_response return response.unpack_response(cursor_id, codec_options) File "/usr/local/lib/python3.5/dist-packages/pymongo/message.py", line 1418, in unpack_response self.raw_response(cursor_id) File "/usr/local/lib/python3.5/dist-packages/pymongo/message.py", line 1384, in raw_response raise CursorNotFound(msg, 43, errobj) pymongo.errors.CursorNotFound: Cursor not found, cursor id: 22364095374 mldl@ub1604:~/ub16_prj/vor-knowledge-graph$ mldl@ub1604:~/ub16_prj/vor-knowledge-graph$

Hello @loveJasmine Two fun facts might surprise you.

1. The issue is caused by a too-heavy concurrent data loading from MongoDB. So the cursor is busy when it iterates to the next task and it ends up having an error message saying CursorNotFound.

2. The workaround to the issue has already been addressed in another outstanding branch I'm working on but it's on halt. https://github.com/starcolon/vor-knowledge-graph/blob/feature/content-segmentation/pylib/knowledge/datasource.py#L20

So I would recommend you to take the extra options you see on the line as linked above to your code. Run it again it will go fine.

mldl@ub1604:~/ub16_prj/vor-knowledge-graph$ python3 build_knowledge.py --verbose --root mypass --limit 10 Initialising knowledge graph database... Connecting OrientDB: vor [WARNING] Preparing graph schema com.orientechnologies.orient.core.exception.OSchemaException - Class 'TOPIC' already exists in current database DB name="vor" [Graph clearance] done Loading POS patterns... Loading stopwords... Initialising wikipedia crawling collection... Iterating over crawling database... mldl@ub1604:~/ub16_prj/vor-knowledge-graph$

Does it looks right?

and more: mldl@ub1604:~/ub16_prj/vor-knowledge-graph$ python3 build_wordvec.py --limit 10 --out out_model Exporting crawled data to text file... [Done] Training word2vec... b'Starting training using file /home/mldl/ub16_prj/vor-knowledge-graph/mine.txt\n'b'Vocab size: 1\n'b'Words in train file: 0\n'[Done] Word2Vec model is saved at : /home/mldl/ub16_prj/vor-knowledge-graph/out_model [Word2Vec model spec] ... Model shape : (1, 10) [Word2Vec REPL]: Enter a word to try: hello Traceback (most recent call last): File "build_wordvec.py", line 102, in repl(model) File "build_wordvec.py", line 87, in repl indexes, metrics = model.cosine(w) File "/usr/local/lib/python3.5/dist-packages/word2vec/wordvectors.py", line 76, in cosine metrics = np.dot(self.vectors, self[word].T) File "/usr/local/lib/python3.5/dist-packages/word2vec/wordvectors.py", line 45, in getitem return self.get_vector(word) File "/usr/local/lib/python3.5/dist-packages/word2vec/wordvectors.py", line 54, in get_vector idx = self.ix(word) File "/usr/local/lib/python3.5/dist-packages/word2vec/wordvectors.py", line 42, in ix return self.vocab_hash[word] KeyError: 'hello' mldl@ub1604:~/ub16_prj/vor-knowledge-graph$

Warnings are fine and ignorable. The error on the bottom part means you are querying for the word which doesn't exist in the vector space. I would suggest you to try inputting a word picked from the content you fetched.

the error raised yet:

Set cover problem processed with 534 nodes. 35 wiki documents processed so far... Traceback (most recent call last): File "build_knowledge.py", line 113, in for topic,sentence in iter_topic(crawl_collection,args['start']): File "build_knowledge.py", line 41, in iter_topic for wiki in crawl_collection.query({'downloaded': True},field=None,skip=start): File "/home/mldl/ub16_prj/vor-knowledge-graph/pylib/knowledge/datasource.py", line 21, in query for n in query: File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1189, in next if len(self.__data) or self._refresh(): File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1126, in _refresh self.__send_message(g) File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 978, in __send_message codec_options=self.__codec_options) File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1067, in _unpack_response return response.unpack_response(cursor_id, codec_options) File "/usr/local/lib/python3.5/dist-packages/pymongo/message.py", line 1418, in unpack_response self.raw_response(cursor_id) File "/usr/local/lib/python3.5/dist-packages/pymongo/message.py", line 1384, in raw_response raise CursorNotFound(msg, 43, errobj) pymongo.errors.CursorNotFound: Cursor not found, cursor id: 30134005583 mldl@ub1604:~/ub16_prj/vor-knowledge-graph$

as suggested, have you tried updating the query function in pylib/knowledge/datasource.py#L20 so it becomes?:

query = self.src.find(conditions) if skip==0 else self.src.find(filter=conditions,skip=skip,no_cursor_timeout=True)

With the cursor timeout option added in you should no longer see such error message.

sure , I did it

interesting. can you please let me know your mongodb version?