Added support for weaviate vector database

[d-vignesh]

• Added support for Weaviate vector database
• Implemented the functionality of document insertion and incremental insertion.

Tested the changes locally using weaviate cloud instance.

Initial Insertion(Added two files, file1 and file2):

Incremental Insertion(Added two more file, file3 and file4):

Weaviate dashboard(4 objects inserted to the collection):

[mnvsk97]

Hello @d-vignesh Please rebase with the latest main version and update the PR. If this is still relevant, we can go ahead and review. Thanks!

[tapishr]

This is a visualization of the code changes in the PR, generated using Vibinex. Nodes are files/functions/objects. Edges denote function calls.

Call Stack Diff

%%{init: { 'theme': 'neutral', 'themeVariables': { 'fontSize': '20px' }, 'flowchart': { 'nodeSpacing': 100, 'rankSpacing': 100 } } }%%
 	flowchart LR
	subgraph snln ["backend/modules/vector_db/weaviate.py"]
	foci["debug"]

	click foci href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/backend/modules/vector_db/weaviate.py#L0" _blank
	jmue["WeaviateVectorDB"]
	class jmue modified
	click jmue href "https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R23" _blank
	end
	class snln modified

	subgraph bplz ["my_lib/utils.py"]
	mwgd["connect_to_local"]

	click mwgd href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/my_lib/utils.py#L0" _blank
	fyaq["Client"]

	click fyaq href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/my_lib/utils.py#L0" _blank
	end


	subgraph fzbb ["my_lib/models.py"]
	pcmy["_get_records_to_be_updated"]

	click pcmy href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/my_lib/models.py#L0" _blank
	fyjh["create_class"]

	click fyjh href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/my_lib/models.py#L0" _blank
	hreq["res.get"]

	click hreq href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/my_lib/models.py#L0" _blank
	end


	subgraph ducq ["weaviate/auth.py"]
	ugei["AuthApiKey"]

	click ugei href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/weaviate/auth.py#L0" _blank
	end


	subgraph mgnc ["weaviate/classes.py"]
	ctyt["collections.create"]

	click ctyt href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/weaviate/classes.py#L0" _blank
	nkvn["delete_many"]

	click nkvn href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/weaviate/classes.py#L0" _blank
	ozvf["keys"]

	click ozvf href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/weaviate/classes.py#L0" _blank
	end


	subgraph vvhg ["cognita/backend/modules/vector_db/weaviate.py"]
	wylz["api_key"]

	click wylz href "https://github.com/Alokit-Innovations-Test/cognita/blob/f6ff788038f583d6ed5791141151c9123cb3e663/cognita/backend/modules/vector_db/weaviate.py#L0" _blank
	end


	classDef modified stroke:black,fill:yellow
	classDef added stroke:black,fill:#b7e892,color:black
	classDef deleted stroke:black,fill:red
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983L34'>Line 34</a>" =====>fyjh
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R119'>Line 119</a>" =====>foci
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R96'>Line 96</a>" =====>foci
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R30'>Line 30</a>" =====>wylz
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R133'>Line 133</a>" =====>ozvf
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R124'>Line 124</a>" =====>nkvn
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R35'>Line 35</a>" =====>ctyt
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R47'>Line 47</a>" =====>pcmy
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983R26'>Line 26</a>" =====>mwgd
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983L23'>Line 23</a>" =====>fyaq
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983L125'>Line 125</a>" =====>hreq
	jmue =="<a target='_blank' href='https://github.com/Alokit-Innovations-Test/cognita/pull/4/files#diff-03072e343ca60d43bffe05672b8c46a0767628a2f0d5a6e41fd4a61c26006983L25'>Line 25</a>" =====>ugei

	linkStyle 1,2,3,4,5,6,7,8 stroke:green,stroke-width:8
	linkStyle 0,9,10,11 stroke:red,stroke-width:10

To modify DiffGraph settings, go to your Vibinex settings page.

[mnvsk97]

Closing this PR as it is outdated. We will open a new one when required.