ipfs
[MICROGRANT] <Rummage Search>
1. What is your project? (max 100 words)
Rummage is an open, collaborative, and decentralised search mechanism for IPFS. A rummage node is able to crawl content on IPFS and add this to the index, which itself is stored in a decentralised manner on IPFS.
Rummage currently supports parsing PDF's and any HTML files stored on IPFS.
How Rummage Works Rummage divides the crawled data into two levels of indices.
- High Level Index (HLI)
- Key Word Index (KWI)
High Level Index stores the CID for the KWI of each keyword, then making this a highly indexable mesh. This data is constantly updated when a new craw is submitted to the network
We allow for two specific actions on the client
- Search
- Crawl
Search will be used to perform sophisticated searches, while Crawl will be used to submit crawl data by clients
GITHUB LINK:
https://github.com/adekunle-oOo/rummage
2. How will IPFS, Filecoin, or related technology be used for this project? (max 100 words)
a) All crawl data submitted will be dumped on IPFS b) Searches performed, will be via indexing the data stored on IPFS. We access the data using the go-ipfs driver c) The CID's given by IPFS acts as the primary index for structured query searches, which enhances the platform usability
3. How will you improve your project with this grant? What steps will you take to meet this objective? (max 200 words)
a) Currently, rummage only supports crawling PDF and HTML files. I'm developing features to include other file types as well b) I'm also looking to add support to other languages for running the node, such as NodeJS. c) Will begin working on features which improve the security measures of Rummage this will also include code audits.
4. Is this project open source?
Yes I have opensourced the project and GPL licensed it
5. Do you agree to share grant reports upon request, including a final grant report at the end of the three month period?
Yes, I will
6. Does your proposal comply with our Community Code of Conduct?
Yes
7. Links and submissions
- If your project began at a hackathon, have you submitted it for the relevant Protocol Labs prizes? Include links here if available:
Additional questions:
-
For each team member(s), please list name, email, Github account, and role in the project. Adekunle Oludele (FULL NAME) [email protected] (EMAIL) https://github.com/adekunle-oOo (GITHUB)
-
How did you learn about our microgrant program? - Online
Are you going to be doing anything for "ranking" search results as part of this microgrant?
@RangerMauve Hello, I have developed a very trivial ranking system which is based on CID's. But I want to develop a more sophisticated ranking algorithm on the client side, which can give more autonomy to the clients. I am hoping that I can start a community discussion for this and develop this feature in the upcoming months.
That's cool, in the future it might be useful to collab using the prolly tree work we're standardizing in #194 to make the indexes perform well at large scales and used ordered searching to make some algorithms easier.
Hi @RangerMauve . It was fascinating to learn about prolly trees. I will surely look into how I can leverage the same. I'll keep in touch! Thanks again
@ErinOCon Greetings! Where can I learn about the updates regarding my application? will they be given over email?
Thanks, Adekunle Oludele
@ErinOCon The protoype is now progressing would love the feedback on design as I myself designed it.
Hi @adekunle-oOo, apologies for the significant delay in reply. Thank you for the provided updates! Do you, by chance, have a link to a demo or website?
Hi @adekunle-oOo, it looks like there are elements of the deece code base reflected in your current repo. Can you identify the transformative factors that will differentiate the two projects?
Hi @adekunle-oOo, I hope all is well! Do you expect an update will be available next week?
Hi @adekunle-oOo, I have closed this item for now, but please feel welcome to re-open to provide more information. Thank you again for your time with this proposal!