Olicorne
Olicorne
Hi, In the process of teaching me how the crawler works, I tend to write missing doc using LLMs as checking is far faster than understanding from raw code. Hence...
This is to track this as mentionned in #242
Hi, Apparently the crawler container is using root and I don't see any reason for that. This is a security issue.
> there's a very noticeable lack of diagrams in the entire documentation. Hence probably some present passed and future misunderstandings. I believe vision LLMs are good enough now to speed...
As mentioned in #242, this issue is to keep track of this: > there is no way for the main server to send messages to the crawler. Wouldn't that be...
This issue is to track down the completion and discussion around a specific feature. All thoughts here are my own and not necessarily the maintainer's. Right now, not much thoughts...
If you move more and more of the code from the server to the client (WASM etc), doesn't that paves the way for a truly distributed search engine of both...
Some interesting datascience questions raised during email exchanged with the maintainer: -what is the distribution of how often a TinyIndex page gets updated? Meaning the crawler uploaded a better data...
Currently when we run the old crawler we receive URLs to crawl form the master node. Now the crawler gets its own URL apparently. How are those URLs found? How...
This issue is to track some TODOs related to the crawler API 1. Allow creating a crawler API key by the website UI: Right now to create a crawler API...