open-semantic-search
open-semantic-search copied to clipboard
Does OSS follow links on pages and documents?
I have a lot of links I've imported but it seems like OSS is not following any links it finds on web pages and documents, for web pages it only follow links on the same domain but not outside it, I even imported some CVS files with only urls on it and none of the urls were indexed, it only index the document and is not following the links it finds inside. Is it because I need to configure something specific for that to work or its just the way OSS works in general? If it's the later I think it would be nice to have an option to force it follow links but that should be the default behavior in my opinion as that's what usually any search engine do.