mwmbl
A way to submit sites to index
Orion Browser has ported some of the Web Extensions API to WebKit. I am really curious if that includes the ones used by mwmbl crawler extension. You can get the beta through TestFlight.
This may not be the right approach, but it looks really interesting.
I would like to add the ability to add sites to be indexed. It needs to be done carefully though to prevent spam. My current thought is that a site gets submitted and then must be approved by a moderator.
I don't know if the projet needs more URLs to crawl, I bet there's already plenty, but here's some resources that might be usefull:
- https://github.com/tb0hdan/domains
- https://github.com/etalab/noms-de-domaine-organismes-publics
This is a feature we discussed about adding to the extension in the future. Just like @daoudclarke said, the main concern is about validating the submitted sites. If you have any ideas on how to tackle this you are free to propose solutions.
My current idea for this is that we have some designated moderators that can approve suggestions. But first we need to come up with a policy for how to define/know what is a good site to crawl
You can triage the suggested URLs with ad/spam/malware filter lists, like EasyList and AdGuard lists.
Perhaps the existing web extension can be used to insert small panels into footers, asking the user if they'd like to crawl this website. This can help with the goal of growing the index while avoiding the spam problem.
