wikiteam
wikiteam copied to clipboard
WikiTeam
Setting up a crawler to list all MediaWiki wikis in the web
From [email protected] on May 04, 2013 12:28:07
To grow our wikiteam collection of wikis, I have to increase our list of wikis. To archive our first 4500 wikis, we've used Andrew Pavlo's list. Now I want to adapt his crawling framework (see source linked at http://www.cs.brown.edu/~pavlo/mediawiki/ , and its README) to have a more up to date and complete list. I created my settings.py, used pip to install django 1.2, installed MySQL-python from my repositories, replaced httplib2 with httplib... and finally got stuck with MySQL errors. Unless someone else runs it for me I need something simpler, most of the features in the original graffiti etc. are excessive and in particular there's no reason why I should need a database, I'd like to modify/strip it and get a self-contained version just to make a list of domains running MediaWiki...
Original issue: http://code.google.com/p/wikiteam/issues/detail?id=59
From [email protected] on August 12, 2013 08:31:31
I checked out that code, and I didn't see django required anywhere. I think I was looking at the wrong thing? Would you mind forking that into a VCS somewhere online, or pointing me directly to the root and I'll put it up on github?
From [email protected] on August 12, 2013 10:03:07
Thanks for looking. It's not my code or repo; I just followed http://graffiti.cs.brown.edu/svn/graffiti/README
From [email protected] on October 25, 2013 05:45:18
Labels: -Type-Defect -Priority-Medium Type-Enhancement Priority-High
From [email protected] on December 01, 2013 04:01:47
I made something with ruby mechanize: https://gist.github.com/nemobis/7718061 I've learnt a lot making it (euphemism for "crashed my head") but I'm not sure it will be useful because the search results actually returned by Google (or Yahoo) are many less than "promised". For instance, searching for "Magnus Manske, Brion Vibber, Lee Daniel Crocker" should be a rather reliable way to find only one page per site (Special:Version) and gives an estimate of 25k results, but actually returns around a hundred.
Currently Google has reduced very much the number of page results it returns. If you find any reliable search engine that doesn't cut the results and returns relevant links, we can research how to scrape it.
Emilio J. Rodríguez-Posada, 27/06/2014 10:06:
Currently Google has reduced very much the number of page results it returns. If you find any reliable search engine that doesn't cut the results and returns relevant links, we can research how to scrape it.
I've already worked around Google's limitations with the script above and found several thousands wikis, but not that many. You can run it from Spain, it will give slightly different results and add some URLs we don't know.
I'm afraid however search engines don't help much, despite all the effort I put into it: so I'm currently working on using services like builtwith.com and 80plus.
Would Internet Archive give us a list of MediaWiki sites from the WayBack Machine? Currently you can't search the archived sites, but I'm sure they can do it from intranet with some interface.
Ask Jason/Alex?
Emilio J. Rodríguez-Posada, 27/06/2014 10:50:
Would Internet Archive give us a list of MediaWiki sites from the WayBack Machine? Currently you can't search the archived sites, but I'm sure they can do it from intranet with some interface.
Ask Jason/Alex?
I don't think the Internet Archive has any reasonable infrastructure to search even just the
of their archived pages; @ab2525 would know. Asking IA's help would mean asking them several person-hours work at best, probably person-days; or transferring petabytes of data to analyse. I can imagine it as rather expensive.Services to crawl a few hundreds thousands/millions sites and search the respective
cost a handful hundreds dollars, I found: do you think they're not worth it?
Obviously I don't know how they have built the Wayback Machine, but I'm sure they have a way to query URLs. There is a basic search engine https://archive.org but not useful for our task.
I only want a list of archived links that ends in "/Special:Version" or something that helps us to find MediaWiki wikis. Later we can do post-processing and exclude false positives, dead urls, etc. I'm sure they have an index anywhere that can be filtered with 'grep' or whatever.
Emilio J. Rodríguez-Posada, 27/06/2014 11:30:
I only want a list of archived links that ends in "/Special:Version"
*Special:Version$ is an option, true. The data in question would be https://archive.org/details/waybackcdx
I have coded a basic spider for Referata results in Google, resulting in findind more than 100 new wikis 7a6ef183393c0a5e777a3fa7db4527d4d7fcdbe6
Run the script: python referata-spider.py > newlist cut -d'/' -f1-3 newlist | sort | uniq > newlist2 cat newlist2 referata.com | sort | uniq > newlist3 diff referata.com newlist3 mv newlist3 referata.com
I will see if other wikifarms are supported with this approach.
Emilio J. Rodríguez-Posada, 27/06/2014 15:03:
I have coded a basic spider for Referata results in Google, resulting in findin > 100 new wikis 7a6ef18 https://github.com/WikiTeam/wikiteam/commit/7a6ef183393c0a5e777a3fa7db4527d4d7fcdbe6
Nice. As you can see in my gist, I forgot to try referata queries.
Added spider for shoutwiki and updated the list 90c442a5b72d42a205e05a9eab58a4b22d5d903d
Added spider for wikkii and updates the list c420d4d843feb1bf66d0b284d87f07b20035ea5e
Isn't Special:Version blocked by robots.txt by default and only linked to from a small number of places on-wiki anyway, so generally not the best way to find MW sites?
Ideas: Search for site:*/Special:Recentchanges, which is linked from every page on a wiki so much more likely to be caught by search engines, though still robots.txt blocked. Search for inurl:/Main_Page and the default translation in a bunch of languages, a large majority of wikis seem to keep main page. (inurl apparently returns ~double the results of a normal site: search?) Search for inurl:/wiki/ mediawiki
Grab but don't yet visit all URLs, use some regex to normalize them, strip duplicates, strip sites you already have, then run checkalive against them to see if they are living MW sites?
etesp, 27/06/2014 21:35:
Isn't Special:Version blocked by robots.txt by default and only linked to from a small number of places on-wiki anyway, so generally not the best way to find MW sites?
No, usually wikis don't set any robot policy; and the most advert ones use short URLs and block only the non-short variants e.g. w/index.php?title=Special:Version but not wiki/Special:Version.
And I already did all the rest that you suggest, see the gist above and run it yourself: https://gist.github.com/nemobis/7718061
Added spider for orain.org wikifarm and updated its list a3e69666feb0a290cd74d6a6aa117cf502e9107f
Added spider for neoseeker wikifarm and updated its list 636c6a91dff894a2dc0e5f1e3fbd3c1b3957b547
wiki-site has some dynamic lists:
http://www.wiki.co.il/active-wiki-en.html http://www.wiki.co.il/active-wiki-all.html
But any of them is up to 3000 wikis. Our list is over 5000. How was generated?
Emilio J. Rodríguez-Posada, 02/07/2014 16:34:
But any of them is up to 3000 wikis. Our list is over 5000. How was generated?
Our list came from mutante's wikistats IIRC, I don't know where his comes from (it's probably years old).
There are 3,140 URLs in those two lists. Would you like me to run my checkalive.pl script on the lists?
Scott
On Wed, Jul 2, 2014 at 9:34 AM, Emilio J. Rodríguez-Posada < [email protected]> wrote:
wiki-site has some dynamic lists:
http://www.wiki.co.il/active-wiki-en.html http://www.wiki.co.il/active-wiki-all.html
But any of them is up to 3000 wikis. Our list is over 5000. How was generated?
— Reply to this email directly or view it on GitHub https://github.com/WikiTeam/wikiteam/issues/59#issuecomment-47783719.
2014-07-03 7:01 GMT+02:00 Scott D. Boyd [email protected]:
There are 3,140 URLs in those two lists. Would you like me to run my checkalive.pl script on the lists?
Tomorrow I will merge that two lists with our wiki-site list (5,000) and remove dupes.
After that, please run your script over the list.
Scott
On Wed, Jul 2, 2014 at 9:34 AM, Emilio J. Rodríguez-Posada < [email protected]> wrote:
wiki-site has some dynamic lists:
http://www.wiki.co.il/active-wiki-en.html http://www.wiki.co.il/active-wiki-all.html
But any of them is up to 3000 wikis. Our list is over 5000. How was generated?
— Reply to this email directly or view it on GitHub https://github.com/WikiTeam/wikiteam/issues/59#issuecomment-47783719.
— Reply to this email directly or view it on GitHub https://github.com/WikiTeam/wikiteam/issues/59#issuecomment-47866971.
2014-07-04 0:27 GMT+02:00 Emilio J. Rodríguez-Posada [email protected]:
2014-07-03 7:01 GMT+02:00 Scott D. Boyd [email protected]:
There are 3,140 URLs in those two lists. Would you like me to run my
checkalive.pl script on the lists?
Tomorrow I will merge that two lists with our wiki-site list (5,000) and remove dupes.
After that, please run your script over the list.
wiki-site.com list updated 767123e89dc4d0449a1b098a054b23fb9ab6d171 Now it has 5839 unique wikis.
I would add a delay to the checkalive because wiki-site is not very reliable. Also, checking some wikis by hand seems that they have some servers down.
Thanks
Scott
On Wed, Jul 2, 2014 at 9:34 AM, Emilio J. Rodríguez-Posada < [email protected]> wrote:
wiki-site has some dynamic lists:
http://www.wiki.co.il/active-wiki-en.html http://www.wiki.co.il/active-wiki-all.html
But any of them is up to 3000 wikis. Our list is over 5000. How was generated?
— Reply to this email directly or view it on GitHub https://github.com/WikiTeam/wikiteam/issues/59#issuecomment-47783719.
— Reply to this email directly or view it on GitHub https://github.com/WikiTeam/wikiteam/issues/59#issuecomment-47866971.
This is getting more difficult on Google, but maybe DuckDuckGo can be good enough. Given the limits to pagination, it needs to be done one TLD at a time.