icwsm2010_tutorial

icwsm2010_tutorial copied to clipboard

This repository contains example code for the tutorial I presented at ICWSM 2010, "Large-scale social media analysis with Hadoop". More information, including the slides, available here:

http://jakehofman.com/icwsm2010

wordcount/ contains the wordcount example on small input text network/ contains network examples on a small toygraph hstream.py is a simple class for implementing streaming jobs

Examples can be run locally, using the "cat data | map | sort | reduce" analog of Hadoop streaming, or with Hadoop streaming.

To install Hadoop locally, just download and untar the source. Quick start guides available at:

http://hadoop.apache.org/common/docs/current/quickstart.html or http://www.ibm.com/developerworks/linux/library/l-hadoop-1/

If installing on Mac OS X, make sure to set JAVA_HOME to point to Java 1.6:

http://blog.sethladd.com/2009/04/mac-os-x-hadoop-0191-and-java-16.html

My Hadoop bookmarks are available here:

http://delicious.com/jhofman/hadoop http://delicious.com/jhofman/hadoop+tutorials

Disclaimer: these examples are written with pedagogy, not efficiency, in mind.