Working with big data, the smart way

Do you think python is slow? Or that it cannot work with big data?

Well, not anymore! With this tutorials I want to show how it is possible to have an astonishingly fast code while you work with big data.

Seems boring you say?

Well, we are working with an adorable racoon, a chicken, Mandelbrot's fractal and The Garden Of The Earthly Delights, this is as fun as it gets.

If you are looking for something more serious, we also have a tutorial working with real world data, in this case a dataset of the stock movements of Microsoft in one day.

Currently available tutorials

Previews are available in two formats, nbviewer a static HTML option or Wakari an online data analysis enviroment.

• Speeding code with numba [nbviewer] [Wakari]
• Working with huge images in python, the painless way [nbviewer] [Wakari]
• Generating huge Mandelbrot fractals with Python, the quick and memory-safe way [nbviewer] [Wakari]
• Comparing speedups when using compression with blz barrays [nbviewer] [Wakari]
• Querying big datasets, the good way [nbviewer] [Wakari]
• Comparing speedups when using compression with blz btables in disk only [nbviewer] [Wakari]
• Comparing performance of btables in memory and disk [nbviewer] [Wakari]

Requirements

• Numba
• blz
• Matplotlib
• IPython
• numpy
• scipy
• pandas

If you have Anaconda installed simply do: conda install numba blz matplotlib numpy scipy ipython pandas

How to run

Just go to the tutorials folder and execute: ipython notebook

My computer specifications

• Model: Alienware M11x r2
• Processor Name: Intel Core i7-640UM
• Processor Speed: 1.2 GHz
• Operating System: Lubuntu
• RAM: 4 GB
• Graphics Card: Nvidia GeForce 335M
• Graphics Memory: 1024
• Disk speed: 7200 rpm