CSV-file parser

Copyright (C) 2012, Dmitry Kolesnikov

This file is free documentation; unlimited permisions are give to copy, distribute and modify the documentation.

This library is free software; you can redistribute it and/or modify it under the terms of the the 3-clause BSD License (the "License"); as published by http://www.opensource.org/licenses/BSD-3-Clause.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! !!! !!! WARNING !!! !!! The library is not supported. !!! !!! Use CSV feature of https://github.com/fogfish/feta !!! !!! !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Introduction

The simple CSV-file parser based on event model. The parser generates an
event/callback when the CSV line is parsed. The parser supports both
sequential and parallel parsing. The major goal is an performance of 
intake procedure with an parsing target of 3 - 4 micro seconds pr line on
the reference hardware.

                       Acc
                   +--------+
                   |        |
                   V        |
              +---------+   |
----Input---->| Parser  |--------> AccN
      +       +---------+
     Acc0          |
                   V
               Event Line 

The parser takes as input binary stream, event handler function and
initial state/accumulator. Event function is evaluated agains current 
accumulator and parsed line of csv-file. Note: The accumaltor allows to
carry-on application specific state throught event functions.

Compile and build

The library source code is available at git repository

git clone https://github.com/fogfish/pts.git

Briefly, the shell command ./configure; make; make install' should configure, build, and assembly distribution package. The following instructions are specific to this package; see the INSTALL' file for instructions specific to GNU build tools.

The `configure' shell script attempts to guess dependencies and system configuration required to build library, the following build time dependencies exists:

--with-erlang={prefix_to_otp} supplied to `./configure' binds the library with chosen Erlang runtime, if you have multiple Erlang environments available at build machine

High performance version of library shall be build with native targets

make BUILD=native

Interface

Briefly, the sequence of operations for data parse/intake is following; see the src/csv.erl file for detailed interface specification and/or example parser at priv/csv_example.erl

%% define an event funtion that takes two arguments line value and %% accumulator. The function shall return a new accumulator state. %% The structure of accumulator is an application specific, that might %% vary from integer to comprex record. Fun = fun({line, L}, #my_record{count = C} = Acc0) -> do_my_intake_to_somewhere(lists:reverse(L)), Acc0#my_record{count = C + 1} end

%% %% A sequential parse, parses whole data stream in client process csv:parse(CSV, Fun, #myrecord{})

%% %% a parallel parse splits the CSV into multiple chunks; %% spawns multiple processes (process per chunk) %% results agregated in the client process. csv:parse(CSV, 20, Fun, #myrecord{})

Performance

Reference platform: * MacMini, Lion Server, * 1x Intel Core i7 (2 GHz), 4x cores * L2 Cache 256KB per core * L3 Cache 6MB * Memory 4GB 1333 MHZ DDR3 * Disk 750GB 7200rpm WDC WD7500BTKT-40MD3T0 * erlang R15B + native build of the library

The data set is has following patterns: key, date, time, float numbers and zz suffix * key{1..300 000},2012-03-25,23:26:15.543,166.280,...,zz

The numbers of keys is 300.000, and number of float fields varies from 8, 24 and 40 in reference data. Reference data set is generated by command

make example or perl priv/gen_set.pl 300 40 > priv/set-300K-40.txt

version 0.0.1

E/Parse Size (MB) Read (ms) Handle (ms) Per Line (us)

300K, 8 flds 23.41 91.722 350.000 1.16 300K, 24 flds 50.42 489.303 697.739 2.33 300K, 40 flds 77.43 780.296 946.003 3.15

ET/hash Size (MB) Read (ms) Handle (ms) Per Line (us)

300K, 8 flds 23.41 91.722 384.598 1.28 300K, 24 flds 50.42 489.303 761.414 2.54 300K, 40 flds 77.43 780.296 1047.329 3.49

ET/tuple Size (MB) Read (ms) Handle (ms) Per Line (us)

300K, 8 flds 23.41 91.722 228.306 0.76 300K, 24 flds 50.42 489.303 601.025 2.00 300K, 40 flds 77.43 780.296 984.676 3.28

ETL/ets Size (MB) Read (ms) Handle (ms) Per Line (us)

300K, 8 flds 23.41 91.722 1489.543 4.50 300K, 24 flds 50.42 489.303 2249.689 7.50 300K, 40 flds 77.43 780.296 2519.401 8.39

ETL/pts Size (MB) Read (ms) Handle (ms) Per Line (us)

300K, 8 flds 23.41 91.722 592.886 1.98 300K, 24 flds 50.42 489.303 1190.745 3.97 300K, 40 flds 77.43 780.296 1734.898 5.78