Saurabh Shrivastava

Find and integrate a text tokenisation library.

2

The current implementation of text tokenisation is pretty naive and doesn't cover all aspects. A nice tokenisation library should be able to generate all possible text tokens like currency, dates,...

Error handling is extremely important but is missing at many crucial places. There are some functions such as `fatal()` (see: https://github.com/saurabhshri/CCAligner/blob/master/src/lib_ccaligner/commons.cpp). Add error handling for those cases to make program...

There are logger functions implemented in the program _(see: https://github.com/saurabhshri/CCAligner/blob/master/src/lib_ccaligner/commons.cpp)_ , but logging is not implemented everywhere. Use those functions and properly perform logging at relevant places.

Alternate naming using og:title meta property and slicing of extra non required words.

Currently they are just stripped not extracted and saved.