colibri-core

colibri-core copied to clipboard

Implement begin-of-sentence and end-of-sentence markers in pattern model training

1

Complicated by the fact that everything is now a patternpointer. Looking for the best approach to tackle this..

proycon

Refactor Pattern.instanceof to make use of v2 changes and add flexgram support

1

(Used by new filterset mechanism)

proycon

Hi, To begin with, Thank you.. For the amazing work you've done so far.. I have a few questions regarding my usage of colibric-core in my project What I am...

manrock007

Discovered by @fkunneman; output file was only 2-bytes (the initial null byte and version marker). Input text was just: ```prachtig apparaat en droogt goed kreukelvrij fijn de verlichting binnenin voelt...

proycon

(requested by Gabor Toth)

proycon

Investigate improved scalability using use of out-of-memory datastructures

1

The following library could be pluggable into our current framework: *STXXL implements containers and algorithms that can process huge volumes of data that only fit on disks.*: http://stxxl.sourceforge.net/

proycon

Load corpora with mmap

1

Would it be possible to load copora with mmap? This would make it possible to work with corpora larger than the available RAM, and is much more efficient if only...

andreasvc

proycon

Currently, Colibri Core only extracts skipgrams in which the skip is not at an initial or final position, but in the middle. For example, patterns like `x {*}` and `{*}...

proycon