multi-threading

[mikkokotila]

Currently we're running everything on a single thread. I wonder if there is a straightforward way to provide a wrapper that allows multi-threading (or even distributing) tokenization.

[drupchen]

What I can think of right now is first running the preprocessing on the input string (here), then distributing pieces of the generated chunks to be tokenized to different threads, and concatenating in the end the lists of Tokens of each worker.

The preprocessing is much faster than the actual tokenizing, and right now, I don't have any idea on how to split this task for concurrent threads...

Does that help?

[10zinten]

Unfortunately we cannot achieve parallelization with multi-threading in python because og the GIL (Global Lock Interpreter), which only allow one thread to run on a process. But we can do mutli-processing to achieve parallelization, which I have implemented in the script to do Google OCR on bunch of images.

https://github.com/Esukhia/Google-OCR/blob/1d19e9e0ce6872988365b0d8805c9cb96269f913/usage/bdrc/image_to_text.py#L114

We will be adding this multi-processing soon

[drupchen]

A simple way of doing it would be to change this line: https://github.com/Esukhia/botok/blob/improve-tok/botok/tokenizers/wordtokenizer.py#L80. Creating a new method that returns tokens where all the multiprocessing happens will keep things simple, and would even allow for two modes to choose from: single processing or multi-processing.

[mikkokotila]

The below example seems to work. The benefit is that it does not require any changes to Botok. The trick is to find the right number for Pool. I could do around 20 on a 16-core machine, but after that massive memory leak takes over. It seems that Pool has a lot of issues with memory, so that's probably expected. It seems that performance gain is linear. 16x processes, 16x faster tokenization.

I have OpenPecha repos in texts/ so that just the directory containing the volumes an the volumes are there.

from botok import TokChunks

def tokenization(text, tokenizer):

    out = []

    preproc = TokChunks(text)
    preproc.serve_syls_to_trie()
    tokens = tokenizer.tokenize(preproc)

    for i in range(len(tokens)):

        out.append(tokens[i]["text"])

    del preproc, tokens
    
    return out

def init_tokenizer():

    from botok import BoSyl, Config, Tokenize, Trie

    config = Config()
    trie = Trie(BoSyl,
                profile=config.profile,
                main_data=config.dictionary,
                custom_data=config.adjustments,
                pickle_path=config.dialect_pack_path.parent)
    
    tokenizer = Tokenize(trie)

    return tokenizer

def create_paths():
    
    import os
    
    paths = []

    for text_dir in os.listdir('texts'):
        for text in os.listdir('texts/' + text_dir):
            paths.append(text_dir + '/' + text)
            
    return paths

def create_tokens(path):
    
    import os
    import gc

    # handle input
    docs = open('texts/' + path, 'r').read()
    tokens = tokenization(docs, tokenizer)

    # handle output
    os.makedirs(os.path.dirname('tokens/' + path), exist_ok=True)
    out = open('tokens/' + path, 'w')
    out.write(' '.join(tokens))
    del tokens
    out.close()
    
    gc.collect()
    
    print(path)
     
from multiprocessing import Pool
import warnings

warnings.simplefilter('ignore')

tokenizer = init_tokenizer()
p = Pool(20)
paths = create_paths()
p.map(create_tokens, paths)

[mikkokotila]

:)

[ngawangtrinley]

Nice!