abhishek thakur

The code doesn't work with Keras 2 API. It needs to be updated to support the new API.

enhancement: convnet

1

An enhancement for convnet would be verbose, which prints the percentage of the files done that are supplied to convnet and time taken

multiprocessing doesnot work properly

1

I have observed that multiprocessing does not work properly if the corpus is huge. Right now Im dealing with around 10k documents and setting n_jobs = 1 works fine. However,...

Unknown token missing from BPE tokenizer trained with BPETrainer

2

I took a look at the short tutorial (https://huggingface.co/transformers/tokenizer_summary.html) and tried to replicate it (the BPE part of it). My initial corpus looks like this: ``` hug hug hug hug...

Pure python solution

2

Here is a solution that I thought of. Runs in 5.5 seconds on my machine: ``` import glob import time start = time.time() path = "../data/" all_files = glob.glob(path +...

Runs in ~192 seconds on my machine ``` import pandas as pd import glob import time start = time.time() path = "../data/" all_files = glob.glob(path + "/*.csv") li = []...

Example incorrect

1

The pybrain example code is wrong. X is not found.