MemoryError workaround

[nortz8]

Kindly consider changing the def _expand_paragraphs function in the cdqa_sklearn.py file to accommodate larger datasets. Modifying the dataframe needs a lot of memory for bigger data so it would be better to set it as a list of dict before making it a dataframe.

Below is the modification I did so I would not get a MemoryError:

@staticmethod
def _expand_paragraphs(df): 
     data=[]
     for n in range(len(df)):  
         stringlist = df.iloc[n][1]  
         for m in range(len(stringlist)): 
             a=df.iloc[n][0] 
             b=stringlist[m] 
             data.append({'title' : a, 'content' : b}) 
     dfx = pd.DataFrame(data) 
     return dfx

[adjouama]

Very good point. +1 @nortz8 However, your workaround did not work for me. I ended up having the following; ValueError: empty vocabulary; perhaps the documents only contain stop words

Any idea why ?