stacking icon indicating copy to clipboard operation
stacking copied to clipboard

Large data loading problem

Open ikki407 opened this issue 9 years ago • 0 comments

Training and test data are now loaded at the beginning of model building.

However, it is not time-efficient if the data is very large.

So, if the data used is not changed through staking script, the data should be stored somewhere of base.py.

STORED_X = pd.DataFrame()

def load_data(stored=True):
    if STORED_X:
        return STORED_X

    else:
        ...

    if stored:
        STORED_X = X.copy()
        return STORED_X

    else:
        return X

ikki407 avatar Aug 24 '16 05:08 ikki407