deep-learning-bitcoin icon indicating copy to clipboard operation
deep-learning-bitcoin copied to clipboard

Facing memory exceptions

Open dhanunjayalakshmi opened this issue 7 years ago • 0 comments

While executing the program, I am facing memory exceptions

Traceback (most recent call last): File "data_generator.py", line 94, in main() File "data_generator.py", line 90, in main data_gen_func(data_folder, bitcoin_file) File "data_generator.py", line 26, in generate_quantiles return generate_cnn_dataset(data_folder, bitcoin_file, get_label) File "data_generator.py", line 43, in generate_cnn_dataset btc_df = file_processor(bitcoin_file) File "/home/mashficool/testing/deep-learning-bitcoin/data_manager.py", line 15, in file_processor v = pd.DataFrame(d['volume'].resample('5Min').sum()) File "/home/mashficool/testing/venv/lib/python3.5/site-packages/pandas/core/generic.py", line 5522, in resample base=base, key=on, level=level) File "/home/mashficool/testing/venv/lib/python3.5/site-packages/pandas/core/resample.py", line 999, in resample return tg._get_resampler(obj, kind=kind) File "/home/mashficool/testing/venv/lib/python3.5/site-packages/pandas/core/resample.py", line 1103, in _get_resampler axis=self.axis) File "/home/mashficool/testing/venv/lib/python3.5/site-packages/pandas/core/resample.py", line 91, in init self.groupby._set_grouper(self._convert_obj(obj), sort=True) File "/home/mashficool/testing/venv/lib/python3.5/site-packages/pandas/core/groupby.py", line 439, in _set_grouper indexer = self.indexer = ax.argsort(kind='mergesort') File "/home/mashficool/testing/venv/lib/python3.5/site-packages/pandas/core/indexes/base.py", line 2151, in argsort return result.argsort(*args, **kwargs) MemoryError

I havent changed any code so, the latest code frrom master branch(commit ID - f489800) has the code.

Can you let me know, how much memory does this program consume? I can plan accordingly on my machine...

dhanunjayalakshmi avatar Apr 18 '18 02:04 dhanunjayalakshmi