Crashing due to malloc failure

[svendaj]

Trying to run example_model.py on Windows 10, 8GB RAM notebook results in following crash:

(numer.ai) C:\Users\lsadmin\Documents\numer.ai\example-scripts>python example_model.py Downloading dataset files... 2022-05-17 16:01:22,904 INFO numerapi.utils: target file already exists 2022-05-17 16:01:22,906 INFO numerapi.utils: download complete 2022-05-17 16:01:24,126 INFO numerapi.utils: target file already exists 2022-05-17 16:01:24,126 INFO numerapi.utils: download complete 2022-05-17 16:01:25,315 INFO numerapi.utils: target file already exists 2022-05-17 16:01:25,315 INFO numerapi.utils: download complete 2022-05-17 16:01:26,494 INFO numerapi.utils: target file already exists 2022-05-17 16:01:26,494 INFO numerapi.utils: download complete 2022-05-17 16:01:28,128 INFO numerapi.utils: target file already exists 2022-05-17 16:01:28,128 INFO numerapi.utils: download complete Reading minimal training data Traceback (most recent call last): File "C:\Users\lsadmin\Documents\numer.ai\example-scripts\example_model.py", line 52, in validation_data = pd.read_parquet('v4/validation.parquet', File "C:\Users\lsadmin\anaconda3\envs\numer.ai\lib\site-packages\pandas\io\parquet.py", line 493, in read_parquet return impl.read( File "C:\Users\lsadmin\anaconda3\envs\numer.ai\lib\site-packages\pandas\io\parquet.py", line 240, in read result = self.api.parquet.read_table( File "pyarrow\array.pxi", line 767, in pyarrow.lib._PandasConvertible.to_pandas File "pyarrow\table.pxi", line 1996, in pyarrow.lib.Table._to_pandas File "C:\Users\lsadmin\anaconda3\envs\numer.ai\lib\site-packages\pyarrow\pandas_compat.py", line 789, in table_to_blockmanager blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes) File "C:\Users\lsadmin\anaconda3\envs\numer.ai\lib\site-packages\pyarrow\pandas_compat.py", line 1135, in _table_to_blocks result = pa.lib.table_to_blocks(options, block_table, categories, File "pyarrow\table.pxi", line 1356, in pyarrow.lib.table_to_blocks File "pyarrow\error.pxi", line 116, in pyarrow.lib.check_status pyarrow.lib.ArrowMemoryError: malloc of size 4209552448 failed

Any idea why or how to overcome? Failed malloc is exactly 4GB, can the example run on machine like above?

[1NoobDev]

Your device doesn't meet the required system specifications as described in the FAQ:

Minimum: 16GB RAM and 4 core (Intel i5) / 6 cores (AMD Ryzen 5). Recommended: 32GB RAM and 8 core (Intel i7/i9) / 12 cores (AMD Ryzen 7/9).

You could try to change the feature_metadata on line 44, to small: features = feature_metadata["feature_sets"]["medium"] # get the medium feature set

Or overcome by adding more RAM in your device.