MemoryError

[2718455213wcx]

I used drian in lopai to parse the thunderbird dataset (29.8gb) without getting MemoryError, but I did get MemoryError when I parsed the split thunderbird dataset (2.92gb) using Brain in logpai. When I parse a 1000m thunderbird dataset there is no MemoryErro.Why is that? Can Brain only parse data sets around 1gb in size? Traceback (most recent call last): File "E:\logbert-main\TBird\data_process.py", line 137, in parse_log(data_dir, output_dir, log_file, parser_type) File "E:\logbert-main\TBird\data_process.py", line 77, in parse_log parser.parse(log_file) File "E:\logbert-main\TBird..\logparser\Brain.py", line 58, in parse group_len, tuple_vector, frequency_vector = self.get_frequecy_vector( File "E:\logbert-main\TBird..\logparser\Brain.py", line 261, in get_frequecy_vector set.setdefault(str(lenth), []).append(token) MemoryError