memory error when train voc dataset

[ShawnLiu1011]

Loading initial model weights from data/imagenet_weights/res101.pth
Loaded.
Traceback (most recent call last):
  File "./tools/trainval_net.py", line 192, in <module>
    sw.train_model(iters_sum)
  File "/home/nieqinqin/liuxiaoyu/SSM-Pytorch/tools/../lib/model/train_val.py", line 295, in train_model
    self.net.train_step_with_summary(blobs, self.optimizer)
  File "/home/nieqinqin/liuxiaoyu/SSM-Pytorch/tools/../lib/nets/network.py", line 478, in train_step_with_summary
    summary = self._run_summary_op()
  File "/home/nieqinqin/liuxiaoyu/SSM-Pytorch/tools/../lib/nets/network.py", line 343, in _run_summary_op
    summaries.append(self._add_score_summary(key, var))
  File "/home/nieqinqin/liuxiaoyu/SSM-Pytorch/tools/../lib/nets/network.py", line 71, in _add_score_summary
    return tb.summary.histogram('SCORE/' + key + '/scores', tensor.data.cpu().numpy(), bins='auto')
  File "/home/nieqinqin/.local/lib/python3.6/site-packages/tensorboardX/summary.py", line 112, in histogram
    hist = make_histogram(values.astype(float), bins)
  File "/home/nieqinqin/.local/lib/python3.6/site-packages/tensorboardX/summary.py", line 119, in make_histogram
    counts, limits = np.histogram(values, bins=bins)
  File "/home/nieqinqin/anaconda3/lib/python3.6/site-packages/numpy/lib/histograms.py", line 710, in histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File "/home/nieqinqin/anaconda3/lib/python3.6/site-packages/numpy/lib/histograms.py", line 385, in _get_bin_edges
    endpoint=True, dtype=bin_type)
  File "/home/nieqinqin/anaconda3/lib/python3.6/site-packages/numpy/core/function_base.py", line 115, in linspace
    y = _nx.arange(0, num, dtype=dt)
MemoryError
Command exited with non-zero status 1
20.45user 9.70system 0:27.26elapsed 110%CPU (0avgtext+0avgdata 2499876maxresident)k
0inputs+349024outputs (0major+724822minor)pagefaults 0swaps

Sometimes it happens but sometimes it disappears. I dont know why...

[HuizhouLi]

I also encountered this problem. Has anyone successfully solved it?

[ShawnLiu1011]

I also encountered this problem. Has anyone successfully solved it?

I solved it by delete /data/cache and /output. But I don't know why...