Train this model... but OOM

[jgyy4775]

I want to train this model. But I encounter out of memory. I use GTX Titan X.

I think that exist memory leak...

Which GPU is right for this model?

[jwyang]

can you post your command?

[jgyy4775]

@jwyang I use this command "python main.py --config-file configs/sgg_res101_step.yaml"

[jwyang]

You need to reduce the bacth size. The default is 8 for 8 gpus.

[jwyang]

A single cpu can usually hold 1 or 2 images.

[jgyy4775]

@jwyang I reduce batch size, learning rate and image size. but result is "out of memory".

[jwyang]

even batch_size=1? can you show me the output in your terminal?

[jgyy4775]

yes, batch size is "1".

[jwyang]

when does this happen? at the beginning or after training?

[jgyy4775]

beginning..

[simonJJJ]

When training, my batch_size is 2, no OOM, but when inference, the result is "out of memory" after 40/26446...

[dreichCSL]

So this means if you don't have a GPU with at least 10GB RAM, you can't run this model at all? That's a shame..