Memory Error

[adimukewar]

@piergiaj thanks for the pyTorch implementation of i3d. I'm trying to reproduce your results.

I'm facing memory error while running extract_features.py for rgb as well as flow part. I'm able to generate few .npy files before encountering this error. I'm using 1080 Ti 11 gb GPU and a machine having 16 gb ram.

[piergiaj]

Try reducing the 1600 frames to a smaller value, whatever will fit into your GPU's memory. I used 4 Titan X GPUs with 12GB memory for this script.

https://github.com/piergiaj/pytorch-i3d/blob/05783d11f9632b25fe3d50395a9c9bb51f848d6d/extract_features.py#L73-L79

[adimukewar]

Thanks a lot for the reply. I already tried reducing the frame size to 100. It still kept on running into memory errors. Is there anything else that you think, can be done?

[pengxiaoxiao]

Thanks a lot for the reply. I already tried reducing the frame size to 100. It still kept on running into memory errors. Is there anything else that you think, can be done? Have you solved the problem? I have also encountered this problem

[wangwen39]

The models which you provided are not in parallel ??

[suvaansh]

You can write the code like the following because 'volatile' flag has been depriciated after pytorch 0.4.

with torch.no_grad():
    for phase in ['train', 'val']:
        i3d.train(False)  # Set model to evaluate mode
                
        tot_loss = 0.0
        tot_loc_loss = 0.0
        tot_cls_loss = 0.0
                    
        # Iterate over data.
        for _, data in tqdm(enumerate(dataloaders[phase])):
            # get the inputs
            inputs, labels, name = data
            if os.path.exists(os.path.join(save_dir, name[0]+'.npy')):
                continue

            b,c,t,h,w = inputs.shape
            if t > 1024:
                features = []
                for start in range(1, t-56, 1024):
                    end = min(t-1, start+1024+56)
                    start = max(1, start-48)
                    ip = Variable(torch.from_numpy(inputs.numpy()[:,:,start:end]).cuda())
                    features.append(i3d.extract_features(ip).squeeze(0).permute(1,2,3,0).data.cpu().numpy())
                np.save(os.path.join(save_dir, name[0]), np.concatenate(features, axis=0))
            else:
                # wrap them in Variable
                inputs = Variable(inputs.cuda())
                features = i3d.extract_features(inputs)
                np.save(os.path.join(save_dir, name[0]), features.squeeze(0).permute(1,2,3,0).data.cpu().numpy())

[chenyr0021]

I met the same problem. The error is: CUDA out of memory.

Maybe you can try to comment the following line:

i3d.cuda()

It worked for me.

[VivaainNg]

I met the same problem. The error is: CUDA out of memory.

Maybe you can try to comment the following line:

i3d.cuda()

It worked for me.

May I know which GPU model you were using?

[YanZhang-bit]

I met the same problem，and I changed the one parameter of dataloader :pin_memory=False. if this parameter is false, the code would need smaller memory，but it requires more time. And I think the frame size must be a multiple of 8，because the temporal stride of I3D is 8。if it isn’t a multiple of 8, it maybe has wrongs when you stitch them together