Proxy-Anchor-CVPR2020
Proxy-Anchor-CVPR2020 copied to clipboard
tjddus9597
CUDA out of memory
python train.py --gpu-id 0 --loss Proxy_Anchor --model r
esnet50 --embedding-size 512 --batch-size 180 --lr 6e-4 --dataset SOP --warm 1 --bn-freeze 0 --lr-decay-step 20 --lr-dec
ay-gamma 0.25
wandb: Currently logged in as: shute (use wandb login --relogin to force relogin)
wandb: wandb version 0.10.10 is available! To upgrade, please run:
wandb: $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.10.8
wandb: Syncing run eager-dream-16
wandb: ⭐️ View project at https://wandb.ai/shute/SOP_ProxyAnchor
wandb: 🚀 View run at https://wandb.ai/shute/SOP_ProxyAnchor/runs/2ca7m47a
wandb: Run data is saved locally in wandb/run-20201113_090754-2ca7m47a
wandb: Run wandb off to turn off syncing.
Random Sampling
Training parameters: {'LOG_DIR': '../logs', 'dataset': 'SOP', 'sz_embedding': 512, 'sz_batch': 180, 'nb_epochs': 60, 'gpu_id': 0, 'nb_workers': 4, 'model': 'resnet50', 'loss': 'Proxy_Anchor', 'optimizer': 'adamw', 'lr': 0.0006, 'weight_decay': 0.0001, 'lr_decay_step': 20, 'lr_decay_gamma': 0.25, 'alpha': 32, 'mrg': 0.1, 'IPC': None, 'warm': 1, 'bn_freeze': 0, 'l2_norm': 1, 'remark': ''}
Training for 60 epochs.
0it [00:00, ?it/s]/home/server8/lst/Proxy-Anchor-CVPR2020-master/code/losses.py:48: UserWarning: This overload of nonzero is deprecated:
nonzero(Tensor input, *, Tensor out)
Consider using one of the following signatures instead:
nonzero(Tensor input, *, bool as_tuple) (Triggered internally at /opt/conda/conda-bld/pytorch_1595629411241/work/torch/csrc/utils/python_arg_parser.cpp:766.)
with_pos_proxies = torch.nonzero(P_one_hot.sum(dim = 0) != 0).squeeze(dim = 1) # The set of positive proxies of data in the batch
Train Epoch: 0 [330/330 (100%)] Loss: 10.849229: : 330it [01:34, 3.49it/s]
Evaluating...
100%|██████████| 337/337 [01:25<00:00, 3.95it/s]
R@1 : 51.770
R@10 : 67.938
R@100 : 81.594
R@1000 : 92.909
0it [00:00, ?it/s]
Traceback (most recent call last):
File "train.py", line 290, in
wandb: Waiting for W&B process to finish, PID 4172153 wandb: Program failed with code 1. Press ctrl-c to abort syncing. wandb: wandb: Find user logs for this run at: wandb/run-20201113_090754-2ca7m47a/logs/debug.log wandb: Find internal logs for this run at: wandb/run-20201113_090754-2ca7m47a/logs/debug-internal.log wandb: Run summary: wandb: loss 12.3987 wandb: R@1 0.5177 wandb: R@10 0.67938 wandb: R@100 0.81594 wandb: R@1000 0.92909 wandb: _step 0 wandb: _runtime 193 wandb: _timestamp 1605276667 wandb: Run history: wandb: loss ▁ wandb: R@1 ▁ wandb: R@10 ▁ wandb: R@100 ▁ wandb: R@1000 ▁ wandb: _step ▁ wandb: _runtime ▁ wandb: _timestamp ▁ wandb: wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: wandb: Synced eager-dream-16: https://wandb.ai/shute/SOP_ProxyAnchor/runs/2ca7m47a
CUDA out of memory in the second epoch. I have set the batch size to 30, 100, 150, 180,. Nothing helps. pytorch 1.6 CUDA 10.1 GPU RTX2080Super 8G
I have spent many hours, but still can not solve.
Many thanks for your help.
If memory problems occur during training, try reducing the batch size. If you run into problems during evaluation, especially only on the SOP dataset, set the line 144, "if len(xs)<10000:" of the "evaluate_cos_SOP" function by reducing the constant value. This line determines the size to compute the cosine similarity. However, if this number is reduced, the number of loops increases, which may increase the evaluation time.
e.g., utils.py
def evaluate_cos_SOP(X, T, normalize=False): if normalize: X = l2_norm(X)
# get predictions by assigning nearest 8 neighbors with cosine
K = 1000
Y = []
xs = []
for x in X:
if len(xs)<1000:
xs.append(x)
else:
xs.append(x)
xs = torch.stack(xs,dim=0)
cos_sim = F.linear(xs,X)
y = T[cos_sim.topk(1 + K)[1][:,1:]]
Y.append(y.float().cpu())
xs = []
# Last Loop
xs = torch.stack(xs,dim=0)
cos_sim = F.linear(xs,X)
y = T[cos_sim.topk(1 + K)[1][:,1:]]
Y.append(y.float().cpu())
Y = torch.cat(Y, dim=0)
# calculate recall @ 1, 2, 4, 8
recall = []
for k in [1, 10, 100, 1000]:
r_at_k = calc_recall_at_k(T, Y, k)
recall.append(r_at_k)
print("R@{} : {:.3f}".format(k, 100 * r_at_k))
return recall
