Semantic Robustness

[dxoigmn]

What does this PR do?

Testing DataLoader 0:  86%|█████████████████████████████████████████████████████████████████████████▋            | 6/7 [20:09<03:21,  0.00it/s]
metrics = {
  'angle': tensor([-5.5439e+01, -2.7251e-03, -3.6145e+01,  1.3271e+00,  7.2796e+01,        -5.9831e+01, -6.6907e+01,  1.1341e+01, -6.5626e+01,  1.1184e+01,        -6.0181e+01,  7.0074e+01,  1.7820e+01, -7.4833e+01], device='cuda:0'),
  'hue': tensor([-1.9470,  0.2596,  0.2614,  0.3296, -1.6336, -1.2026, -1.5837,  0.2451,         0.4145,  0.1754,  2.7241, -1.2809,  0.4522, -1.0182], device='cuda:0'),
  'sat': tensor([ 0.5000,  0.5000,  0.2928, -0.0047,  0.5000,  0.0540,  0.2465, -0.2716,         0.4478,  0.2954, -0.3278, -0.0228,  0.3537,  0.4321], device='cuda:0'),
  'gain': tensor([-0.0181, -0.0190, -0.0142, -0.0280, -0.0275, -0.0149, -0.0156, -0.0255,        -0.0233, -0.0214, -0.0175, -0.0233, -0.0079, -0.0302], device='cuda:0'),
  'step': tensor([59, 17, 32, 61, 50, 47, 43,  3, 77,  3, 80, 21,  8, 80],       device='cuda:0', dtype=torch.int32)}
Testing DataLoader 0: 100%|██████████████████████████████████████████████████████████████████████████████████████| 7/7 [23:49<00:00,  0.00it/s]INFO:anomalib.callbacks.timer:Testing took 1669.4315311908722 seconds
Throughput (batch_size=16) : 0.06589069269677242 FPS
[05/22/24 15:57:50] INFO     Testing took 1669.4315311908722 seconds                                                               timer.py:109
                             Throughput (batch_size=16) : 0.06589069269677242 FPS
[05/22/24 15:57:50] INFO     Testing took 1669.4315311908722 seconds                                                               timer.py:109
                             Throughput (batch_size=16) : 0.06589069269677242 FPS
Testing DataLoader 0: 100%|██████████████████████████████████████████████████████████████████████████████████████| 7/7 [23:50<00:00,  0.00it/s]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │    0.32107144594192505    │
│       image_F1Score       │    0.7073171138763428     │
│        pixel_AUROC        │    0.9192594289779663     │
│       pixel_F1Score       │    0.2511599361896515     │
└───────────────────────────┴───────────────────────────┘

Compare to:

venv $ CUDA_VISIBLE_DEVICES=1 python3 batch_adversarial_attack.py  --clip_backbone ViT-B-16-plus-240 --object hazelnut
attack:   0%|                                                                                                                                                                                                                          | 0/100 [00:00<?, ?it/s
/lib/python3.10/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: No positive samples in targets, true positive value should be meaningless. Returning zero tensor in true positive score
  warnings.warn(*args, **kwargs)
attack: 100%|█████████████████████████████████████████████████████████████████████████| 100/100 [36:51<00:00, 21.93s/it, current pauroc=0.898, current f1-max=0.201, current PRO=0.562, per-sample pauroc=0.803, per-sample f1-max=0.115, per-sample PRO=0.345]
myclass = 'hazelnut' init_pauroc = 0.9544397592544556 best_pauroc = 0.8982865810394287
paper_results/hazelnut_018fa2d77a6e769fb325a12b44d49851.pt
attack: 100%|█████████████████████████████████████████████████████████████████████████| 100/100 [36:52<00:00, 22.12s/it, current pauroc=0.898, current f1-max=0.201, current PRO=0.562, per-sample pauroc=0.803, per-sample f1-max=0.115, per-sample PRO=0.345]

Type of change

Please check all relevant options.

• [ ] Improvement (non-breaking)
• [ ] Bug fix (non-breaking)
• [ ] New feature (non-breaking)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

• [ ] pytest
• [ ] CUDA_VISIBLE_DEVICES=0 python -m mart experiment=CIFAR10_CNN_Adv trainer=gpu trainer.precision=16 reports 70% (21 sec/epoch).
• [ ] CUDA_VISIBLE_DEVICES=0,1 python -m mart experiment=CIFAR10_CNN_Adv trainer=ddp trainer.precision=16 trainer.devices=2 model.optimizer.lr=0.2 trainer.max_steps=2925 datamodule.ims_per_batch=256 datamodule.world_size=2 reports 70% (14 sec/epoch).

Before submitting

• [ ] The title is self-explanatory and the description concisely explains the PR
• [ ] My PR does only one thing, instead of bundling different changes together
• [ ] I list all the breaking changes introduced by this pull request
• [ ] I have commented my code
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] New and existing unit tests pass locally with my changes
• [ ] I have run pre-commit hooks with pre-commit run -a command without errors

Did you have fun?

Make sure you had fun coding 🙃