anomalib icon indicating copy to clipboard operation
anomalib copied to clipboard
verified publisher icon open-edge-platform

[Bug]: Hardcoded team in `wandb` writer blocks usage for general public

Open kteppris opened this issue 2 years ago • 2 comments

Describe the bug

Description:

In the current implementation of the benchmark.py script, the team for wandb (Weights & Biases) is hardcoded to "anomalib". This prevents regular users, who might not have access to this team, from utilizing this feature without modifying the source code.

Location: anomalib/tools/benchmarking/benchmark.py

Snippet:

if "wandb" in config.writer:
    upload_to_wandb(team="anomalib", folder=runs_folder)

Suggested Solution:

  1. Configuration-based team setting: Introduce a team field in the YAML configuration for the wandb writer. This would allow users to specify their own team when setting up the benchmark. If not provided, it can default to "anomalib" or any suitable default.

YAML example:

writer:
  - wandb:
      team: my_team_name
  - tensorboard
  1. Update the benchmark script: Read the team from the config when available.

Example modification:

wandb_team = config.writer.wandb.team if "wandb" in config.writer and hasattr(config.writer.wandb, "team") else "anomalib"
upload_to_wandb(team=wandb_team, folder=runs_folder)

This way, users can customize the wandb team setting without modifying the library's core code.

I would be happy to take over this issue myself.

Dataset

N/A

Model

N/A

Steps to reproduce the behavior

Steps to reproduce:

  1. Create a benchmark config
seed: 42
compute_openvino: false
optimization:
  export_mode: openvino
hardware:
  - gpu
writer:
  - wandb
grid_search:
  dataset: 
    category: bottle
    num_workers: 0
  model_name:
    - padim
    - stfpm
"""
with open("benchmark_config.yaml", "w", encoding="utf8") as f:
    f.writelines(benchmarking_params)
  1. Login wandb
wandb login <key>
  1. Run benchmark
python ./tools/benchmarking/benchmark.py --config benchmark_config.yaml

OS information

OS information:

  • OS: [Ubuntu 20.04.6 LTS (Focal Fossa)]
  • Python version: [3.10.10]
  • Anomalib version: [1.0.0dev]
  • PyTorch version: [1.13.1+cu116]
  • CUDA/cuDNN version: [12.1]
  • GPU models and configuration: [1x NVIDIA A2]

Expected behavior

Saving the benchmark run to the own wandb.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

1.0.0dev

Configuration YAML

seed: 42
compute_openvino: false
optimization:
  export_mode: openvino
hardware:
  - gpu
writer:
  - wandb
grid_search:
  dataset: 
    category: bottle
    num_workers: 0
  model_name:
    - padim
    - stfpm

Logs

Benchmarking started 🏃‍♂️. This will take a while ⏲ depending on your configuration.
Global seed set to 42
2023-10-24 12:27:16,185 - anomalib.data - INFO - Loading the datamodule
2023-10-24 12:27:16,185 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-10-24 12:27:16,186 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-10-24 12:27:16,186 - anomalib.models - INFO - Loading the model.
2023-10-24 12:27:16,186 - anomalib.models.components.base.anomaly_module - INFO - Initializing PadimLightning model.
2023-10-24 12:27:16,192 - anomalib.models.components.feature_extractors.timm - WARNING - FeatureExtractor is deprecated. Use TimmFeatureExtractor instead. Both FeatureExtractor and TimmFeatureExtractor will be removed in a future release.
2023-10-24 12:27:16,458 - timm.models.helpers - INFO - Loading pretrained weights from url (https://download.pytorch.org/models/resnet18-5c106cde.pth)
Downloading: "https://download.pytorch.org/models/resnet18-5c106cde.pth" to /home/jovyan/.cache/torch/hub/checkpoints/resnet18-5c106cde.pth
2023-10-24 12:27:17,270 - anomalib.data.mvtec - INFO - Found the dataset.
2023-10-24 12:27:25,335 - anomalib.models.padim.lightning_model - INFO - Aggregating the embedding extracted from the training set.
2023-10-24 12:27:25,360 - anomalib.models.padim.lightning_model - INFO - Fitting a Gaussian to the embedding collected from the training set.
2023-10-24 12:27:36,253 - anomalib.utils.callbacks.timer - INFO - Training took 17.92 seconds
2023-10-24 12:27:36,262 - anomalib.data.mvtec - INFO - Found the dataset.
2023-10-24 12:27:40,959 - anomalib.utils.callbacks.timer - INFO - Testing took 4.657652378082275 seconds
Throughput (batch_size=32) : 17.820136253743804 FPS
2023-10-24 12:27:45,879 - __mp_main__ - INFO - One sweep run complete for model padim On category bottle{'Training Time (s)': 18.99006986618042, 'Testing Time (s)': 4.793054103851318, 'Inference Throughput gpu (fps)': 18.663128258521095, 'OpenVINO Inference Throughput (fps)': nan, 'pixel_F1Score': 0.72203129529953, 'pixel_AUROC': 0.9830057621002197, 'image_F1Score': 0.9763779044151306, 'image_AUROC': 0.9936507940292358}
Global seed set to 42
2023-10-24 12:27:45,958 - anomalib.data - INFO - Loading the datamodule
2023-10-24 12:27:45,959 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-10-24 12:27:45,959 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-10-24 12:27:45,959 - anomalib.models - INFO - Loading the model.
2023-10-24 12:27:45,959 - anomalib.models.components.base.anomaly_module - INFO - Initializing StfpmLightning model.
2023-10-24 12:27:45,965 - anomalib.models.components.feature_extractors.timm - WARNING - FeatureExtractor is deprecated. Use TimmFeatureExtractor instead. Both FeatureExtractor and TimmFeatureExtractor will be removed in a future release.
2023-10-24 12:27:46,213 - timm.models.helpers - INFO - Loading pretrained weights from url (https://download.pytorch.org/models/resnet18-5c106cde.pth)
2023-10-24 12:27:46,344 - anomalib.models.components.feature_extractors.timm - WARNING - FeatureExtractor is deprecated. Use TimmFeatureExtractor instead. Both FeatureExtractor and TimmFeatureExtractor will be removed in a future release.
2023-10-24 12:27:46,634 - anomalib.data.mvtec - INFO - Found the dataset.
2023-10-24 12:32:18,762 - anomalib.utils.callbacks.timer - INFO - Training took 272.05 seconds
2023-10-24 12:32:18,768 - anomalib.data.mvtec - INFO - Found the dataset.
2023-10-24 12:32:23,364 - anomalib.utils.callbacks.timer - INFO - Testing took 4.57196569442749 seconds
Throughput (batch_size=32) : 18.15411697011725 FPS
2023-10-24 12:32:27,907 - __mp_main__ - INFO - One sweep run complete for model stfpm On category bottle{'Training Time (s)': 272.1340026855469, 'Testing Time (s)': 4.634397268295288, 'Inference Throughput gpu (fps)': 18.909745970341994, 'OpenVINO Inference Throughput (fps)': nan, 'pixel_F1Score': 0.7797383069992065, 'pixel_AUROC': 0.9870787262916565, 'image_F1Score': 0.9921259880065918, 'image_AUROC': 0.9992063045501709}
wandb: Currently logged in as: <username>. Use `wandb login --relogin` to force relogin
wandb: ERROR Error while calling W&B API: permission denied (<Response [403]>)
Problem at: /home/jovyan/work/WiMi/Basler/AnomalyDetection/anomalib/tools/benchmarking/utils/metrics.py 115 upload_to_wandb
wandb: ERROR It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
Traceback (most recent call last):
  File "/home/jovyan/work/WiMi/Basler/AnomalyDetection/anomalib/./tools/benchmarking/benchmark.py", line 329, in <module>
    distribute(_sweep_config)
  File "/home/jovyan/work/WiMi/Basler/AnomalyDetection/anomalib/./tools/benchmarking/benchmark.py", line 248, in distribute
    upload_to_wandb(team="anomalib", folder=runs_folder)
  File "/home/jovyan/work/WiMi/Basler/AnomalyDetection/anomalib/tools/benchmarking/utils/metrics.py", line 115, in upload_to_wandb
    wandb.init(
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1189, in init
    raise e
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1170, in init
    run = wi.init()
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 785, in init
    raise error
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
Traceback (most recent call last):
  File "/home/jovyan/work/WiMi/Basler/AnomalyDetection/anomalib/./tools/benchmarking/benchmark.py", line 329, in <module>
    distribute(_sweep_config)
  File "/home/jovyan/work/WiMi/Basler/AnomalyDetection/anomalib/./tools/benchmarking/benchmark.py", line 248, in distribute
    upload_to_wandb(team="anomalib", folder=runs_folder)
  File "/home/jovyan/work/WiMi/Basler/AnomalyDetection/anomalib/tools/benchmarking/utils/metrics.py", line 115, in upload_to_wandb
    wandb.init(
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1189, in init
    raise e
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1170, in init
    run = wi.init()
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 785, in init
    raise error
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)

Code of Conduct

  • [X] I agree to follow this project's Code of Conduct

kteppris avatar Oct 24 '23 12:10 kteppris

Thanks for reporting @kteppris

samet-akcay avatar Oct 24 '23 12:10 samet-akcay

@samet-akcay do you think this should be fixed like recommended? If so, you are welcome to do that @kteppris 😄

blaz-r avatar Oct 25 '23 11:10 blaz-r

I am closing this as benchmark.py has been replaced by pipelines. I've taken a note of this issue for when third-party logging is restored https://github.com/openvinotoolkit/anomalib/issues/2056

ashwinvaidya17 avatar May 14 '24 10:05 ashwinvaidya17