How to call onnx model to detect the presence of defects in images

[laogonggong847]

What is the motivation for this task?

Many thanks to all the members of the Anomalib team for open sourcing such an excellent tool. We have no doubt that the emergence of Anomalib will push the development of anomaly detection, and once again, we salute all the authors and contributors who have improved the Anomalib community.

After using padim in Anomalib to train my own dataset, I ended up with an onnx model because I set the export_mode in the corresponding config.yaml file to "onnx" (export_mode: "onnx"). However, I was confused about how to call this model to inference.

I know that Anomalib provides 4 programs for inference under . /tools/inference, which provides four programs for inference, and I am grateful to the developer for being so careful and comprehensive. But I'm new to onnx models, and I'm not sure if one of these four programs can be used directly to call onnx models, and if so, can you tell me explicitly which one it is. If not, can you tell me what I should do. Thank you very much!

Describe the solution you'd like

I would like to understand the method and steps of calling onnx model to achieve detection, thank you very much.

Besides that I have another question, why do I use a lot of CPU and memory resources when training padim models. While the GPU (RTX2080Ti) only takes up about 1%, sometimes even 0

Additional context

pytorch: 1.12.1+cu113
cuda: 11.3 (torch.cuda.is_available = True)
python: 3.8
anomalib: 0.5.0.dev0 (latest)

Yaml:

dataset:
  name: mydata
  format: folder
  path: ./Mydatasets/cubes
  normal_dir: normal # 独有 name of the folder containing normal images.
  abnormal_dir: abnormal # 独有 name of the folder containing abnormal images.
  normal_test_dir: null # 独有 name of the folder containing normal test images.
  task: classification # classification or segmentation
  mask: null #optional
  extensions: null
  split_ratio: 0.2 # ratio of the normal images that will be used to create a test split
  image_size: 256
  train_batch_size: 32
  test_batch_size: 32
  num_workers: 4
  transform_config:
    train: null
    eval: null
  create_validation_set: true
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: from_dir # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.1 # fraction of train/test images held out for validation (usage depends on val_split_mode)

model:
  name: padim
  backbone: resnet18
  pre_trained: true
  layers:
    - layer1
    - layer2
    - layer3
  normalization_method: min_max # options: [none, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: "onnx" #options: onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 30
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0 # Don't validate before extracting features.
  log_every_n_steps: 50
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

[blaz-r]

Hello,
glad to hear that you find Anomalib useful :)

If you want to use onnx model, you can use openvino_inference or gradio_inference (this one opens gradio that is used via browser). This is example of open_vino instruction (where you pass onnx model):

python tools/inference/openvino_inference.py 
--weights results/padim/mvtec/bottle/run/openvino/model.onnx     
--metadata results/padim/mvtec/bottle/run/openvino/metadata.json     
--input datasets/MVTec/bottle/test/broken_large/000.png     
--output results/padim/mvtec/bottle/images 
--config src/anomalib/models/padim/config.yaml

Regarding the GPU usage, I can't test it right now but I think you can set accelerator to gpu in config.yaml under trainer. @samet-akcay or @ashwinvaidya17 will probably know more.

[laogonggong847]

Hello, thank you very much for all your help in the last week, this week I found a strange problem when training data with padim, it doesn't seem to call the GPU to complete the training. My CUDA version is 11.3, my graphics card is a 2080Ti, and my python version is 3.8. I'm pretty sure I can use my CUDA properly, and confirmed that the GPU version of pytorch works fine. However, when I trained with padim, my Yaml used the same Yaml I sent before, but I only made one change to the previous version of YAML, which was to change "num_workers: 8" to "num_workers: 0". My program reports an error at "num_workers: 8". At "num_workers: 0" I am training correctly, but it seems to be training with the CPU.

Please take a look at the following graph, it may have some Chinese information, but it does not affect the observation of CPU and GPU calls. How can I solve this problem?

As you can see from the graph, my CPU takes up a lot of memory during training, but the GPU utilization is 0%, which I think is due to the fact that I am not calling CUDA correctly. This is also the case when I change the "num_workers" to 1.

I am looking forward to your answer, thank you very much!

[blaz-r]

Hello,
as I suggested in previous post, try to change accelerator: auto in config yaml to accelerator: gpu. I'm not sure if this will solve your issue, but it's worth trying before others with more knowledge about Anomalib confirm or provide additional info.

[samet-akcay]

@laogonggong847, padim model is a memory-bank based model, so it is a bit memory intensive.

If you could refer to the trainining logs, you will see if you utilize your GPU.

Global seed set to 42
2023-04-24 10:57:57,953 - anomalib.data - INFO - Loading the datamodule
2023-04-24 10:57:57,954 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-04-24 10:57:57,954 - anomalib.data.utils.transform - INFO - No config file has been provided. Using default transforms.
2023-04-24 10:57:57,954 - anomalib.models - INFO - Loading the model.
2023-04-24 10:57:57,954 - anomalib.models.components.base.anomaly_module - INFO - Initializing PadimLightning model.
2023-04-24 10:57:58,202 - timm.models.helpers - INFO - Loading pretrained weights from url (https://download.pytorch.org/models/resnet18-5c106cde.pth)
2023-04-24 10:57:58,301 - anomalib.utils.loggers - INFO - Loading the experiment logger(s)
2023-04-24 10:57:58,302 - anomalib.utils.callbacks - INFO - Loading the callbacks
!!![HERE] 2023-04-24 10:57:58,327 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True
2023-04-24 10:57:58,327 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores
2023-04-24 10:57:58,327 - pytorch_lightning.utilities.rank_zero - INFO - IPU available: False, using: 0 IPUs
2023-04-24 10:57:58,327 - pytorch_lightning.utilities.rank_zero - INFO - HPU available: False, using: 0 HPUs

[samet-akcay]

Regarding the onnx inference, you could use the openvino inferencer.

[Kyle-fang]

Hello! I'm using onnx obtained from anomalib and then reasoning with onnxruntime, the input image shape is [1, 3, 256, 256] but the output is [1, 1, 256, 256], isn't it supposed to be the number of categories? Is there a problem with the .onnx file?

[blaz-r]

Hello. The output of anomaly detection is usually an anomaly map, whose width and height is the same as the input image. It does have only one channel, unlike the input that has 3 (R, G and B) because anomaly map is per pixel score of how anomalous that pixel/region is, therefore just one number -> 1 channel.

[Kyle-fang]

Hello! I used onnxruntime to reason about onnx model and got the output as an anomaly map of [1, 1, 256, 256], but I want to know if the current image is an anomaly or a person normal sample what should I do? Is there code to accomplish this?

[blaz-r]

@Kyle-fang you can do the following, which is considered a standard in anomaly classification, where maximum value is treated as anomaly score.

pred_score = predictions.reshape(-1).max()

# Common practice in anomaly detection is to assign anomalous
# label to the prediction if the prediction score is greater
# than the image threshold.
if "image_threshold" in metadata:
    pred_label = pred_score >= metadata["image_threshold"]

where metadata is that json produced by onnx export.

[Kyle-fang]

Thank you very much for your reply, it helps me a lot. Do the predications in pred_score = predictions.reshape(-1).max() represent the output of the model with shape [1, 1, 256, 256]? Then take the maximum value and compare it to "image_threshold", if it is greater then it is an anomaly sample? But the output of the normal samples also gets a maximum value much larger than the "image_threshold". I think I still don't understand what you mean, would you like to elaborate further?

[Kyle-fang]

The source code can distinguish abnormal samples very well, but I didn't find any details on how to do it in the source code. Can you tell me what the differentiation method in the source code is?