Unsure on how to use on custom dataset

[lekshmijk]

Hi! I'm kind of a newbie here so I have a small custom dataset which I'd like to use anomalib on..
the structure of the dataset is a bit different from MVTec ( mainly I don't have a folder with the ground truth images):

dataset
|-- battery_cell
|-----|----- test
|-----|--------|------ good
|-----|--------|------ not good
|-----|----- train
|-----|--------|------ good

I'm not sure how to go about modifying anomalib to train on this dataset...I did modify the config.yaml accordingly but that didn't quite do the job....
I'd be very grateful if someone can help me out here!!

[djdameln]

Hi, Anomalib models can be trained on custom datasets using the Folder dataset format. Please have a look at this guide in our documentation for a detailed explanation and an example of training on custom data.

In your case, the dataset section of your config will look something like this:

dataset:
  name: battery  # you are free to choose your own name here
  format: folder
  path: <path-to-dataset>/battery_cell
  normal_dir: train/good
  abnormal_dir: test/not_good
  task: classification  # classification or segmentation
  mask: null  # optional, required when task type is segmentation
  normal_test_dir: test/good  # optional
  ...

Please note that training with the segmentation task type is only possible when pixel-level ground truth annotations, in the form of binary pixel masks, are available for the anomalous images in your dataset. The path to these masks should be provided in the dataset.mask parameter. If pixel-level annotations are not available for your dataset, you will be required to set the dataset.task parameter to classification in your config file during training. During inference you may still use the segmentation task type.

Please let us know if you run into any problems while following the steps in the guide. We are always happy to help!

[lekshmijk]

Thank you so much for your reply! The classification works perfectly using Patchcore but while using DRAEM and FastFlow, I get this error :

Is there any other parameters I should change in the config file? Currently it looks like this:

dataset:
  name: battery_cell #options: [mvtec, btech, folder]
  format: folder
  path: /home/lekshmi/Desktop/battery_cell
  task: classification
  normal_dir: train/good
  abnormal_dir: test/not_good
  mask: null #/home/lekshmi/Desktop/pcb1/ground_truth/bad  # optional, required when task type is segmentation
  normal_test_dir: test/good  # optional
  extensions: null
  split_ratio: 0.2
  image_size: 256 # options: [256, 256, 448, 384] - for each supported backbone
  train_batch_size: 1
  test_batch_size: 1
  num_workers: 8
  transform_config:
    train: null
    val: null
  create_validation_set: false
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

model:
  name: fastflow
  backbone: resnet18 # options: [resnet18, wide_resnet50_2, cait_m48_448, deit_base_distilled_patch16_384]
  pre_trained: true
  flow_steps: 8 # options: [8, 8, 20, 20] - for each supported backbone
  hidden_ratio: 1.0 # options: [1.0, 1.0, 0.16, 0.16] - for each supported backbone
  conv3x3_only: True # options: [True, False, False, False] - for each supported backbone
  lr: 0.001
  weight_decay: 0.00001
  early_stopping:
    patience: 3
    metric: pixel_AUROC
    mode: max
  normalization_method: min_max # options: [null, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: null #options: onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  accumulate_grad_batches: 1
  amp_backend: native
  auto_lr_find: false
  auto_scale_batch_size: false
  auto_select_gpus: false
  benchmark: false
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  default_root_dir: null
  detect_anomaly: false
  deterministic: false
  devices: 1
  enable_checkpointing: true
  enable_model_summary: true
  enable_progress_bar: true
  fast_dev_run: false
  gpus: null # Set automatically
  gradient_clip_val: 0
  ipus: null
  limit_predict_batches: 1.0
  limit_test_batches: 1.0
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  log_every_n_steps: 50
  log_gpu_memory: null
  max_epochs: 500
  max_steps: -1
  max_time: null
  min_epochs: null
  min_steps: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle
  num_nodes: 1
  num_processes: null
  num_sanity_val_steps: 0
  overfit_batches: 0.0
  plugins: null
  precision: 32
  profiler: null
  reload_dataloaders_every_n_epochs: 0
  replace_sampler_ddp: true
  strategy: null
  sync_batchnorm: false
  tpu_cores: null
  track_grad_norm: -1
  val_check_interval: 1.0 # Don't validate before extracting features.

Also, in the images generated by from the classification task, the images are either classified as "normal" or "anomalous" along with a certain percentage..what does that percentage actually signify? Looking at the code, I see that it is the predicted score, but I'm not sure how exactly it is calculated...Could you please explain to me how these numbers are generated?

[djdameln]

Models like DRAEM and FastFlow require fine-tuning of the neural network weights, and therefore these models need to run for multiple epochs. To prevent overfitting during this training process, we compute some performance metric on the validation set after each epoch and stop training when the performance stops increasing. By default, we use pixel-level AUROC score as early stopping metric to accommodate segmentation tasks. When training with the classification task type however, this metric cannot be computed due to lack of pixel-level annotations. Instead, we need to use one of the image-level metrics to determine early stopping.

So, you should be able to solve your issue by simply setting the model.early_stopping.metric parameter to either image_AUROC or image_F1.

Could you please explain to me how these numbers are generated?

The model predicts an anomaly score, but this anomaly score is unbounded, and the absolute values may differ widely between different models in Anomalib. To make the model predictions more human interpretable, we normalize the model predictions to the [0, 1] range. By default, we use min-max normalization, where the values are re-scaled between the minimum and maximum values observed in the validation set. The class responsible for the normalization is the MinMaxNormalizationCallback.

In addition, a thresholding mechanism is applied, which allows us to assign the normal vs anomalous label to the images based on the value of the anomaly score. By default, the threshold is computed adaptively by maximizing the F1 score over the validation set. During the normalization process, the threshold value is centered around 0.5, so that all normal samples will have a normalized score between 0 and 0.5 and all anomalous samples will have a score between 0.5 and 1.0.

You may also want to have a look at this discussion thread, which also covers these topics.

[lekshmijk]

Thank you for your reply!

[ardiya]

Adding the new link to the guide(as per 2024) since this is the first result on search engine: https://anomalib.readthedocs.io/en/latest/markdown/guides/how_to/data/custom_data.html

[cory619-seungwon]

Thanks all but, how to inference without Folder()? When using Folder(), it must have split ratio but, I don't want to use split ratio to inference my custom dataset. I'd like to inference only 1 image.