[HELP] Segmentation task using only good/bad images (no masks)

[FedericoDeBona]

Hello,
I need to train a model based only on good/bad images without masks to perform segmentation, but I have some issues.
I tried Anomalib version 0.4.0 and 0.7.0 comparing a custom screw dataset with the MVTec screw dataset.

Anomalib 0.4.0

Custom dataset

This are examples of images from the custom dataset

0	1	2	3

With a total of 280 images.

This is the config.yaml using Padim

dataset:
  name: screw
  format: folder
  path: /home/rdover/anomalib_test/screw/
  normal_dir: good # name of the folder containing normal images.
  abnormal_dir: null # name of the folder containing abnormal images.
  task: segmentation # classification or segmentation
  mask_dir: null #optional
  normal_test_dir: null # optional
  extensions: null
  split_ratio: 0.2  # normal images ratio to create a test split
  image_size: 960
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 12
  normalization: imagenet # data distribution to which the images will be normalized
  test_split_mode: from_dir # options [from_dir, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  transform_config:
      train: null
      eval: null
  val_split_mode: from_test # determines how the validation set is created, options [same_as_test, from_test]

Other settings are the defaults.

After training here's the results:

Inference on good screw

good_original	good_inference

In this case the model recognizes a problem on the tip, but it's actually fine.

Inference on a missing tip screw

missing_tip_original	missing_tip_inference

it recognizes that there are scratches, but not that the tip is missing.

MVTec screw dataset

Same as before, the model is trained only on good images from the MVTec dataset. Here's the config file, again with padim

dataset:
  name: screw
  format: folder
  path: /home/rdover/anomalib/datasets/MVTec/screw/train/
  normal_dir: good # name of the folder containing normal images.
  abnormal_dir: null # name of the folder containing abnormal images.
  task: segmentation # classification or segmentation
  mask_dir: null #optional
  normal_test_dir: null # optional
  extensions: null
  split_ratio: 0.2  # normal images ratio to create a test split
  image_size: 256
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 12
  normalization: imagenet # data distribution to which the images will be normalized
  test_split_mode: from_dir # options [from_dir, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  transform_config:
      train: null
      eval: null
  val_split_mode: from_test # determines how the validation set is created, options [same_as_test, from_test]

Inference on good screw

good_inference	good_original

This is actually the worst case, but even with other good screw, the inference finds errors

Inference on a screw with bad thread side

thread_side_inference	thread_side_original

The inference is not bad but the tip shouldn't be considered

A nice inference on a bad screw

scratch_neck_original	scratch_neck_inference

As you can see sometimes it works great

Anomalib version 0.7.0

Using the latest version with this config on the custom dataset (this time using abnormal_dir):

dataset:
  name: screw
  format: folder
  path: /home/rdover/anomalib_test/screw/
  normal_dir: good # name of the folder containing normal images.
  abnormal_dir: test/bad # name of the folder containing abnormal images.
  task: segmentation # classification or segmentation
  mask_dir: null #optional
  normal_test_dir: null # optional
  extensions: null
  split_ratio: 0.2  # normal images ratio to create a test split
  image_size: 960
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 12
  normalization: imagenet # data distribution to which the images will be normalized
  test_split_mode: from_dir # options [from_dir, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  transform_config:
      train: null
      eval: null
  val_split_mode: from_test # determines how the validation set is created, options [same_as_test, from_test]

I get this error:
ValueError: Segmentation task requires mask directory if test_split_mode is from_dir. You could set test_split_mode to none or provide a mask directory. so I change the config file as it suggest and do the training.

Now doing inference with this model, what I get is always an inference result of 0.5.

Conclusions

Is there a way to achieve a model only using good/bad images, or even only good for a segmentation task?

[abc-125]

Hello,

Is there a way to achieve a model only using good/bad images, or even only good for a segmentation task?

Most of the models here are trained with good images only, so it should be possible.
You can change the task from segmentation to classification, so you won't get ValueError: Segmentation task requires mask directory... The output model will still be able to do segmentation anyway. Here is the corrected dataset config:

dataset:
  name: screw
  format: folder
  path: /home/rdover/anomalib_test/screw/
  normal_dir: good # name of the folder containing normal images.
  abnormal_dir: test/bad # name of the folder containing abnormal images.
  task: classification # classification or segmentation
  mask_dir: null #optional
  normal_test_dir: null # optional
  extensions: null
  split_ratio: 0.2  # normal images ratio to create a test split
  image_size: 256
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 12
  normalization: imagenet # data distribution to which the images will be normalized
  test_split_mode: from_dir # options [from_dir, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  transform_config:
      train: null
      eval: null
  val_split_mode: from_test # determines how the validation set is created, options [same_as_test, from_test]

an inference result of 0.5.

It's hard to say why you got this result. You can try to use the config above. I tested it with the PatchCore model (make sure you are using its config.yaml and change only dataset), and it worked well.

In this case the model recognizes a problem on the tip, but it's actually fine.
it recognizes that there are scratches, but not that the tip is missing.

I might suggest changing the image size in your config from 960x960 to 256x256 (like in the MVTec AD dataset).

Hope that helps!

[FedericoDeBona]

Hi, thank you for the reply.

I tried your settings for the training using MVTec screw dataset again (I tried both Padim and Patchcore):

dataset:
  name: mvtec_screw
  format: folder
  path: /home/rdover/anomalib_0_7_0/datasets/mvtec_screw/train
  normal_dir: good # name of the folder containing normal images.
  abnormal_dir: bad # name of the folder containing abnormal images.
  task: classification # classification or segmentation
  mask_dir: null #optional
  normal_test_dir: null # optional
  extensions: null
  split_ratio: 0.2  # normal images ratio to create a test split
  image_size: 256
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 12
  normalization: imagenet # data distribution to which the images will be normalized
  test_split_mode: from_dir # options [from_dir, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  transform_config:
      train: null
      eval: null
  val_split_mode: from_test # determines how the validation set is created, options [same_as_test, from_test]
  tiling:
      apply: false
      tile_size: null
      stride: null
      remove_border_count: 0
      use_random_tiling: False
      random_tile_count: 16

And then I changed task: classification to task: segmentation for the inferences, but the results are not that good.

Padim

bad_inference	good_inference

In bad_inference the error is found, but the tip is also selected, which is actually fine.
In good_inference the screw is fine, but the inference is quite bad

Patchcore

bad_inference	good_inference

In this case the situation is not that bad, but why is the background like this? Because of it I get a pred_score of around 0.4 with the good screw

What am I doing wrong?

[abc-125]

In this case the situation is not that bad, but why is the background like this? Because of it I get a pred_score of around 0.4 with the good screw

What am I doing wrong?

I think it's how everything is supposed to work :) You can check these examples. Patchcore tends to have relatively high scores for noisy areas, you can try to use more images for training if it's possible. Or simply ignore these higher anomaly scores, you already have a threshold that shows actual high scores on the bad_inference image (in red circle).

[abc-125]

In bad_inference the error is found, but the tip is also selected, which is actually fine.
In good_inference the screw is fine, but the inference is quite bad

Padim is not that good for that task, Image-Level AUC for Screw with Padim is 0.845, with Patchcore 0.960, which is much better and you can see it from the anomaly maps as well.

[samet-akcay]

If you don't provide a mask, the adaptive thresholding mechanism cannot utilize any masks. Instead it assigns the pixel thresholds based on the image thresholds (bad images). This overall might reduce the performance of your model.

You could potentially try using synthetic anomalies, which synthetically produces anomalous regions and masks using the normal images. This might give you better performance.

[FedericoDeBona]

I tried it, but it's not really the case, since, in this case, the screws are not full screen and I don't want the background to get involved

[samet-akcay]

test_split_mode: from_dir # options [from_dir, synthetic]

As you could see above, you could set the test split mode to synthetic.