PatchCore: Training only on good images and then inference on good and bad images

[luliuzee]

First of all, great work! I found this work very interesting! I have a question related to PatchCore implementation.

When there is both training, and testing dataset (images and ground truth) available, then the PatchCore works good and I think during the validation step, it is using the testing data (GT and images) to calculate coreset subsampling.

But imagine a realworld scenario when only good images are available to teach the anomaly detection algorithm, and no defective images are available (GT and images), then how to train the model? I've tried limit_val_batches and limit_test_batches to zero, so during training, no validation and testing are performed. But with this setup, during inference time, the model is producing this error

raceback (most recent call last):
  File "tools/inference.py", line 118, in <module>
    infer()
  File "tools/inference.py", line 101, in infer
    output = inference.predict(image=args.image_path, superimpose=True)
  File "/notebooks/anomalib/anomalib/deploy/inferencers/base.py", line 90, in predict
    predictions = self.forward(processed_image)
  File "/notebooks/anomalib/anomalib/deploy/inferencers/torch.py", line 118, in forward
    return self.model(image)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/notebooks/anomalib/anomalib/models/components/base/anomaly_module.py", line 75, in forward
    return self.model(batch)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/notebooks/anomalib/anomalib/models/patchcore/model.py", line 170, in forward
    patch_scores = self.nearest_neighbors(embedding=embedding, n_neighbors=9)
  File "/notebooks/anomalib/anomalib/models/patchcore/model.py", line 238, in nearest_neighbors
    distances = torch.cdist(embedding, self.memory_bank, p=2.0)  # euclidean norm
  File "/opt/conda/lib/python3.8/site-packages/torch/functional.py", line 1119, in cdist
    return _VF.cdist(x1, x2, p, None)  # type: ignore
RuntimeError: cdist only supports at least 2D tensors, X2 got: 1D

Any guidance will be highly appreciated!

[samet-akcay]

Hi @luliuzee , yes this a known issue. In fact, almost none of the existing anomaly detection algorithms addresses this issue. We are currently working on a solution that wouldn't require any validation or test images to find the threshold in an unsupervised manner. We have some promising results. We'll make it available here once we publish our results.

Meanwhile, what you could do is to choose manual threshold and set it in the config.yaml file. By default, config.yaml file uses adaptive threshold that uses validation/tests sets.

[nixczhou]

So anomalib is calculating the threshold using the validation data only right?

[djdameln]

Since the feature is still under construction, it would probably be best to wait for one of the next releases which will include this feature.

Yes, Anomalib's adaptive thresholding mechanism relies only on the validation data.

[nixczhou]

Thanks for replying. But i could play with the poc features using your branch right? cause im also curious with the result. And also does not sure when will it be merged and released XD.

[djdameln]

Yes, you could definitely check out the branch. Just be aware that the feature is still under construction and we cannot guarantee that it is stable.

[nixczhou]

Yeah this is what I am looking for !!!! Cause I really want a fully unsupervised method, even for calculating the threshold.

[samet-akcay]

@djdameln would you be able to comment here?

[djdameln]

Setting the adaptive threshold parameter to false should be sufficient to prevent anomalib from computing the optimal threshold value based on the validation set. It will then use the entered values for the image_default and pixel_default parameters instead. The values of these parameters should be chosen by trial and error, and may vary between models and datasets.

Please note however that even when adaptive thresholding is disabled, Anomalib still expects some abnormal images to be provided. This is because by design, anomalib evaluates each model after training in order to present some performance metrics to the user. This design is based on the assumption that for any use case, there will be at least a few examples available of the anomalous class.

I feel it would be good to remove this constraint, and allow training a model without performing evaluation afterwards. This would at least require some changes to the dataset classes, but possibly other parts of the pipeline will be affected as well.

[nixczhou]

Many thanks for clarification!

[nixczhou]

But may I know which part of the code did this mechanism?

[djdameln]

The adaptive threshold is computed by the AdaptiveThreshold class, which is a torchmetrics Metric object. The class is called here in the base anomaly module at the end of the validation epoch.

[nixczhou]

Thanks, @djdameln I am sorry but i could not see where does it effectively return the maximum observed pixel value in the anomaly heatmaps of the normal images?

[djdameln]

The maximal observed pixel value is not retrieved explicitly, but implicitly as a side effect of the precision-recall curve computation which is a part of the adaptive threshold computation. The precision-recall curve is computed by calculating the precision and recall scores over the validation set for a range of threshold values. The candidate values for the threshold at which precision and recall are evaluated are obtained by incrementally lowering the anomaly score between the highest and lowest value in the predictions, while monitoring for changes in the precision and recall scores.

If the validation set has no anomalous samples, then the true positive and false negative counts will both be zero for every possible threshold value. This means that precision (tp/(tp+fp)) will always yield zero, and recall (tp/(tp+fn)) cannot be computed due to zero in the denominator, and will therefore also default to zero.

So, both precision and recall will not change when lowering the value of the threshold candidate. As a result, we only retrieve a single threshold candidate, which coincides with the value of the highest score that occurs in the predictions on the validation set