Performance measurements

[chaiiii12345]

Hi, I wanted to know if it's possible to get performance measurements like accuracy, precision, recall etc. other than the AUROC and F1Score? And since the latter two does require the use of recall, precision, etc. it should be possible to get them as output as well? I checked the codes in /anomalib/utils/metrics but I'm unsure how to do it 😅
Please help!!
I checked other discussions, and found #689, but it's unanswered...

[djdameln]

Anomalib uses the Torchmetrics package to evaluate model performance. Anomalib supports many metrics classes from the torchmetrics package out of the box, but also implements several custom metric implementations such as AUROC, PRO and AUPRO.

The metrics that will be used to evaluate your models can be configured in the metrics section of the config.yaml. Image- and pixel-level metrics can be configured separately, respectively under metrics.image and metrics.pixel.

When adding a metric to the evaluation pipeline, Anomalib will first search for a metric with the specified name in the anomalib.utils.metrics module. If the metric is not found in Anomalib, it will look for the metric in the available metrics of the Torchmetrics package. If no matching metrics are found in both Anomalib and Torchmetrics, a warning will be raised. The code responsible for this metric selection mechanism can be found here.

In your case, Precision, Recall and Accuracy are all available from the Torchmetrics package. So, to evaluate your Anomalib models using these metrics, simply list them under metrics in your config.yaml. Please note that the Accuracy metric will only work for the image-level evaluation, because the Torchmetrics implementation of this metric expects flattened inputs.

So to summarize, your desired use-case could be achieved by entering the following parameter values in the metrics section of your config.yaml.

metrics:
  image:
    - Precision
    - Recall
    - Accuracy
  pixel:
    - Precision
    - Recall

[nixczhou]

Btw, I am still confused of where does the anomalib do the computation to get the maximal pixel distance from good data for threshold if we do not has the defective image for validation

[djdameln]

I just added a brief answer to this question in the other thread: #192 (reply in thread)

[electro020]

Anomalib uses the Torchmetrics package to evaluate model performance. Anomalib supports many metrics classes from the torchmetrics package out of the box, but also implements several custom metric implementations such as AUROC, PRO and AUPRO.

The metrics that will be used to evaluate your models can be configured in the metrics section of the config.yaml. Image- and pixel-level metrics can be configured separately, respectively under metrics.image and metrics.pixel.

When adding a metric to the evaluation pipeline, Anomalib will first search for a metric with the specified name in the anomalib.utils.metrics module. If the metric is not found in Anomalib, it will look for the metric in the available metrics of the Torchmetrics package. If no matching metrics are found in both Anomalib and Torchmetrics, a warning will be raised. The code responsible for this metric selection mechanism can be found here.

In your case, Precision, Recall and Accuracy are all available from the Torchmetrics package. So, to evaluate your Anomalib models using these metrics, simply list them under metrics in your config.yaml. Please note that the Accuracy metric will only work for the image-level evaluation, because the Torchmetrics implementation of this metric expects flattened inputs.

So to summarize, your desired use-case could be achieved by entering the following parameter values in the metrics section of your config.yaml.

metrics:
  image:
    - Precision
    - Recall
    - Accuracy
  pixel:
    - Precision
    - Recall

Hello,
Is there a similar way to obtain the loss curve during the training ?
I would like to have a graph

[ashwinvaidya17]

@electro020 we just merged a PR to log the loss. The lightning documentation will be useful to you.

[electro020]

@electro020 we just merged a PR to log the loss. The lightning documentation will be useful to you.

Hello,
When i looked at the log , i couldnt see the loss:

Is it supposed to appear there ?

thank you