How metrics are computed?

[William9Baker]

When it comes to 'trainer.fit(model=model, datamodule=datamodule)', how does it automatically jump to this position that 'fpr, tpr = self._compute()' in the file of 'aupro.py'?

[samet-akcay]

@William9Baker, can you elaborate your question?

[samet-akcay]

model that we pass to Trainer is always a subclass of AnomalyModule that contains all these metric and threshold values as attributes, which are shown below:

anomalib/src/anomalib/models/components/base/anomaly_module.py

Lines 32 to 54 in 9323985

	class AnomalyModule(pl.LightningModule, ABC):
	"""AnomalyModule to train, validate, predict and test images.
	Acts as a base class for all the Anomaly Modules in the library.
	"""
	def __init__(self) -> None:
	super().__init__()
	logger.info("Initializing %s model.", self.__class__.__name__)
	self.save_hyperparameters()
	self.model: nn.Module
	self.loss: nn.Module
	self.callbacks: list[Callback]
	self.threshold_method: ThresholdMethod
	self.image_threshold = AnomalyScoreThreshold().cpu()
	self.pixel_threshold = AnomalyScoreThreshold().cpu()
	self.normalization_metrics: Metric
	self.image_metrics: AnomalibMetricCollection
	self.pixel_metrics: AnomalibMetricCollection

When the validation is complete, Lightning's Trainer object calls _collect_outputs methods which computes and updates the metrics.

anomalib/src/anomalib/models/components/base/anomaly_module.py

Lines 138 to 147 in 9323985

	def validation_epoch_end(self, outputs: EPOCH_OUTPUT) -> None:
	"""Compute threshold and performance metrics.
	Args:
	outputs: Batch of outputs from the validation step
	"""
	if self.threshold_method == ThresholdMethod.ADAPTIVE:
	self._compute_adaptive_threshold(outputs)
	self._collect_outputs(self.image_metrics, self.pixel_metrics, outputs)
	self._log_metrics()

self._collect_outputs method does the following to compute the metrics:

anomalib/src/anomalib/models/components/base/anomaly_module.py

Lines 172 to 182 in 9323985

	def _collect_outputs(
	image_metric: AnomalibMetricCollection,
	pixel_metric: AnomalibMetricCollection,
	outputs: EPOCH_OUTPUT,
	) -> None:
	for output in outputs:
	image_metric.cpu()
	image_metric.update(output["pred_scores"], output["label"].int())
	if "mask" in output.keys() and "anomaly_maps" in output.keys():
	pixel_metric.cpu()
	pixel_metric.update(torch.squeeze(output["anomaly_maps"]), torch.squeeze(output["mask"].int()))