Why do the last two epochs of EfficientAD training result in significant metric improvements?

[stti98]

Hello,

I have a question regarding the training of EfficientAD:

Regardless of how many epochs I train, all metrics (F1, Image AUROC, Pixel Auroc) increase strongly at the last two epochs. All previous epochs seem to contribute nothing to the good result. What is the reason for this? Is this a peculiarity of EfficientAD?

I have attached two diagrams that show the Imge AUROC score over the trained epochs. The diagrams were created when training the exact same dataset with the exact same configuration. The only difference is the number of epochs trained. It can be seen that regardless of the number of epochs, the last two epochs always lead to a good result. All previous epochs seem to contribute nothing to the result.

The following is the configuration:

dataset:
name: custom_dataset
format: folder
root: ./datasets/MVTec/alle
normal_dir: good # name of the folder containing normal images.
abnormal_dir: [anormal] # name of the folder containing abnormal images.
task: segmentation # classification or segmentation
mask_dir: [ground_truth/anormal] #optional
normal_test_dir: null # optional
extensions: null
split_ratio: 0.2 # normal images ratio to create a test split
seed: 0
image_size: 256 # dimensions to which images are resized (mandatory)
train_batch_size: 1
eval_batch_size: 16
num_workers: 8
normalization: imagenet # data distribution to which the images will be normalized
test_split_mode: from_dir # options [from_dir, synthetic]
val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
transform_config:
train: null
eval: null
val_split_mode: from_test # determines how the validation set is created, options [same_as_test, from_test]
tiling:
apply: false
tile_size: null
stride: null
remove_border_count: 0
use_random_tiling: False
random_tile_count: 16

model:
name: efficient_ad
teacher_out_channels: 384
model_size: small # options: [small, medium]
lr: 0.0001
weight_decay: 0.00001
padding: false
pad_maps: true # relevant for "padding: false", see EfficientAd in lightning_model.py
generic params
normalization_method: min_max # options: [null, min_max, cdf]

metrics:
image:
- F1Score
- AUROC
pixel:
- F1Score
- AUROC
threshold:
method: adaptive #options: [adaptive, manual]
manual_image: null
manual_pixel: null

visualization:
show_images: False # show images on the screen
save_images: True # save images to the file system
log_images: True # log images to the available loggers (if any)
image_save_path: null # path to which images will be saved
mode: full # options: ["full", "simple"]

project:
seed: 42
path: ./results

logging:
logger: [tensorboard] # options: [comet, tensorboard, wandb, csv] or combinations.
log_graph: true # Logs the model graph to respective logger.

optimization:
export_mode: openvino # options: torch, onnx, openvino
PL Trainer Args. Don't add extra parameter here.
trainer:
enable_checkpointing: true
default_root_dir: null
gradient_clip_val: 0
gradient_clip_algorithm: norm
num_nodes: 1
devices: 1
enable_progress_bar: true
overfit_batches: 0.0
track_grad_norm: -1
check_val_every_n_epoch: 1
fast_dev_run: false
accumulate_grad_batches: 1
max_epochs: 300
min_epochs: null
max_steps: 70000
min_steps: null
max_time: null
limit_train_batches: 1.0
limit_val_batches: 1.0
limit_test_batches: 1.0
limit_predict_batches: 1.0
val_check_interval: 1.0
log_every_n_steps: 50
accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
strategy: null
sync_batchnorm: false
precision: 32
enable_model_summary: true
num_sanity_val_steps: 0
profiler: null
benchmark: false
deterministic: false
reload_dataloaders_every_n_epochs: 0
auto_lr_find: false
replace_sampler_ddp: true
detect_anomaly: false
auto_scale_batch_size: false
plugins: null
move_metrics_to_cpu: false
multiple_trainloader_mode: max_size_cycle

[blaz-r]

Hello. This is quite a big leap, I believe it comes from anomaly map normalization that is part of the model. The reason it happens at the end is due to these lines:

anomalib/src/anomalib/models/efficient_ad/lightning_model.py

Lines 245 to 251 in 66532fc

	def on_validation_start(self) -> None:
	"""
	Calculate the feature map quantiles of the validation dataset and push to the model.
	"""
	if (self.current_epoch + 1) == self.trainer.max_epochs:
	map_norm_quantiles = self.map_norm_quantiles(self.trainer.datamodule.val_dataloader())
	self.model.quantiles.update(map_norm_quantiles)

So only on the very last step the quantiles, used for normalizing, are calculated.

However, I'm not entirely sure, @alexriedel1 might know more.

[abc-125]

I think that's most probably the case. EfficientAD produces 2 anomaly maps, which need to be normalized before summarizing them and calculating the anomaly score. I have trained EfficientAD on a custom dataset with normalization on every epoch, and there are no such leaps.

[abc-125]

I have added a quick fix here.

[stti98]

Thank you, now it works i think.

PS: Keep up the good work with this library