Use EfficientAD with large datasets in a 12gb vram gpu.

[isaacdominguez]

Hi guys, I'm kinda new in training models but after 2 weeks I couldn't train efficientAD using a large dataset, only small ones. How do you do it when you face a large number of images?

Here is my config file:

dataset:
  name: folder
  format: folder
  path: #Path with 12k normal images and 500 abnormal
  normal_dir: nofod
  abnormal_dir: fod
  task: classification
  normal_test_dir: null
  mask: null
  extensions: null
  train_batch_size: 1
  eval_batch_size: 1
  num_workers: 2
  image_size: 256 # dimensions to which images are resized (mandatory)
  center_crop: null # dimensions to which images are center-cropped after resizing (optional)
  normalization: none # data distribution to which the images will be normalized: [none, imagenet]
  transform_config:
    train: null
    eval: null
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)

model:
  name: efficientad
  teacher_out_channels: 384
  model_size: small # options: [small, medium]
  lr: 0.0001
  weight_decay: 0.00001
  padding: true
  # generic params
  normalization_method: min_max # options: [null, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: False # save images to the file system
  log_images: False # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: null # options: torch, onnx, openvino
# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 
  - 0
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 20
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0
  log_every_n_steps: 50
  accelerator: gpu # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

I get CUDA error because I don't have enought VRAM in the GPU. I have tried everything, lowering batch size, num_workers but the result is the same. Any help?

[blaz-r]

Does it crash early into the process or later in training? I think that EfficientAD shouldn't have problems with training on smaller GPUs. What could be causing problems though are metrics, since they store large amount of data if your dataset is big. In any case, it'd be good to know at which point exactly the issue appears.

[isaacdominguez]

Hi, sorry, I should have been more explicit explaining what failed. It crashes at the beginning of the training, when it's calculating the teacher mean in 'channel_mean_std = self.teacher_channel_mean_std(self.trainer.datamodule.train_dataloader())'. Exact error is:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 34.00 MiB. GPU 2 has a total capacty of 10.75 GiB of which 16.50 MiB is free. Process 1569020 has 934.00 MiB memory in use. Process 1525754 has 9.82 GiB memory in use. Of the allocated memory 8.47 GiB is allocated by PyTorch, and 1.16 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I know some gpu vram is in use, and this is not the one with 24gb, but its the same error in another GPU, its just to give you info about the error. I can see while calculating the teacher mean the VRAM going max capacity until crash. The only thing I could do is to reduce the dataset size to fit it in the GPU. About the metrics, with fastflow I never had that problem, gonna try it!

[blaz-r]

This part of EfficientAD was adjusted to not store all the data in memory anymore. Are you using the latest version of anomalib?

[isaacdominguez]

I was using 0.6.0. Updated to 0.7.0 but still getting the same error in calculating teacher channel mean.

[blaz-r]

If I remember correctly this was introduced after 0.7.0, but since the main branch is now on v1, you can just change the code for the teacher mean by hand to this:
https://github.com/openvinotoolkit/anomalib/blob/main/src%2Fanomalib%2Fmodels%2Fimage%2Fefficient_ad%2Flightning_model.py#L152-L189

[isaacdominguez]

@blaz-r Thanks a lot, worked like a charm. Thanks to everyone who is working in this amazing project too.

Have a good day