Trainer Class Predict Func == Decreasing Prediction Time

[semaegrii]

Hi everyone,

I have a dataloader with 100 images.

• If I set batch size =100 in “InferenceDataset” func,Total prediction time==23,6 s, one image =236 ms

• If I set batch size=2 in “InferenceDataset” func, Total prediction time==36,7 s, one image =367 ms

  inference_dataset = InferenceDataset(path=test_images_path, image_size=(256, 256))
  test_dataloader = DataLoader(dataset=inference_dataset, batch_size=100, pin_memory=True)
  torch.cuda.synchronize()
  start=time.time()
  predictions = trainer_result.predict(model=model, dataloaders=test_dataloader)[0]                 
  end=time.time()
  torch.cuda.synchronize()
  print(f'Total prediction time: {round((end-start)*1000,2)} ms ') 
  

• why does batch size affect my prediction time? How can ı fix this problem ?

• Could you give me a suggestion to decrease the estimated time?

  import torch
  print(torch.cuda.is_available())
  print(torch.cuda.current_device())
  print(torch.cuda.device(0))
  print(torch.cuda.device_count())
  print(torch.cuda.get_device_name(0)) 
  

True
0
<torch.cuda.device object at 0x0000013E3BE594B0>
1
NVIDIA RTX A4000

[samet-akcay]

Hi @semaegrii, this might be related to the warm up. As shown by @alexriedel1 here, it might be an idea to add a warmup predictions but exclude them from the throughput computation.

[alexriedel1]

Batch size does in fact matter at inference time! Due to parallel computation the inference will be faster per sample with a higher batch size (that is why for a fair comparison algorithm speed in papers should be measured at batch size 1)

https://huggingface.co/docs/setfit/how_to/batch_sizes

[samet-akcay]

Ah that's a great point!

[semaegrii]

@samet-akcay @alexriedel1 I understand When I calculate it according to the situation you mentioned here, the model's prediction time is approximately 4.7 ms, that is, 209 FPS. This looks nice. However, what I need during this time is to be able to access ('pred_scores', 'pred_labels' ) information in my test images. That's why I use trainer.predict.

[alexriedel1]

This can have various reasons!
Can you tell more about the data you are passing to your model? Dimensions, what kinds of transformation are done before, etc.

Why are you using pin_memore=True? Did you try to optimize the DataLoader in any way? Setting the num_workers e.g.

The model will return the results each time it gets called in this way model(x), so you dont necessarily need the trainer.predict method. Thouth this should not be the bottleneck and I get that you have some advantages using the pytorch lightning api.

[semaegrii]

When I set num_worker to optimize dataloader the prediction time increased even more. num_worker=1,2,4,8,16 I tried them all. I think pin_memore=True is an unnecessary usage. But adding or removing it had no effect.

The dimensions of my test images are 128x128 pix. And I don't do any action to use it in the code below.

     inference_dataset = InferenceDataset(path=images_path, image_size=(256, 256))
     test_dataloader = DataLoader(dataset=inference_dataset, batch_size=100)
     torch.cuda.synchronize()
     start=time.time()
     predictions = trainer.predict(model=model, dataloaders=test_dataloader)[0]
     end=time.time()
     torch.cuda.synchronize()
     print(f'Total prediction time: {round((end-start)*1000,2)} ms ')

But I used the code you suggested as follows:

def pred_time(images):
   input_size = [256, 256]
   model = Padim(layers=['layer1','layer2','layer3'],input_size=input_size,backbone='resnet18').model.eval().cuda()
   time_list = []
   num = len(images)
   exclude_first = 10
   for i in tqdm(range(num)):
       pil_image = Image.open(images[i])
       pil_image = pil_image.convert("RGB")

       transform = transforms.Compose([
           transforms.Resize(input_size),
           transforms.ToTensor(),
       ])
       tensor_image = transform(pil_image).unsqueeze(0).cuda()
       with torch.no_grad():
           torch.cuda.synchronize()
           start = time.time()
           model(tensor_image)
           end=time.time()
           torch.cuda.synchronize()
      
       time_list.append((end - start))

   time_list = time_list[exclude_first:]
   print("+ Frame Per Second: {:.2f}".format(1/(sum(time_list)/(num-exclude_first))))
   print(f'Mean Pred Time:{round(np.mean(time_list)*1000,2)} ms')

I then got a prediction time of around 4.7ms. I can use this method, but I don't know how to access anomaly_map and pred_score information from here. Can you help with that?

[alexriedel1]

you have to save the return of your model call in a variable. then you can access its content. For Padim, the anomaly score is the max of the anomaly map.

In this case the pred_score is calculated as follows:

anomalib/src/anomalib/deploy/inferencers/torch_inferencer.py

Lines 212 to 215 in 4459c2b

	# Case I: Predictions could be a tensor.
	if isinstance(predictions, torch.Tensor):
	anomaly_map = predictions.detach().cpu().numpy()
	pred_score = anomaly_map.reshape(-1).max()

You also might want to post process the map and the score. Please see the torch inference for details:

anomalib/src/anomalib/deploy/inferencers/torch_inferencer.py

Lines 246 to 259 in 4459c2b

	# Common practice in anomaly detection is to assign anomalous
	# label to the prediction if the prediction score is greater
	# than the image threshold.
	pred_label: str | None = None
	if "image_threshold" in metadata:
	pred_idx = pred_score >= metadata["image_threshold"]
	pred_label = "Anomalous" if pred_idx else "Normal"
	pred_mask: np.ndarray | None = None
	if "pixel_threshold" in metadata:
	pred_mask = (anomaly_map >= metadata["pixel_threshold"]).squeeze().astype(np.uint8)
	anomaly_map = anomaly_map.squeeze()
	anomaly_map, pred_score = self._normalize(anomaly_maps=anomaly_map, pred_scores=pred_score, metadata=metadata)

output =  model(tensor_image)
anomaly_map = output["anomaly_map"].detach().cpu().numpy()
pred_score = anomaly_map.reshape(-1).max()

[semaegrii]

I haven't seen this, thank you very much! I wish for convenience. Best regards :)

[semaegrii]

Hello, this time I will ask another question. @samet-akcay @alexriedel1
When I make predictions with the Torch library, my speed increases, but the accuracy and AUC values are decreased this time. What is the reason for this? The Torch library could not make successful inferences in every model. While Padim works better, I got almost no results in Dfkde. I got high accuracy when estimating with Lightning. But is it normal that I get such bad results when I convert it to Torch format and guess? Do you think the model does not support it or am I making a mistake?

[alexriedel1]

can you give some more insights on how you do the inference? looks like something is missing in the torch inferencer

[semaegrii]

sure, here are my codes: @alexriedel1

I edited the torch inference according to my own work. With this function, I obtain the true values (good or anomaly) of my images and the prediction score in the form of a list.

def get_parser() -> ArgumentParser:
    """Get parser.
    Returns:
        ArgumentParser: The parser object.
    """
    parser = ArgumentParser()
    parser.add_argument("--model", type=str, default="padim", help="Name of the algorithm to train/test")
    parser.add_argument("--config", type=str, default="src/anomalib/models/padim/config.yaml", help="Path to a model config file")
    parser.add_argument("--testdirectory", type=str, default="src/datasets/boyteks/test", help="Path to a model config file")
    parser.add_argument("--checkpoint",type=str,default='results/padim/boyteks/run/weights/torch/model.pt',help="Path to the metadata file.")
    return parser

def pre_process(device,image, transform):
    processed_image = transform(image=image)["image"].to(device)
    if len(processed_image) == 3:
        processed_image = processed_image.unsqueeze(0)
    return processed_image

def testTorch(device,images_path,images,model):    
    labelList=[]
    predList=[]
    elapsedTime=[]
    sayac=0
    for image in images:
        with torch.no_grad():
            start = time.time()
            prediction = model(image.to(device))
            end = time.time()
            elapsedTime.append((end-start)*1000)
        if isinstance(prediction, torch.Tensor): 
            anomaly_map = prediction.detach().cpu().numpy() 
            pred_score = anomaly_map.reshape(-1).max()
            predList.append(pred_score)

args=get_parser().parse_args()

testGoodImagesPath = glob.glob(f'{args.testdirectory}/good/images/*')
testBadImagesPath = glob.glob(f'{args.testdirectory}/bad/images/*')
test_images_path = testGoodImagesPath + testBadImagesPath

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
checkpoint = torch.load(args.checkpoint, map_location=device)
model = checkpoint["model"]
metadata = checkpoint["metadata"]
transform = A.from_dict(metadata["transform"])

transformed_images = []
for image_path in test_images_path:
    image = cv2.imread(image_path)
    processed_image = pre_process(device,image,transform)
    transformed_images.append(processed_image)

torch.cuda.empty_cache()
testTorch(device,test_images_path, transformed_images, model)
torch.cuda.synchronize()

Afterwards, I calculate AUC and accuracy values from the formula.

def testScore():
    n_ok=len(testGoodImagesPath)
    n_nok=len(testBadImagesPath)
    
    torch.cuda.empty_cache()
    labelList,predScoreList=testTorch(device,test_images_path,transformed_images,model)
    torch.cuda.synchronize()
       fpr,tpr,threshold=roc_curve(labelList,predScoreList)
    AUC = roc_auc_score(labelList, predScoreList)
    AUC=round(AUC,2)
    accuracy= np.max(((n_ok*tpr)+(n_nok*(1-fpr)))/(n_ok+n_nok))

[alexriedel1]

There are some flaws in your script:

1. images are loaded like this in anomalib:

   anomalib/src/anomalib/data/utils/image.py

   Lines 348 to 349 in 05f0281

   	image = cv2.imread(path)
   	image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

2. DFKDE does not output an anomaly map, so I'm not sure if the max function will harm the results.

3. your labelList never gets populated with values

4. generally you should return the lists you populate in testTorch from the function. I don't understand how you can access the lists from another function with a local scope..

5. Inferencing is mode of three steps: pre-process, inference, post-process:
   

   anomalib/src/anomalib/deploy/inferencers/base_inferencer.py

   Lines 78 to 80 in 05f0281

   	processed_image = self.pre_process(image_arr)
   	predictions = self.forward(processed_image)
   	output = self.post_process(predictions, metadata=metadata)

This is important because your raw number results from the model might need to be normalized. You find the post-process step in the inferencer scripts.

[semaegrii]

First of all, thank you for your quick return. @alexriedel1

3. labelList= contains the value 0 or 1.
   predList= It comes from ==> output['pred_score']'.
   "pred_score" contains normalized values, the "roc_curve" and "roc_auc_score" functions calculate it correctly.

4. I didn't add the return part while sending the codes piece by piece, sorry for the confusion.

5. I inherited the "TorchInferencer" class to avoid making mistakes. And my calculation results came back correct. As you said, I made mistakes while trying to write it myself. Resolved.
   Best Regard,