Skip to content

Latest commit

 

History

History
372 lines (347 loc) · 11.5 KB

BENCHMARK.md

File metadata and controls

372 lines (347 loc) · 11.5 KB
Raw

Benchmark results

Introductoin

  • Benchmark results using the weights from Detectron model zoo are basically the same. Hence, I won't list them here.
  • All the testing results below are on COCO2017 val-set, using the models trained from scratch on COCO2017 train-set.
  • Evaluation scores are rounded to 3 decimal places and presented in the format of percentage.
  • Multiple evaluation on using pytorch may be listed. Each represents different training trial with same settings.

COCO evaluation metrics

Average Precision
AP50:95 AP at IoU=0.50:0.05:0.95
AP50 AP at IoU=0.50
AP75 AP at IoU=0.75
APs AP for small objects: area < 322
APm AP for medium objects: 322 < area < 962
APl AP for large objects: area > 962
Average Recall
AR1 AR given 1 detection per image
AR10 AR given 10 detection per image
AR100 AR given 100 detection per image
ARs AR for small objects: area < 322
ARm AR for medium objects: 322 < area < 962
ARl AR for large objects: area > 962

Faster-RCNN

e2e_faster_rcnn-R-50-FPN_1x

  • Training command:

    python tools/train_net_step.py \
  --dataset coco2017 --cfg configs/e2e_faster_rcnn_R-50-FPN_1x.yaml \
  --bs 8 --iter_size 2 --use_tfboard

    on two 1080ti GPUs.

Box
source AP50:95 AP50 AP75 APs APm APl AR1 AR10 AR100 ARs ARm ARl
PyTorch 37.1 59.1 40.0 21.5 39.8 48.3 30.8 48.0 50.3 31.4 53.9 63.6
Detectron 36.7 58.4 39.6 21.1 39.8 48.1 30.6 48.0 50.4 31.8 54.1 63.4

  • Total loss comparison:

    img

Mask-RCNN

  • Training command:

    python train_net_step.py \
  --dataset coco2017 --cfg configs/e2e_mask_rcnn_R-50-FPN_1x.yml --use_tfboard

    on four M40 GPUs.

Box
source AP50:95 AP50 AP75 APs APm APl AR1 AR10 AR100 ARs ARm ARl
PyTorch 37.7 59.1 41.0 21.5 40.7 49.6 31.3 48.7 51.1 32.7 54.4 64.9
PyTorch 37.6 59.1 40.9 21.6 40.7 49.0 31.2 49.0 51.4 32.3 55.0 64.7
Detectron 37.7 59.2 40.9 21.4 40.8 49.7 31.3 48.9 51.2 32.3 54.8 64.8
Mask
source AP50:95 AP50 AP75 APs APm APl AR1 AR10 AR100 ARs ARm ARl
PyTorch 33.7 55.5 35.8 14.9 36.3 50.4 29.1 44.2 46.1 26.7 49.7 62.2
PyTorch 33.8 55.7 35.5 15.3 36.3 50.5 29.2 44.5 46.4 26.2 50.0 62.5
Detectron 33.9 55.8 35.8 14.9 36.3 50.9 29.2 44.4 46.2 26.2 50.1 62.0

  • Total loss comparison:

    img

Keypoint-RCNN

  • Training command:

    python tools/train_net_step.py \
  --dataset keypoints_coco2017 --cfg configs/e2e_keypoint_rcnn_R-50-FPN_1x.yaml \
  --bs 8 --iter_size 2 --use_tfboard

    on four 1080 GPUs

Box
source AP50:95 AP50 AP75 APs APm APl AR1 AR10 AR100 ARs ARm ARl
PyTorch 52.2 81.9 56.4 35.3 59.6 68.4 18.3 53.1 61.2 47.0 66.9 75.9
PyTorch 53.5 82.8 58.4 36.7 61.2 69.5 18.6 54.2 62.2 47.8 68.3 76.7
Detectron 53.6 82.8 58.3 36.5 61.2 69.7 18.7 54.3 62.2 47.6 68.3 76.8
Keypoint
source AP50:95 AP50 AP75 APm APl AR1 AR10 AR100 ARm ARl
PyTorch 62.8 85.5 68.3 57.0 71.9 69.7 90.1 74.9 63.7 78.1
PyTorch 63.9 86.0 69.2 58.5 72.7 70.6 90.7 75.8 65.0 78.6
Detectron 64.2 86.4 69.9 58.5 73.4 70.7 90.9 75.9 64.9 79.0

  • Total loss comparison:

    img