Debug validation in training

[Kampffussel03]

Hi, I propose a solution for the validation bug, and am wondering if you think this results in a correct output.

In ctrl's config file, I added a path to the tracklet proposal file in the data dict's validation entry:

val=dict(
            pipeline=eval_pipeline,
            min_tracklet_points=1,
            samples_per_gpu=2,
            tracklet_proposals_file='./data/waymo/tracklet_data/case_try_out/fsd_base_vehicle_val.pkl',)
        ),

In the evaluate function from WaymoTrackletDataset, I changed the default for pklfile_prefix from None to a default path. There, I also changed the path to the gt file to a generated val_gt.file.

ret_bytes = subprocess.check_output(
            'mmdet3d/core/evaluation/waymo_utils/' +
            f'compute_detection_metrics_main {pklfile_prefix}.bin ' +
            f'{waymo_root}/val_gt.bin',

I think all of these few changes sovled the bug. I noticed that the validation here is not, as usual, based on the same loss metrics as for training, but is just based on waymo's eval tool. Please let me know if my solution and observation here is wrong.

2024-06-23 12:40:07,309 - mmdet - INFO - workflow: [('train', 1)], max: 1 epochs
2024-06-23 12:40:07.618643: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
/workspace/SST/mmdet3d/core/bbox/structures/lidar_tracklet.py:356: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  torch.tensor(world2tgt_pose, device=pose.device)
/workspace/SST/mmdet3d/datasets/pipelines/tracklet_pipelines.py:168: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  tgt_pose_inv = torch.tensor(tgt_pose_inv, device=center_pose.device)
Empty input occurs!!!
Empty input occurs!!!
Empty input occurs!!!
Empty input occurs!!!
2024-06-23 12:40:19,746 - mmdet - INFO - Epoch [1][50/52]       lr: 1.003e-05, eta: 0:00:00, time: 0.227, data_time: 0.053, memory: 7029, loss_rcnn_cls: 0.4197, num_pos_rois: 151.5400, num_neg_rois: 4.0400, loss_rcnn_bbox: 2.8161, loss_rcnn_corner: 0.8765, refined_iou: 0.3921, roi_iou: 0.7117, num_good: 6.3400, num_good_rois: 90.9000, loss: 4.1123, grad_norm: 80.9055
2024-06-23 12:40:19,906 - mmdet - INFO - Saving checkpoint at 1 epochs
[                                                  ] 0/52, elapsed: 0s, ETA:/workspace/SST/mmdet3d/core/bbox/structures/lidar_tracklet.py:356: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  torch.tensor(world2tgt_pose, device=pose.device)
/workspace/SST/mmdet3d/datasets/pipelines/tracklet_pipelines.py:168: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  tgt_pose_inv = torch.tensor(tgt_pose_inv, device=center_pose.device)
[>                                                 ] 2/52, 4.1 task/s, elapsed: 0s, ETA:    12sEmpty input occurs!!!
Empty input occurs!!!
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 52/52, 28.9 task/s, elapsed: 2s, ETA:     0s
Starting convert to waymo ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 52/52 [00:00<00:00, 2517.59it/s]

Convert finished.
44165 examples found.

OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2: [mAP 0.0221724] [mAPH 0.0221197]
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2: [mAP 0] [mAPH 0]
OBJECT_TYPE_TYPE_SIGN_LEVEL_2: [mAP 0] [mAPH 0]
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_2: [mAP 0.026753] [mAPH 0.0266863]
RANGE_TYPE_VEHICLE_[30, 50)_LEVEL_2: [mAP 0.0207235] [mAPH 0.020678]
RANGE_TYPE_VEHICLE_[50, +inf)_LEVEL_2: [mAP 0.0167152] [mAPH 0.0166762]
RANGE_TYPE_PEDESTRIAN_[0, 30)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_PEDESTRIAN_[30, 50)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_PEDESTRIAN_[50, +inf)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[0, 30)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[30, 50)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[50, +inf)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_CYCLIST_[0, 30)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_CYCLIST_[30, 50)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_CYCLIST_[50, +inf)_LEVEL_2: [mAP 0] [mAPH 0]

2024-06-23 12:40:32,576 - mmdet - INFO - Exp name: ctrl_veh_24e.py
2024-06-23 12:40:32,576 - mmdet - INFO - Epoch(val) [1][26]     Vehicle/L1 mAP: 0.0222, Vehicle/L1 mAPH: 0.0221, Vehicle/L2 mAP: 0.0000, Vehicle/L2 mAPH: 0.0000, Pedestrian/L1 mAP: 0.0000, Pedestrian/L1 mAPH: 0.0000, Pedestrian/L2 mAP: 0.0000, Pedestrian/L2 mAPH: 0.0000, Sign/L1 mAP: 0.0268, Sign/L1 mAPH: 0.0267, Sign/L2 mAP: 0.0207, Sign/L2 mAPH: 0.0207, Cyclist/L1 mAP: 0.0167, Cyclist/L1 mAPH: 0.0167, Cyclist/L2 mAP: 0.0000, Cyclist/L2 mAPH: 0.0000, Overall/L1 mAP: 0.0130, Overall/L1 mAPH: 0.0129, Overall/L2 mAP: 0.0000, Overall/L2 mAPH: 0.0000

Best Friedrich