Multi GPU Training Code for Deep Learning with PyTorch. Train PyramidNet for CIFAR10 classification task. This code is for comparing several ways of multi-GPU training.
- Python 3
- PyTorch 1.0.0+
- TorchVision
- TensorboardX
cd single_gpu
python train.py
cd data_parallel
python train.py --gpu_devices 0 1 2 3 --batch_size 768
cd dist_parallel
python train.py --gpu_device 0 1 2 3 --batch_size 768
- batch size: 240
- batch time: 6s
- training time: 22 min
- gpu util: 99 %
- gpu memory: 10 G
- batch size: 768
- batch time: 5s
- training time: 5 min
- gpu util: 99 %
- gpu memory: 10 G * 4