Core Migration

Overview

Migration is the movement of data using MPI from one uniquely owned decomposition to another uniquely owned decomposition. In Cabana, the communication plan for migration is encapsulated in the Distributor class and is moved with the migrate function.

In the figure below different spatial domains are represented by the dashed lines and the owning MPI rank by different colors:

Implementation

Header File: Cabana_Distributor.hpp

Usage: Set up communication plan for moving particle data

Interface

template <class DeviceType>
class Distributor

Template Parameters

• DeviceType: The device type for which the halo data will be allocated and where parallel execution will occur

Examples

Example: Distributor Tutorial

Example: Distributor Unit Test

// Get MPI parameters
MPI_Comm_rank( MPI_COMM_WORLD, &comm_rank );
MPI_Comm_size( MPI_COMM_WORLD, &comm_size );

// Define particle data 
using DataTypes = Cabana::MemberTypes<double[3],int>;
using DeviceType = Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>;
Cabana::AoSoA<DataTypes,MemorySpace> aosoa( num_tuple );

// Define MPI ranks to send each particle to
Kokkos::View<int*,DeviceType> export_ranks( "export_ranks", num_tuple );
Kokkos::deep_copy(export_ranks, next_rank);

// Define unique neighboring MPI ranks
std::vector<int> neighbors = { previous_rank, comm_rank, next_rank };

// Create the distributor
Cabana::Distributor<DeviceType> distributor( MPI_COMM_WORLD, export_ranks, neighbors );

// Migrate the data (in-place, could be put into a new AoSoA or slice)
Cabana::migrate( distributor, aosoa );

This is part of the Programming Guide series