Build CUDA

The following shows how to build a simple working version of Cabana that operates on NVIDIA GPU. This is more complex, and you may choose to initially work with a CPU-only build.

Building Kokkos

Existing Kokkos installation

If you prefer to use an existing install of Kokkos, please do so but ensure it has CUDA support enabled.

Please note, you can also obtain Kokkos from spack if desired (using +cuda).

With any existing Kokkos CUDA installation, you can skip to the Cabana build.

Background information

You will need an install of CUDA, which can normally be loaded from a module

   module load cuda

Please ensure you replace the correct architecture flags below for your target platform. For NVIDIA GPU configuration to use the Kokkos CUDA backend, we need to know the architecture capability of the GPU. We can check this by using NVIDIA device query:

  /usr/local/cuda-9.0/samples/1_Utilities/deviceQuery/deviceQuery Starting...

   CUDA Device Query (Runtime API) version (CUDART static linking)

  Detected 1 CUDA Capable device(s)

  Device 0: "GeForce GTX 770"
    CUDA Driver Version / Runtime Version          9.1 / 9.0
    CUDA Capability Major/Minor version number:    3.0
    Total amount of global memory:                 4035 MBytes (4231200768 bytes)
    ( 8) Multiprocessors, (192) CUDA Cores/MP:     1536 CUDA Cores
    GPU Max Clock rate:                            1189 MHz (1.19 GHz)
    Memory Clock rate:                             3505 Mhz
    Memory Bus Width:                              256-bit
    L2 Cache Size:                                 524288 bytes

    ...

  deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.1, CUDA Runtime Version = 9.0, NumDevs = 1
  Result = PASS

The output we are looking for is CUDA Capability Major/Minor version number which for this card is version 3.0. We have used the Kepler30 option in the example Kokkos configuration below to reflect this.

Building from source

To build Kokkos manually:

    # TODO: YOU, THE USER, SHOULD CHANGE THESE TO YOUR DESIRED PATHS
    export KOKKOS_SRC_DIR=`pwd`/kokkos
    export KOKKOS_INSTALL_DIR=$KOKKOS_SRC_DIR/build/install

    cd ./kokkos
    mkdir build
    cd build
    cmake \
      -D CMAKE_BUILD_TYPE="Release" \
      -D CMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR \
      -D Kokkos_ENABLE_SERIAL=ON \
      -D Kokkos_ENABLE_OPENMP=ON \
      -D Kokkos_ENABLE_CUDA=ON \
      -D Kokkos_ENABLE_CUDA_LAMBDA=ON \
      -D Kokkos_ARCH_KEPLER30=ON \
      \
      .. ;
    make install
    cd ../.. # Go back to top level dir 

There are a few key features to this configuration:

• CUDA-enabled builds previously required explicitly using nvcc_wrapper to compile Kokkos. You can now simply change the host C++ compiler if needed.
• We have specified the NVIDIA GPU compute capability as an extra compiler flag.
• We use lambda functions with Kokkos within Cabana for CUDA builds so we enable that option as well.

Extra Kokkos options may always be added or options removed from this configuration depending on the architecture for which Kokkos and Cabana are being configured.

Building Cabana

    # TODO: YOU, THE USER, SHOULD CHANGE THESE TO YOUR DESIRED PATHS
    export KOKKOS_SRC_DIR=`pwd`/kokkos
    export KOKKOS_INSTALL_DIR=`pwd`/kokkos/build/install
    export CABANA_INSTALL_DIR=`pwd`/Cabana/build/install

    cd ./Cabana
    mkdir build
    cd build
    cmake \
     -D CMAKE_BUILD_TYPE="Debug" \
     -D CMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR \
     -D CMAKE_INSTALL_PREFIX=$CABANA_INSTALL_DIR \
     -D Cabana_REQUIRE_CUDA=ON \
     -D Cabana_ENABLE_TESTING=ON \
     -D Cabana_ENABLE_EXAMPLES=ON \
     .. ;
    make install