docs: updates and setup script to add executable to path (#415)

docs updates and setup script to add executable to path

docs/add-a-dryrun.rst

@@ -24,9 +24,9 @@ For example, if you are contributing a system called foo you would test it with
 
   - name: Dry run dynamic saxpy on dynamic foo
     run: |
-      ./bin/benchpark system init --dest=foo-system foo
-      ./bin/benchpark experiment init --dest=saxpy-openmp saxpy
-      ./bin/benchpark setup ./saxpy ./foo-system workspace/
+      benchpark system init --dest=foo-system foo
+      benchpark experiment init --dest=saxpy-openmp saxpy
+      benchpark setup ./saxpy ./foo-system workspace/
       . workspace/setup.sh
       ramble \
         --workspace-dir workspace/saxpy/foo-{hashID}/workspace \
@@ -43,9 +43,9 @@ For example, if you are contributing a new benchmark called bar:
 
   - name: Dry run dynamic bar on dynamic genericx86
     run: |
-      ./bin/benchpark system init --dest=x86-system genericx86 
-      ./bin/benchpark experiment init --dest=bar-benchmark bar
-      ./bin/benchpark setup ./bar-benchmark ./x86-system workspace/
+      benchpark system init --dest=x86-system genericx86 
+      benchpark experiment init --dest=bar-benchmark bar
+      benchpark setup ./bar-benchmark ./x86-system workspace/
       . workspace/setup.sh
       ramble \
         --workspace-dir workspace/new-benchmark/genericx86-{hashID}/workspace \

docs/add-a-system-config.rst

@@ -210,7 +210,7 @@ this will generate the software configurations for spack (``software.yaml``). Th
     """
 
 Once the modified system subclass is written, run: 
-``./bin/benchpark system init --dest=modifiedx86-system modifiedx86``
+``benchpark system init --dest=modifiedx86-system modifiedx86``
 
 This will generate the required yaml configurations for your system and you can validate it works with a static experiment test.
 
@@ -220,9 +220,9 @@ Validating the System
 
 To manually validate your new system, you should initialize it and run an existing experiment such as saxpy. For example::
 
-  ./bin/benchpark system init --dest=modifiedx86-system modifiedx86
-  ./bin/benchpark experiment init --dest=saxpy saxpy
-  ./bin/benchpark setup ./saxpy ./modifiedx86-system workspace/
+  benchpark system init --dest=modifiedx86-system modifiedx86
+  benchpark experiment init --dest=saxpy saxpy
+  benchpark setup ./saxpy ./modifiedx86-system workspace/
 
 Then you can run the commands provided by the output, the experiments should be built and run successfully without any errors. 
 

docs/add-an-experiment.rst

@@ -49,10 +49,10 @@ or ``openmp`` for an experiment using OpenMP (on a CPU).::
 Multiple types of experiments can be created using variants as well (e.g., strong scaling, weak scaling). See AMG2023 or Kripke for examples.
 
 Once an experiment class has been written, an experiment is initialized with the following command, with any variants that have been defined in your experiment.py passed in as key-value pairs: 
-``./bin/benchpark experiment init --dest {path/to/dest} experiment={experiment_variant} programming_model={prog_model_variant}``
+``benchpark experiment init --dest {path/to/dest} experiment={experiment_variant} programming_model={prog_model_variant}``
 
 For example, to run the AMG2023 strong scaling experiment for problem 1, using CUDA the command would be:
-``./bin/benchpark experiment init --dest amg2023 programming_model=cuda workload=problem1 experiment=strong``
+``benchpark experiment init --dest amg2023 programming_model=cuda workload=problem1 experiment=strong``
 
 Initializing an experiment generates the following yaml files:
 
@@ -70,9 +70,9 @@ Validating the Benchmark/Experiment
 To manually validate your new experiments work, you should initialize an existing system, and run your experiments. 
 For example if you just created a benchmark *baz* with OpenMP and strong scaling variants it may look like this:::
 
-  ./bin/benchpark system init --dest=genericx86-system genericx86 
-  ./bin/benchpark experiment init --dest=baz-benchmark baz programming_model=openmp scaling=strong
-  ./bin/benchpark setup ./baz-benchmark ./x86 workspace/
+  benchpark system init --dest=genericx86-system genericx86 
+  benchpark experiment init --dest=baz-benchmark baz programming_model=openmp scaling=strong
+  benchpark setup ./baz-benchmark ./x86 workspace/
 
 
 When this is complete you have successfully completed the :doc:`benchpark-setup` step and can run and analyze following the Benchpark output or following steps in :doc:`build-experiment`.

docs/benchpark-setup.rst

@@ -11,8 +11,8 @@ To setup an experiment workspace you must first initialize both an experiment an
 Any system or experiment variants are specified at the end of the command as shown below.
 The order of the two init commands does not matter, but they both need to be run before the setup command.::
 
-./bin/benchpark system init --dest=</output/path/to/system_root> <SystemName> compiler=<Compiler>
-./bin/benchpark experiment init --dest=</output/path/to/experiments_root> <Benchmark> experiment=<Experiment> programming_model=<ProgrammingModel>
+benchpark system init --dest=</output/path/to/system_root> <SystemName> compiler=<Compiler>
+benchpark experiment init --dest=</output/path/to/experiments_root> <Benchmark> experiment=<Experiment> programming_model=<ProgrammingModel>
 
 
 Once you have a benchmark experiment to run, along with the programming model to use, and a system to run them on.

docs/getting-started.rst

@@ -14,11 +14,16 @@ Installation
 Git is needed to clone Benchpark, and Python 3.8+ is needed to run Benchpark::
 
     git clone https://github.com/LLNL/benchpark.git
+
+To add the executable to your path, source the setup script::
+
     cd benchpark
+    . setup-env.sh
+    
 
-The executable is in ``/bin``, to check the version you can run:: 
+Now, to check the version you can run:: 
 
-    ./bin/benchpark --version
+    benchpark --version
 
 --------------------
 System Prerequisites

docs/llnl-tutorial.rst

@@ -11,22 +11,28 @@ Running on an LLNL System
 
     We might add console outputs for these, so users know what to expect
 
-This tutorial will guide you through the process of using Benchpark on an LLNL
-system, this example uses the cuda version of the Saxpy benchmark on Tioga. 
-But the steps can be reused for a different system, benchmark or experiment.
+This tutorial will guide you through the process of using Benchpark on LLNL
+systems. 
 
-First, initialize the system for LLNL's tioga system using the existing
+------------------------
+CTS (Ruby, Dane, Magma)
+------------------------
+
+This example uses the openmp version of the Saxpy benchmark on one of our CTS systems (Ruby, Dane, Magma). 
+The variant ``cluster`` determines which of the three systems to initialize.
+
+First, initialize the desired cluster variant of the LLNL cts ruby (or dane, magma) system using the existing
 system specification in Benchpark::
 
-    benchpark system init --dest=tioga-system tioga ~gtl
+    benchpark system init --dest=ruby-system cts cluster=ruby
 
-To run the cuda version of the Saxpy benchmark, initialize it for experiments::
+To run the cuda version of the AMG20223 benchmark, initialize it for experiments::
 
-    benchpark experiment init --dest=saxpy-benchmark saxpy programming_model=cuda
+    benchpark experiment init --dest=amg2023-benchmark amg2023 programming_model=openmp
 
 Then setup the workspace directory for the system and experiment together::
 
-    ./bin/benchpark setup ./saxpy-benchmark ./tioga-system workspace/
+    benchpark setup ./amg2023-benchmark ./ruby-system workspace/
 
 Benchpark will provide next steps to the console but they are also provided here.
 Run the setup script for dependency software, Ramble and Spack::
@@ -35,10 +41,27 @@ Run the setup script for dependency software, Ramble and Spack::
 
 Then setup the Ramble experiment workspace, this builds all software and may take some time::
 
-    cd ./workspace/saxpy-benchmark/Tioga-975af3c/workspace/
+    cd ./workspace/amg2023-benchmark/Cts-6d48f81/workspace/
     ramble --workspace-dir . --disable-progress-bar workspace setup
 
 Next, we run the Saxpy experiments, which will launch jobs through the
 scheduler on Tioga::
 
     ramble --workspace-dir . --disable-progress-bar on
+
+------
+Tioga
+------
+
+This second tutorial will guide you through the process of using the cuda 
+version of the Saxpy benchmark on Tioga. 
+The parameters for initializing the system are slightly different due to the 
+different variants defined for the system. For example, the variant ``~gtl`` turns off gtl-enabled MPI, ``+gtl`` turns it on::
+
+    benchpark system init --dest=tioga-system tioga ~gtl
+    benchpark experiment init --dest=saxpy-benchmark saxpy programming_model=cuda
+    benchpark setup ./saxpy-benchmark ./tioga-system workspace/
+    . workspace/setup.sh
+    cd ./workspace/saxpy-benchmark/Tioga-975af3c/workspace/
+    ramble --workspace-dir . --disable-progress-bar workspace setup
+    ramble --workspace-dir . --disable-progress-bar on

docs/modifiers.rst

@@ -94,39 +94,53 @@ We have implemented a Caliper modifier to enable profiling of Caliper-instrument
 benchmarks in Benchpark. More documentation on Caliper can be found `here
 <https://software.llnl.gov/Caliper>`_.
 
-To turn on profiling with Caliper, add ``--modifier=<caliper_modifier>`` to the Benchpark
+To turn on profiling with Caliper, add ``caliper=<caliper_variant>`` to the experiment init
 setup step::
 
-    ./benchpark setup </output/path/to/experiments_root> </output/path/to/system_root> --modifier=<caliper_modifier> </output/path/to/workspace> 
+    benchpark experiment init --dest=</path/to/experiment_root> caliper=<caliper_modifier> </output/path/to/workspace> 
 
-Valid values for ``<caliper_modifier>`` are found in the **Caliper Modifier**
+Valid values for ``<caliper>`` are found in the **Caliper Variant**
 column of the table below.  Benchpark will link the experiment to Caliper,
 and inject appropriate Caliper configuration at runtime.  After the experiments
 in the workspace have completed running, a ``.cali`` file
 is created which contains the collected performance metrics.
 
-.. list-table:: Available caliper modifiers
+.. list-table:: Available caliper variants
    :widths: 20 20 50
    :header-rows: 1
 
-   * - Caliper Modifier
+   * - Caliper Variant
      - Where Applicable
      - Metrics Collected
-   * - caliper
+   * - time
      - Platform-independent
      - | - Min/Max/Avg time/rank: Minimum/Maximum/Average time (in seconds) across all ranks
        | - Total time: Aggregated time (in seconds) for all ranks
-   * - caliper-mpi
+   * - mpi
      - Platform-independent
      - | - Same as basic caliper modifier above
        | - Profiles MPI functions
-   * - caliper-topdown
-     - x86 Intel CPUs
-     - | - Retiring
-       | - Bad speculation
-       | - Front end bound
-       | - Back end bound
-   * - caliper-cuda
+   * - cuda
      - NVIDIA GPUs
      - | - CUDA API functions (e.g., time.gpu)
+   * - topdown-counters-all
+     - x86 Intel CPUs
+     - | - Raw counter values for Intel top-down analysis (all levels)
+   * - topdown-counters-toplevel
+     - x86 Intel CPUs
+     - | - Raw counter values for Intel top-down analysis (top level)
+   * - topdown-all
+     - x86 Intel CPUs
+     - | - Top-down analysis for Intel CPUs (all levels)
+   * - topdown-toplevel
+     - x86 Intel CPUs
+     - | - Top-down analysis for Intel CPUs (top level)
+
+
+An experiment must inherit from the Caliper experiment class to make use of 
+the Caliper functionality. Most existing experiments should already do this, but if 
+adding to a new experiment, it is as simple as adding it to the class definition signature.
+For example::
+  
+  class Amg2023(Experiment, Caliper):
 

setup-env.sh

@@ -0,0 +1 @@
+export PATH=${PWD}/bin:$PATH