Is it possible to add a Final step to a study

[doutriaux1]

I can't find a way to have a "final" step in a study that would happen once all the combination of parameters have been run

For example if I want to generate a report after a study is finished

I "think" I could add a additional step, but I feel like if I have 20 parameters value then that step would be executed 20 times.

What I would like is a "post" step to be ran once all the parameter values have been ran. In this case it would for example generate a pdf with pictures from whatever was run by the study, it could also include plots that show data generated for each set of parameters (but all in one figure).

Does that makes sense?

[FrankD412]

@doutriaux1 -- If I'm understanding correctly, you're looking for a "funnel" that will collect all parameterized steps. Maestro does have something like that. In the depends key for a step, you're probably accustomed to depends: [step1, step2] -- this does a linear dependency on prior steps. If you'd like to funnel all combinations, you can do depends: [step1_*]. The _* is what you're looking for I believe. You can mix and match the funnel behavior with non-funnel dependencies so you can get expansions of it.

There is a caveat to the funnel steps though -- all of the combinations being funneled in must succeed. I've been toying with the idea of relaxing this requirement and making it so that it's the user's responsibility to figure out how to handle it if one of the steps fails (since the primary use case would be post processing); but I've not made up my mind on that one.

Does that help? Is that what you're looking for @doutriaux1 ?

[doutriaux1]

@FrankD412 thanks! This is exactly what I was looking for.

I'm pasting a sample yaml in here based on your doc example to help people looking at this in the future:

description:
    name: hello_world
    description: A simple 'Hello World' study.

env:
    variables:
        OUTPUT_PATH: ./sample_output/hello_world

study:
    - name: hello_world
      description: Say hello to someone!
      run:
          cmd: |
            echo "Hello, $(NAME)!" > hello_world.txt
    - name: step2
      description: Do step 2 with someone.
      run:
          cmd: |
            echo "$(NAME) is doing something" > hello_world.txt
          depends: ["hello_world",]
          
    - name: final_report
      description: Compile everything
      run:
        cmd: |
            echo "We are compiling results in a report" >> compile.txt
        depends: ["hello_world_*", "step2_*"]

global.parameters:
    NAME:
        values: [Pam, Jim, Michael, Dwight]
        label: NAME.%%

Output final_report is executed only once

======================================================================================================================================================
Step Name                 Job ID    Workspace     State        Run Time        Elapsed Time    Start Time           Submit Time          End Time               Number Restarts
------------------------  --------  ------------  -----------  --------------  --------------  -------------------  -------------------  -------------------  -----------------
final_report              --        final_report  INITIALIZED  --:--:--        --:--:--        --                   --                   --                                   0
step2_NAME.Dwight         --        NAME.Dwight   INITIALIZED  --:--:--        --:--:--        --                   --                   --                                   0
hello_world_NAME.Dwight   20010     NAME.Dwight   FINISHED     0d:00h:00m:00s  0d:00h:00m:00s  2021-02-23 06:42:25  2021-02-23 06:42:25  2021-02-23 06:42:25                  0
step2_NAME.Michael        --        NAME.Michael  INITIALIZED  --:--:--        --:--:--        --                   --                   --                                   0
hello_world_NAME.Michael  20009     NAME.Michael  FINISHED     0d:00h:00m:00s  0d:00h:00m:00s  2021-02-23 06:42:25  2021-02-23 06:42:25  2021-02-23 06:42:25                  0
step2_NAME.Pam            --        NAME.Pam      INITIALIZED  --:--:--        --:--:--        --                   --                   --                                   0
step2_NAME.Jim            --        NAME.Jim      INITIALIZED  --:--:--        --:--:--        --                   --                   --                                   0
hello_world_NAME.Pam      20007     NAME.Pam      FINISHED     0d:00h:00m:00s  0d:00h:00m:00s  2021-02-23 06:42:25  2021-02-23 06:42:25  2021-02-23 06:42:25                  0
hello_world_NAME.Jim      20008     NAME.Jim      FINISHED     0d:00h:00m:00s  0d:00h:00m:00s  2021-02-23 06:42:25  2021-02-23 06:42:25  2021-02-23 06:42:25                  0

======================================================================================================================================================

[FrankD412]

Much appreciated, @doutriaux1!