What is the correct method for generating the benchmark results for gemm improvements

[goplanid]

I am using OpenBLAS in multi-threaded applications and working on few ideas to improve OpenBLAS. What is the correct method for generating benchmark results for showcasing gemm improvements so that it is accepted by the community.

[martin-frbg]

As with any benchmark, ensure that the hardware is otherwise idle (as far as possible), and running at a constant clock speed. Repeat runs to arrive at a somewhat stable average - the simple codes provided in the benchmark folder read an environment variable OPENBLAS_LOOPS for the number of runs to perform. (If you have more elaborate or application-specific benchmarks, their results would of course be of interest as well, as long as they can be reproduced without the need for commercial and/or difficult to set up software).
And I suspect anything that changes fundamental algorithms like Kazushige Goto's matrix subdivision or otherwise entails a major rewrite would have to show significant improvements across multiple architectures to be considered

[goplanid]

Thank you, Martin, this is useful. To be specific, I am working on OpenBLAS OpenMP multithreaded issues and planning to produce application-specific benchmarks. Let me know if I need to publish any specific benchmarks.

[martin-frbg]

Oh, that's interesting - I am currently looking into OpenMP-related issues and best practises for OMP environment variable setting in an OpenBLAS context myself. Sadly most of the present code dates from the original GotoBLAS of more than ten years ago, where OpenMP was basically employed for the locking mechanisms and the only option to get thread safety at all. There are a few spots that make use of OMP_PARALLEL directives, but these are all in the LAPACK Fortran codes imported from the Reference-LAPACK project