GTest development

This is a list of recommendations for GTests development (either fixing current tests or writing new tests and porting from the old test infrastructure).

• Test naming convention:

  TEST_P([TESTSUITE_NAME], [TEST_INFO])
  INSTANTIATE_TEST_SUITE_P([TEST_TYPE], [TESTSUITE_NAME], ...);
  

  TEST_TYPE = {Smoke|Full|Perf}
  

  • Smoke is for small set of short running tests quickly verifying basic functionality
  • Full is for the main set of tests
  • Perf is for dedicated performance tests

  TESTSUITE_NAME = [HW_TYPE]_[NAME]_[DATATYPE]
  HW_TYPE = {CPU|GPU}
  

  • CPU is for the tests which do not require GPU and do not verify kernels

  DATATYPE = {FP8..64|BFP16|BFP8|I64|I32|I16|I8|NONE}
  

  • NONE is for the tests which are datatype agnostic, they must have CPU prefix, like db tests or similar

  NAME = {ALGORITHM_DIRECTION|AnyOtherTestNamePattern}
  TEST_INFO = {Test|Regression|AnyOtherAdditionalTestInfo}
  

  Example:

  TEST_P(GPU_Mha_fwd_FP32, Test)
  INSTANTIATE_TEST_SUITE_P(Smoke, GPU_Mha_fwd_FP32, testing::ValuesIn(GenSmokeTestCases()));
  INSTANTIATE_TEST_SUITE_P(Full, GPU_Mha_fwd_FP32, testing::ValuesIn(GenFullTestCases()));
  

  Will result in the following testname

  Smoke/GPU_Mha_fwd_FP32.Test/0
  ...
  Smoke/GPU_Mha_fwd_FP32.Test/100
  Full/GPU_Mha_fwd_FP32.Test/0
  ...
  Full/GPU_Mha_fwd_FP32.Test/10
  

• All the test parameters must have an appropriate test name generator:

  Either by the operator<<

  friend std::ostream& operator<<(std::ostream& os, const TestCaseType& tc) {
      return os << "Param1: " << tc.param1;
  }
  

  or by the explicit INSTANTIATE_TEST_SUITE_P test name generator:

  INSTANTIATE_TEST_SUITE_P(..., ..., testing::ValuesIn(GenFullTestCases()), [](const auto& case){
      return "Param1: " + std::to_string(case.param.param1);
  });
  

  See https://google.github.io/googletest/advanced.html#specifying-names-for-value-parameterized-test-parameters

• All the tests must not use env variables to adjust it's state. Separated test must be created with appropriate naming instead of using env variables.

• TestSuite/TestCase must detect compatible HW and automatically skip non-compatible. See #2675 and #2605

• Use EXPECT_ macro for non-fatal checks. For the cases when we want to continue the test even if the check failed (for example, various numeric checks).

• Use ASSERT_ macro for fatal check. For the cases when we can't continue the test execution or the test execution does not make sense (for example, failed handle creation).

• Exit from the test as early as possible. Any test skipping functionality must reside in void SetUp() override. For example, right now 198 AddLayerNorm tests take around 20s to skip, but if we move skip routine at the beginning of the SetUp, it takes 1ms.

• Do not use headers if there is no intention to include it into multiple cpp files. Use anonymous namespace for all the helper functions in cpp file.

• Appropriately use testing::Values, testing::ValuesIn, etc. See https://google.github.io/googletest/reference/testing.html#param-generators

• Use EXPECT_PRED* instead of EXPECT_TRUE if possible. See https://google.github.io/googletest/reference/assertions.html#predicates

• Use testsuite shared resources as much as possible. See https://google.github.io/googletest/advanced.html#sharing-resources-between-tests-in-the-same-test-suite
  NOTE: If your test fixture defines SetUpTestSuite() or TearDownTestSuite() they must be declared public rather than protected in order to use TEST_P.

• Reset PRNG everywhere to make the tests order-agnostic: void SetUp() override{ prng::reset_seed(); ... }

• Reset internal environment values.