Improve reliability of Playwright visual regression testing

[danielmx-dev]

Description

Recently (as of the date of creating this issue), the PW E2E tests started failing without any noticeable change to the codebase. When looking at the reasons, the source of the failures were a couple of toHaveScreenshot assertions.

When looking at the visual diff, we notice that the changes are minimal, the contents did not change at all, only moved slightly lower.

These screenshots were updated in #9712 to make the tests pass again; however, we need to make sure that the tests are more reliable and at the same time help us to spot regressions/unintended behavior. (UPDATE: I had to disable all visual regression tests in #9718 since they started failing again.)

Acceptance criteria

We have a couple of options here:

A) Update the maxDiffPixelRatio setting in the playwright.config.ts file so it has a higher tolerance for changes; However, how can we decide which value is acceptable?
B) Avoid using toHaveScreenshot and use other assertions instead?

We could also have a combination of both, in which we make sure that the contents match our expectation, and that the visual diff is close enough (with a higher threshold for tolerated changes).

Testing instructions

1. See instructions for running pw tests locally in the tests/e2e-pw/README.md file.

Dev notes

maxDiffPixelRatio location in the code

[danielmx-dev]

There's some discussion here about other potential solutions. E.g. Separating visual regression into their own workflow: p1731518139549869-slack-CGGCLBN58

[alopezari]

Thanks for creating this issue @danielmx-dev!

I think a combination of masking the locator or component we want to test + adding some tolerance to the pixel amount might make the test more robust.

This post shows an example:
https://medium.com/@thananjayan1988/playwright-mask-the-dynamic-web-content-in-webpage-2e5583a204bf