same values in google web vitals across transactions

[prim84]

currently we are testing client side metrics using Jmeter selenium framework. Since this framework only provides response time metrics, we are migrating to sitespeed to gather more client side metrics like FCP, LCP,CLS,TTFB, page size.

Application tested- Salesforce (It is not a SPA application . The URL changes for each transaction)
Sitespeed tool version used - 34.12.1 browser time 22.10.2 coach 8.0.2
Browser used - chrome 129

How to reproduce
Execute a user journey (from launch to logout scripted using javascript) and run it locally (not using docker) using the following command.
node sitespeed.js --script.mjs --multi --browsertime.iterations 1 --spa

Issue
For ex, if there are 5 transactions in a user journey. Except the launch transaction, all other transactions shows the same values for google web vitals (TTFB,FCP,LCP,CLS) . But the values for page metrics , CPU metrics, waterfall HAR, Coach score varies.

Transaction Name Total page transfer size Request Fully Loaded TTFB FCP LCP
Launch 7 10 9.25 1.78 1.96 5.23
Login 7.6 226 21.49 2.37 2.52 9.69
Search 39.2 5 2.218 2.37 2.52 9.69
Edit 18.3 7 834ms 2.37 2.52 9.69

[soulgalore]

Hi @prim84 sorry I missed your message. Those metrics are direct from the browser APIs. At the moment Chrome and other s do not update them (I think your app is SPA). Until soft navigations and browser update the APIs, they will be the same. I usually use visuals metrics for those pages to get some valuable data from the tests.

[prim84]

Hi @soulgalore.. Salesforce has mentioned that their application is not a SPA.. But the lightening components uses AJAX that refreshes parts of the page. I also logged the URL between the transactions, and it changes like below.
Launch -test.salesforce.com
Login - https://XXXX--XXX.sandbox.lightning.force.com/one/one.app
Contact view - https://XXX--XXX.sandbox.lightning.force.com/lightning/r/Contact/.........../view?ws=%2Flightning%2Fr%2FAccount%..........%2Fview
Edit - When I click edit button, a pop up appears, and the URL does not change

[prim84]

@soulgalore please let us know the what are the dependencies required for visualmetrics.

[soulgalore]

Hi @prim84 what OS are you using? You need FFmpeg and a couple of Python dependencies. For Linux you can see what's needed at https://www.sitespeed.io/documentation/sitespeed.io/installation/#linux

[prim84]

Thanks for the response @soulgalore . We are using windows OS. can you share the link for windows also .

Meanwhile, We did follow the below link and installed the python modules listed. https://github.com/sitespeedio/browsertime/blob/main/.github/workflows/windows.yml#L22-L30C37

But , while executing python visualmetrics.py --check
Compare and convert fails.

Are these also python files? Please let us know from where should that be installed?

[soulgalore]

Ah yes that should work. I think the check could be old, so you can skip that (it was for an old version where imagemagick was used). Let me remove that.

[prim84]

Followed the steps and installed the dependencies. and then tried to execute sitespeed script as node sitespeed.js --script.mjs --multi --browsertime.iterations 1 --browsertime.skipHar --video --visualMetrics. It throws the following error.

[2024-10-25 12:13:35] INFO: Get visual metrics from the video
[2024-10-25 12:13:38] ERROR: VisualMetrics failed to run Error: Command failed with exit code 1: python C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\node_modules\browsertime\browsertime\visualmetrics-portable.py --dir C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\bin\sitespeed-result\05_SF_CV_ContactEdit_spa_mjs\2024-10-25-12-11-52\pages******\05_SF_CV_ContactEdit_02_Login\data\filmstrip\1 --video C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\bin\sitespeed-result\05_SF_CV_ContactEdit_spa_mjs\2024-10-25-12-11-52\pages*m\05_SF_CV_ContactEdit_02_Login\data\video\1.mp4 --orange --force --renderignore 5 --json --viewport --viewportretries 60 --viewportminheight 100 --viewportminwidth 100 -q 75 --logfile C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\bin\sitespeed-result\05_SF_CV_ContactEdit_spa_mjs\2024-10-25-12-11-52\pages**************\05_SF_CV_ContactEdit_02_Login\data\video\1.visualmetrics.log -vvv
C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\node_modules\browsertime\browsertime\visualmetrics-portable.py:502: SyntaxWarning: invalid escape sequence '.'
matcher = re.compile(".com.android.version.")
C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\node_modules\browsertime\browsertime\visualmetrics-portable.py:502: SyntaxWarning: invalid escape sequence '.'
matcher = re.compile(".com.android.version.")
at makeError (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/node_modules/execa/lib/error.js:60:11)
at handlePromise (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/node_modules/execa/index.js:124:26)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async run (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/node_modules/browsertime/lib/video/postprocessing/visualmetrics/visualMetrics.js:129:20)
at async getVideoMetrics (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/node_modules/browsertime/lib/video/postprocessing/visualmetrics/getVideoMetrics.js:33:21)
at async Video.postProcessing (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/node_modules/browsertime/lib/video/video.js:90:22)
at async Iteration.run (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/node_modules/browsertime/lib/core/engine/iteration.js:300:34)
at async Engine.runByScript (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/node_modules/browsertime/lib/core/engine/index.js:304:20)
at async analyzeUrl (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/lib/plugins/browsertime/analyzer.js:210:12)
at async BrowsertimePlugin.processMessage (file:///C:/Tools/node-v21.2.0-win-x64%20_1/node_modules/sitespeed.io-34.12.1/lib/plugins/browsertime/index.js:173:26)
[2024-10-25 12:13:38] ERROR: Log from VisualMetrics: 12:13:38.312 - Processing frames from video C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\bin\sitespeed-result\05_SF_CV_ContactEdit_spa_mjs\2024-10-25-12-11-52\pages*\05_SF_CV_ContactEdit_02_Login\data\video\1.mp4 to C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\bin\sitespeed-result\05_SF_CV_ContactEdit_spa_mjs\2024-10-25-12-11-52\pages**************\05_SF_CV_ContactEdit_02_Login\data\filmstrip\1
12:13:38.315 - [WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File "C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\node_modules\browsertime\browsertime\visualmetrics-portable.py", line 1907, in main
video_to_frames(
File "C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\node_modules\browsertime\browsertime\visualmetrics-portable.py", line 387, in video_to_frames
is_mobile = find_recording_platform(video)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Tools\node-v21.2.0-win-x64 _1\node_modules\sitespeed.io-34.12.1\node_modules\browsertime\browsertime\visualmetrics-portable.py", line 496, in find_recording_platform
proc = subprocess.Popen(command, stderr=subprocess.PIPE, encoding="UTF-8")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Program Files\Python312\Lib\subprocess.py", line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

[prim84]

@soulgalore.. For an SPA application, google web vitals like FCP LCP, TTFB remains the same for all transactions But Page metrics CPU metrics, total blocking time, max potential FID varies.

These repetitive values... should they be considered as wrong values or it is the right values since that is how the application is built ( it refreshes only a portion of the page).

[soulgalore]

Chrome only report FCP/LCP etc for the first view. You can read about how they handle metrics in https://web.dev/articles/vitals-spa-faq

[prim84]

@soulgalore . We tried the same usecase with google lighthouse through the same chrome browser and we are able to see FCP,LCP for all the transactions. This issue is only with sitespeed?

Also is there a way to get google lighthouse metrics through sitespeed? This way we can overcome this challenge. I know visualmetrics is another option. But currently we are facing lots of issues in getting it enabled.

[soulgalore]

We tried the same usecase with google lighthouse through the same chrome browser and we are able to see FCP,LCP for all the transactions. This issue is only with sitespeed?

No I don't think so, can you show me exactly how you do that? Then it's easier to see what's going on.

You can run the lighthouse container to get Lighthouse metrics, but it will get them page by page and not as a user journey.