Node graph displays incorrect values

[joli-sys]

Description

The node graph in Grafana Tempo plugin is showing incorrect values

• Response time values are incorrect

Steps to Reproduce

1. Open Tempo node graph
2. Observe response time values
3. Compare with actual values

Expected Behavior

• Node graph should display accurate values matching actual traffic
• Average response time should match actual values

Current Behavior

• Node graph shows dramatically higher ms/req values for response time

System Information

• Grafana version: 10.1.10
  • Helm deployment ( Helm chart version 8.6.0)
• Tempo version (Tempo distributed): 2.6.0
  • Helm deployment ( Helm chart version 1.21.1)
• Browser: Arc Browser/Chrome

Additional Context

• Screenshot of node graph showing incorrect values
  

• Traces metrics are correct in Tempo
  

Possible Related Issues

• The rps of the service graph node are incorrect #2445

[joe-elliott]

Can you check the underlying histograms to see if they agree with the service graph or not?

traces_service_graph_request_client_seconds

traces_service_graph_request_server_seconds

[joli-sys]

Can you check the underlying histograms to see if they agree with the service graph or not?

traces_service_graph_request_client_seconds

traces_service_graph_request_server_seconds

Hey @joe-elliott , thanks for replying.
It seems like these metrics truly correlates with service_graph values.
Here is graph with average latency per request for last 5 minutes

It seems for me, like traces_service_graph_request_client_seconds and traces_service_graph_request_client_seconds are in ms instead of seconds in reality. We have other Prometheus metrics, where we have our endpoints latency, and it never goes that high. I will try investigate further our setup, but if you have any clue, I would really appreciate any tip.
Thanks a lot!