How to measure workflow throughput in Knative Eventing

[Yangfisher1]

Hello,

I'm currently working on a serverless project that utilizes Knative Serving & Eventing, and I deployed some functions and a workflow based on these functions with Sequence and Parallel CRDs.

In Serving, evaluating throughput is easy since you just need to call the function and wait for its response. However, when things turn to Eventing, everything is different. Because requests to Sequence or Parallel is asynchronous, you have no way of knowing when the request was completed.

Could anyone provide guidance or recommend tools that are effective in capturing this type of performance metric within Knative Eventing? Any insights or experiences shared would be greatly appreciated.

Thank you!

[pierDipi]

Hi @Yangfisher1, I think the best way is to leverage distributed tracing and metrics.

On the website there are 2 good resources to get started with tracing:

• documentation: https://knative.dev/docs/eventing/accessing-traces/
• blog post https://knative.dev/blog/articles/distributed-tracing/

for metrics, we export them in prometheus format, so any collector capable of collecting prometheus metrics work, here is the list: https://knative.dev/docs/eventing/observability/metrics/eventing-metrics/. Depending on which channel implementation you're using there might be more you can leverage to derive throughput (and latency) metrics.

We also have a monitoring sub-project, this one I'm not sure how up to date is but it might provide some hints / useful info for collecting metrics https://github.com/knative-extensions/monitoring.

Please let me know if this helps or if we can make it better somehow

[Yangfisher1]

@pierDipi Thanks for your reply! Using tracing is a good idea however we might worry about the overhead that tracing brings. Also due to the sample-rate setting, you can't collect thpt of requests with distributed tracing I guess.