API metrics for forwarded vs direct calls

[heidihoward]

/api/metrics seems to include endpoints invoked directly by client and forwarded by other CCF nodes. Is there an easy way to distinguish between the two?

[achamayou]

@heidihoward not right now, RpcFrontend::process() would need to look at ctx->get_session_context()->is_forwarded() before/when calling update_metrics().

I guess there's the question of what the metrics would then look like, would we want a breakdown per-category? Should there be a timeout counter on the forwarder side as well?

[heidihoward]

Thanks! I can work around it but in an ideal world it might be useful to have a version of /api/metric for forwarded only and then it could even be broken down by original node. The timeout counter on the forwarding node would also be helpful.

[achamayou]

Broken down by original node is going to be problematic, because node ids are digests of their public keys, and so the document will become rather large.

One approach would be to split stats like "errors": {"forwarded": 5, "local": 3}, which is natural but not backwards compatible.

Another would be to add stats on the side "errors": 8, "errors_forwarded": 3, which is less natural, but wouldn't break existing usage.