[FR] Consider dial protocol tracing

[tallclair]

It would be cool to have tracing on the initial dial request, but I think this would require a protocol-level enhancement to DialResponse (grpc).

The initial flow (happy path) looks like:

client -> server -> agent -[ endpoint ]- agent -> server -> client

It would be great to have latency information for each hop. In other words, something like:

1. Server records dial_req received timestamp (and server ID?)
2. Agent records dial_req received timestamp (and agent ID?)
3. Agent records endpoint dial complete timestamp
4. Agent includes traces in the DialResponse
5. Server records dial_resp received timestamp
6. Server adds request & response received timestamps to DialResponse
7. Client constructs the full trace, records latency metrics for each hop, logs full trace at high verbosity

It looks like there's OpenCensus gRPC integration that we should investigate, but I'm not sure if it would be work with our multiplexed streams. At the very least, we should make sure our design fits the OpenCensus tracing spec

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tallclair]

/lifecycle frozen