What would you like to see in the Grafana Agent?

[rfratto]

Hi everyone!

I'd like to collect feedback from the community for how we can improve the Grafana Agent. The goal is to help us determine what areas need attention or what we can add to improve the Agent.

All feedback is welcome here! Do you have a feature request you're dying to see? Or maybe a pain point that drives you a little crazy? Please share!

If you see a comment from another user you agree with, vote it up! This will help us get a sense of how popular a comment is, which can help with prioritizing.

[BitProcessor]

It would be really nice if grafana agent would be able to add the container image + tag as label to the logs.

This can be helpful when debugging logs during an automatic (or even manual) deploy of a new container image on kubernetes.
Or even to see if logs are still coming from a old version of the image that is supposed to be long gone etc...

[verygoodsoftwarenotvirus]

I want to give an up-front disclaimer to this wall of text that I have great appreciation, respect, and gratitude for Grafana, Grafana Cloud, the Agent, and everybody who works on it. Any requests or suggestions I make here are meant to be humble requests, not vigorous demands. :)

Configuration

The by-and-large impression I get from the Agent documentation is that it is primarily designed to be run either as a sidecar in a Kubernetes pod, or as a daemon on a box. In my case, I'm deploying to Amazon ECS, so a lot of the available documentation has to be taken with a fair amount of salt.

For instance, configuring the agent requires changing the command to point to the relevant config file, but in ECS the only real way to mount such a volume would be to build a one-off container that inherits from the Grafana-provided agent image. So I'd have to have something like:

FROM grafana/agent:v0.21.2

ADD ./configs/agent-config.yaml /etc/agent/agent.yaml

then build and push that image to ECR and use it as the sidecar. I know there is a remote configuration option in beta, but a better option for me, IMO, is what the AWS OTel Collector does: allow me to set an environment variable with an SSM Parameter name, and fetch the config from there. This way, I don't have to worry about securing an HTTP route serving a configuration file, because everything is kept within AWS proper.

If I'm allowed to make wishes, a complete start-to-finish guide for using Grafana Agent in ECS/Fargate would be incredible. :)

Debugging

My recent goal was to use Grafana Cloud to monitor my ECS service by using the Grafana Agent as a sidecar. To validate that I had set up my configuration file correctly, I ran the example docker-compose instance, only I made the relevant changes to the config file that I had assumed would yield logs/metrics/traces in Grafana Cloud. The only thing I was able to get working was logs, but the Agent didn't log or note any particular reason why. The only file I changed was this one.

I guess I'd issue a minor challenge to whoever reads this if you're up for it: do what I did. Set up a Stack in Grafana Cloud, and use the provided configuration details to get the local load test stuff working. If you can do it, I'd really really really love some details as to what I did wrong (which I acknowledge is hard to ascertain without seeing my configs). I'll try to reproduce and add supplementary evidence to this portion once I'm back at my personal machine.

I reserve the right to blab more in here later as I think of things. :)

[rfratto]

I do think the matter of not being able to send things to Grafana Cloud from the example as I had described ended up being user error on my part.

Not to play the blame game, but I think we (i.e., Grafana Agent devs) could be more helpful to users here. I've been trying to drive Grafana Agent to be broadly useful in both OSS and Grafana Cloud. Our docs here are more focused on the OSS aspect, which means that Grafana Cloud users might have a hard time if this is the first place they look.

We have Grafana Cloud-specific docs, but we need to start thinking about ways to guide people to the right place to get the docs they're looking for.

After some fiddling, I am able to get metrics and logs to show up,

🎉 Glad to hear it :)

but I can't seem to find a working query that yields traces (``, for instance, yields nothing).

@mapno should be able to help you out better here, but if my understanding of the state of traces is still correct, you currently can't search for traces. Instead, users are recommended to log trace IDs and use derived fields to navigate to a specific trace.

[rfratto]

(Sorry for the late reply, I was on PTO)

[mapno]

but I can't seem to find a working query that yields traces (``, for instance, yields nothing).

@mapno should be able to help you out better here, but if my understanding of the state of traces is still correct, you currently can't search for traces. Instead, users are recommended to log trace IDs and use derived fields to navigate to a specific trace.

As @rfratto points out, Tempo's default functionality is search by ID. So, you'll need to provide an existing trace ID for it to return data.

We're currently working on Tempo Search which will allow users to search for traces based on different parameters. You can already make use of this feature in Tempo and Grafana.

There are other trace discovery mechanism such as logs. You can read about all the available options (including Tempo Search) in the docs.

[jmherbst]

Just wanted to echo @verygoodsoftwarenotvirus's wish here that I would also love to see a complete start-to-finish guide for using Grafana Agent in ECS/Fargate

[tabacco]

ECS/Fargate is not an uncommon platform, and officially supported docs and support for it would be pretty good to have.