Limit the amount of tags in an API response

[joey-grafana]

Is your feature request related to a problem? Please describe.
Up until recently, we attempted to display unbounded amounts of tags/values on the front end.

Describe the solution you'd like
The limiting of the amount of tags/values send in API responses.

Additional context
We currently limit the amount of tags displayed in front end drop downs to 1000 tags (however users can still search for all of them as they are stored in memory).

Some customers can have 100k+ tags and sending all these back to the front end creates a sluggish experience if not limited in the front end. This limitation has to be added everywhere the tags are rendered. However, the response size is unbounded meaning the actual request itself could freeze up the browser if it is too large.

Ideally, we would not send so many tags to the front end.

While there are improvements (currently in progress) to allow the rendering of large amounts of drop down items within the select component, the question remains, should we really render that many?

[joe-elliott]

Tempo does have the ability to limit the number of tags sent back but only by total bytes and not tag count.

max_bytes_per_tag_values_query

https://grafana.com/docs/tempo/latest/configuration/#standard-overrides

This is currently set to 5MB across the board in cloud. Perhaps that's too much?

[javiermolinar]

Would it be possible to sort and paginate them?

[joe-elliott]

Would it be possible to sort and paginate them?

I'm not sure how to do this well. Imagine a query that finds 100 total values and we return 10 values per page. What do we do if:

1. Someone queries the first page and receives 10 values.
2. New data is ingested by Tempo that adds 10 more values BEFORE the data just returned.
3. Someone queries the second page. They will receive the same 10 values as they did in step 1 b/c these values got shifted down 10 slots.

This would only work on historical data with an absolute time range and in no other case.

[joey-grafana]

max_bytes_per_tag_values_query is good, but the user has to explicitly set this.

If they don't and the front end is sent tens of thousands of tags it could previously hang the UI. We've since limited the items we render in the drop down but we still receive large amounts of tags in the API response which may cause a slowdown in the UI as the data is being transferred. The user could still raise an escalation here :D

It sounds like the best option is to just set max_bytes_per_tag_values_query but I was looking for some kind of sensible default here.

This is currently set to 5MB across the board in cloud. Perhaps that's too much?

Given that I've heard of responses with 132,000 tags or possibly even larger perhaps 5MB is too high. cc @mapno

[joe-elliott]

Given that I've heard of responses with 132,000 tags or possibly even larger perhaps 5MB is too high. cc @mapno

Agreed. Given that this has turned into a discussion on balancing cloud config, i think we should take it offline.