Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: limit tags and tag values search #4320

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+678 −383
Open

feat: limit tags and tag values search #4320

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9f36136
limit tags and tag values search
javiermolinar Nov 13, 2024
573695f
added documentation
javiermolinar Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion docs/sources/tempo/api_docs/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -360,6 +360,8 @@ Parameters:
Optional. Along with `end`, defines a time range from which tags should be returned.
- `end = (unix epoch seconds)`
Optional. Along with `start`, defines a time range from which tags should be returned. Providing both `start` and `end` includes blocks for the specified time range only.
- `limit = (integer)`
Optional. Limits the maximum number of tags
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Optional. Limits the maximum number of tags
Optional. Limits the maximum number of tags.



### Search tags V2
Expand All @@ -385,6 +387,8 @@ Parameters:
Optional. Along with `end` define a time range from which tags should be returned.
- `end = (unix epoch seconds)`
Optional. Along with `start` define a time range from which tags should be returned. Providing both `start` and `end` includes blocks for the specified time range only.
- `limit = (integer)`
Optional. Limits the maximum number of tags per scope
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Optional. Limits the maximum number of tags per scope
Optional. Limits the maximum number of tags per scope.


#### Example

Expand Down Expand Up @@ -515,6 +519,8 @@ Parameters:
Optional. Along with `end`, defines a time range from which tags should be returned.
- `end = (unix epoch seconds)`
Optional. Along with `start`, defines a time range from which tags should be returned. Providing both `start` and `end` includes blocks for the specified time range only.
- `limit = (integer)`
Optional. Limits the maximum number of tags values
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Optional. Limits the maximum number of tags values
Optional. Limits the maximum number of tags values.



### Search tag values V2
Expand Down Expand Up @@ -561,7 +567,16 @@ $ curl -G -s http://localhost:3200/api/v2/search/tag/.service.name/values | jq
}
}
```
This endpoint can also receive `start` and `end` optional parameters. These parameters define the time range from which the tags are fetched
Parameters:
- `start = (unix epoch seconds)`
Optional. Along with `end`, defines a time range from which tags values should be returned.
- `end = (unix epoch seconds)`
Optional. Along with `start`, defines a time range from which tags values should be returned. Providing both `start` and `end` includes blocks for the specified time range only.
- `q = (traceql query)`
Optional. A TraceQL query to filter tag values by. Currently only works for a single spanset of `&&`ed conditions. For example: `{ span.foo = "bar" && resource.baz = "bat" ...}`. See also [Filtered tag values](#filtered-tag-values).
- `limit = (integer)`
Optional. Limits the maximum number of tags values


#### Filtered tag values

Expand Down
6 changes: 3 additions & 3 deletions integration/e2e/multi_tenant_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,9 @@ func assertRequestCountMetric(t *testing.T, s *e2e.HTTPService, route string, re

// getAttrsAndSpanNames returns trace attrs and span names
func getAttrsAndSpanNames(trace *tempopb.Trace) traceStringsMap {
rAttrsKeys := collector.NewDistinctString(0)
rAttrsValues := collector.NewDistinctString(0)
spanNames := collector.NewDistinctString(0)
rAttrsKeys := collector.NewDistinctString(0, 0)
rAttrsValues := collector.NewDistinctString(0, 0)
spanNames := collector.NewDistinctString(0, 0)

for _, b := range trace.ResourceSpans {
for _, ss := range b.ScopeSpans {
Expand Down
16 changes: 8 additions & 8 deletions modules/frontend/combiner/search_tag_values.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ var (
_ GRPCCombiner[*tempopb.SearchTagValuesV2Response] = (*genericCombiner[*tempopb.SearchTagValuesV2Response])(nil)
)

func NewSearchTagValues(limitBytes int) Combiner {
func NewSearchTagValues(maxDataBytes int, maxTagsValues uint32) Combiner {
// Distinct collector with no limit
d := collector.NewDistinctStringWithDiff(limitBytes)
d := collector.NewDistinctStringWithDiff(maxDataBytes, maxTagsValues)
inspectedBytes := atomic.NewUint64(0)

c := &genericCombiner[*tempopb.SearchTagValuesResponse]{
Expand Down Expand Up @@ -56,13 +56,13 @@ func NewSearchTagValues(limitBytes int) Combiner {
return c
}

func NewTypedSearchTagValues(limitBytes int) GRPCCombiner[*tempopb.SearchTagValuesResponse] {
return NewSearchTagValues(limitBytes).(GRPCCombiner[*tempopb.SearchTagValuesResponse])
func NewTypedSearchTagValues(maxDataBytes int, maxTagsValues uint32) GRPCCombiner[*tempopb.SearchTagValuesResponse] {
return NewSearchTagValues(maxDataBytes, maxTagsValues).(GRPCCombiner[*tempopb.SearchTagValuesResponse])
}

func NewSearchTagValuesV2(limitBytes int) Combiner {
func NewSearchTagValuesV2(maxDataBytes int, maxTagsValues uint32) Combiner {
// Distinct collector with no limit and diff enabled
d := collector.NewDistinctValueWithDiff(limitBytes, func(tv tempopb.TagValue) int { return len(tv.Type) + len(tv.Value) })
d := collector.NewDistinctValueWithDiff(maxDataBytes, maxTagsValues, func(tv tempopb.TagValue) int { return len(tv.Type) + len(tv.Value) })
inspectedBytes := atomic.NewUint64(0)

c := &genericCombiner[*tempopb.SearchTagValuesV2Response]{
Expand Down Expand Up @@ -113,6 +113,6 @@ func NewSearchTagValuesV2(limitBytes int) Combiner {
return c
}

func NewTypedSearchTagValuesV2(limitBytes int) GRPCCombiner[*tempopb.SearchTagValuesV2Response] {
return NewSearchTagValuesV2(limitBytes).(GRPCCombiner[*tempopb.SearchTagValuesV2Response])
func NewTypedSearchTagValuesV2(maxDataBytes int, maxTagsValues uint32) GRPCCombiner[*tempopb.SearchTagValuesV2Response] {
return NewSearchTagValuesV2(maxDataBytes, maxTagsValues).(GRPCCombiner[*tempopb.SearchTagValuesV2Response])
}
16 changes: 8 additions & 8 deletions modules/frontend/combiner/search_tags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ var (
_ GRPCCombiner[*tempopb.SearchTagsV2Response] = (*genericCombiner[*tempopb.SearchTagsV2Response])(nil)
)

func NewSearchTags(limitBytes int) Combiner {
d := collector.NewDistinctStringWithDiff(limitBytes)
func NewSearchTags(maxDataBytes int, maxTagsPerScope uint32) Combiner {
d := collector.NewDistinctStringWithDiff(maxDataBytes, maxTagsPerScope)
inspectedBytes := atomic.NewUint64(0)

c := &genericCombiner[*tempopb.SearchTagsResponse]{
Expand Down Expand Up @@ -56,13 +56,13 @@ func NewSearchTags(limitBytes int) Combiner {
return c
}

func NewTypedSearchTags(limitBytes int) GRPCCombiner[*tempopb.SearchTagsResponse] {
return NewSearchTags(limitBytes).(GRPCCombiner[*tempopb.SearchTagsResponse])
func NewTypedSearchTags(maxDataBytes int, maxTagsPerScope uint32) GRPCCombiner[*tempopb.SearchTagsResponse] {
return NewSearchTags(maxDataBytes, maxTagsPerScope).(GRPCCombiner[*tempopb.SearchTagsResponse])
}

func NewSearchTagsV2(limitBytes int) Combiner {
func NewSearchTagsV2(maxDataBytes int, maxTagsPerScope uint32) Combiner {
// Distinct collector map to collect scopes and scope values
distinctValues := collector.NewScopedDistinctStringWithDiff(limitBytes)
distinctValues := collector.NewScopedDistinctStringWithDiff(maxDataBytes, maxTagsPerScope)
inspectedBytes := atomic.NewUint64(0)

c := &genericCombiner[*tempopb.SearchTagsV2Response]{
Expand Down Expand Up @@ -121,6 +121,6 @@ func NewSearchTagsV2(limitBytes int) Combiner {
return c
}

func NewTypedSearchTagsV2(limitBytes int) GRPCCombiner[*tempopb.SearchTagsV2Response] {
return NewSearchTagsV2(limitBytes).(GRPCCombiner[*tempopb.SearchTagsV2Response])
func NewTypedSearchTagsV2(maxDataBytes int, maxTagsPerScope uint32) GRPCCombiner[*tempopb.SearchTagsV2Response] {
return NewSearchTagsV2(maxDataBytes, maxTagsPerScope).(GRPCCombiner[*tempopb.SearchTagsV2Response])
}
39 changes: 20 additions & 19 deletions modules/frontend/combiner/search_tags_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import (
func TestTagsCombiner(t *testing.T) {
tests := []struct {
name string
factory func(int) Combiner
limit int
factory func(int, uint32) Combiner
limitBytes int
result1 proto.Message
result2 proto.Message
expectedResult proto.Message
Expand All @@ -31,7 +31,7 @@ func TestTagsCombiner(t *testing.T) {
expectedResult: &tempopb.SearchTagsResponse{TagNames: []string{"tag1", "tag2", "tag3"}, Metrics: &tempopb.MetadataMetrics{}},
actualResult: &tempopb.SearchTagsResponse{},
sort: func(m proto.Message) { sort.Strings(m.(*tempopb.SearchTagsResponse).TagNames) },
limit: 100,
limitBytes: 100,
},
{
name: "SearchTagsV2",
Expand All @@ -49,7 +49,7 @@ func TestTagsCombiner(t *testing.T) {
return scopes[i].Name < scopes[j].Name
})
},
limit: 100,
limitBytes: 100,
},
{
name: "SearchTagValues",
Expand All @@ -59,7 +59,7 @@ func TestTagsCombiner(t *testing.T) {
expectedResult: &tempopb.SearchTagValuesResponse{TagValues: []string{"tag1", "tag2", "tag3"}, Metrics: &tempopb.MetadataMetrics{}},
actualResult: &tempopb.SearchTagValuesResponse{},
sort: func(m proto.Message) { sort.Strings(m.(*tempopb.SearchTagValuesResponse).TagValues) },
limit: 100,
limitBytes: 100,
},
{
name: "SearchTagValuesV2",
Expand All @@ -73,7 +73,7 @@ func TestTagsCombiner(t *testing.T) {
return m.(*tempopb.SearchTagValuesV2Response).TagValues[i].Value < m.(*tempopb.SearchTagValuesV2Response).TagValues[j].Value
})
},
limit: 100,
limitBytes: 100,
},
// limits
{
Expand All @@ -85,7 +85,7 @@ func TestTagsCombiner(t *testing.T) {
actualResult: &tempopb.SearchTagsResponse{},
sort: func(m proto.Message) { sort.Strings(m.(*tempopb.SearchTagsResponse).TagNames) },
expectedShouldQuit: true,
limit: 5,
limitBytes: 5,
},
{
name: "SearchTagsV2 - limited",
Expand All @@ -104,7 +104,7 @@ func TestTagsCombiner(t *testing.T) {
})
},
expectedShouldQuit: true,
limit: 2,
limitBytes: 2,
},
{
name: "SearchTagValues - limited",
Expand All @@ -115,7 +115,7 @@ func TestTagsCombiner(t *testing.T) {
actualResult: &tempopb.SearchTagValuesResponse{},
sort: func(m proto.Message) { sort.Strings(m.(*tempopb.SearchTagValuesResponse).TagValues) },
expectedShouldQuit: true,
limit: 5,
limitBytes: 5,
},
{
name: "SearchTagValuesV2 - limited",
Expand All @@ -130,7 +130,7 @@ func TestTagsCombiner(t *testing.T) {
})
},
expectedShouldQuit: true,
limit: 10,
limitBytes: 10,
},
// with metrics
{
Expand All @@ -141,7 +141,7 @@ func TestTagsCombiner(t *testing.T) {
expectedResult: &tempopb.SearchTagsResponse{TagNames: []string{"tag1", "tag2", "tag3"}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 2}},
actualResult: &tempopb.SearchTagsResponse{},
sort: func(m proto.Message) { sort.Strings(m.(*tempopb.SearchTagsResponse).TagNames) },
limit: 100,
limitBytes: 100,
},
{
name: "SearchTagsV2 - metrics",
Expand All @@ -159,7 +159,7 @@ func TestTagsCombiner(t *testing.T) {
return scopes[i].Name < scopes[j].Name
})
},
limit: 100,
limitBytes: 100,
},
{
name: "SearchTagValues - metrics",
Expand All @@ -169,7 +169,7 @@ func TestTagsCombiner(t *testing.T) {
expectedResult: &tempopb.SearchTagValuesResponse{TagValues: []string{"tag1", "tag2", "tag3"}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 2}},
actualResult: &tempopb.SearchTagValuesResponse{},
sort: func(m proto.Message) { sort.Strings(m.(*tempopb.SearchTagValuesResponse).TagValues) },
limit: 100,
limitBytes: 100,
},
{
name: "SearchTagValuesV2 - metrics",
Expand All @@ -183,12 +183,13 @@ func TestTagsCombiner(t *testing.T) {
return m.(*tempopb.SearchTagValuesV2Response).TagValues[i].Value < m.(*tempopb.SearchTagValuesV2Response).TagValues[j].Value
})
},
limit: 100,
limitBytes: 100,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
combiner := tc.factory(tc.limit)
// TODO ADD LIMIT
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 :)

combiner := tc.factory(tc.limitBytes, 0)

err := combiner.AddResponse(toHTTPResponse(t, tc.result1, 200))
assert.NoError(t, err)
Expand Down Expand Up @@ -228,7 +229,7 @@ func metrics(message proto.Message) *tempopb.MetadataMetrics {
}

func TestTagsGRPCCombiner(t *testing.T) {
c := NewTypedSearchTags(0)
c := NewTypedSearchTags(0, 0)
res1 := &tempopb.SearchTagsResponse{TagNames: []string{"tag1"}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
res2 := &tempopb.SearchTagsResponse{TagNames: []string{"tag1", "tag2"}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
diff1 := &tempopb.SearchTagsResponse{TagNames: []string{"tag1"}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
Expand All @@ -238,7 +239,7 @@ func TestTagsGRPCCombiner(t *testing.T) {
}

func TestTagsV2GRPCCombiner(t *testing.T) {
c := NewTypedSearchTagsV2(0)
c := NewTypedSearchTagsV2(0, 0)
res1 := &tempopb.SearchTagsV2Response{Scopes: []*tempopb.SearchTagsV2Scope{{Name: "scope1", Tags: []string{"tag1"}}}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
res2 := &tempopb.SearchTagsV2Response{Scopes: []*tempopb.SearchTagsV2Scope{{Name: "scope1", Tags: []string{"tag1", "tag2"}}, {Name: "scope2", Tags: []string{"tag3"}}}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
diff1 := &tempopb.SearchTagsV2Response{Scopes: []*tempopb.SearchTagsV2Scope{{Name: "scope1", Tags: []string{"tag1"}}}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
Expand All @@ -255,7 +256,7 @@ func TestTagsV2GRPCCombiner(t *testing.T) {
}

func TestTagValuesGRPCCombiner(t *testing.T) {
c := NewTypedSearchTagValues(0)
c := NewTypedSearchTagValues(0, 0)
res1 := &tempopb.SearchTagValuesResponse{TagValues: []string{"tag1"}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
res2 := &tempopb.SearchTagValuesResponse{TagValues: []string{"tag1", "tag2"}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
diff1 := &tempopb.SearchTagValuesResponse{TagValues: []string{"tag1"}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
Expand All @@ -265,7 +266,7 @@ func TestTagValuesGRPCCombiner(t *testing.T) {
}

func TestTagValuesV2GRPCCombiner(t *testing.T) {
c := NewTypedSearchTagValuesV2(0)
c := NewTypedSearchTagValuesV2(0, 0)
res1 := &tempopb.SearchTagValuesV2Response{TagValues: []*tempopb.TagValue{{Value: "v1", Type: "string"}}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
res2 := &tempopb.SearchTagValuesV2Response{TagValues: []*tempopb.TagValue{{Value: "v1", Type: "string"}, {Value: "v2", Type: "string"}}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
diff1 := &tempopb.SearchTagValuesV2Response{TagValues: []*tempopb.TagValue{{Value: "v1", Type: "string"}}, Metrics: &tempopb.MetadataMetrics{InspectedBytes: 1}}
Expand Down
Loading
Loading