Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce index cache tracing #5699

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Introduce index cache tracing #5699

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f6cd2da
introduce index cache tracing
yeya24 Dec 7, 2023
c4f1338
changelog update
yeya24 Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
* [CHANGE] Index Cache: Multi level cache backfilling operation becomes async. Added `-blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency` and `-blocks-storage.bucket-store.index-cache.multilevel.max-async-buffer-size` configs and metric `cortex_store_multilevel_index_cache_backfill_dropped_items_total` for number of dropped items. #5661
* [FEATURE] Ingester: Add per-tenant new metric `cortex_ingester_tsdb_data_replay_duration_seconds`. #5477
* [FEATURE] Query Frontend/Scheduler: Add query priority support. #5605
* [FEATURE] Index Cache: Add tracing instrumentation. #5699
* [ENHANCEMENT] Store Gateway: Added `-store-gateway.enabled-tenants` and `-store-gateway.disabled-tenants` to explicitly enable or disable store-gateway for specific tenants. #5638
* [ENHANCEMENT] Compactor: Add new compactor metric `cortex_compactor_start_duration_seconds`. #5683
* [ENHANCEMENT] Upgraded Docker base images to `alpine:3.18`. #5684
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ require (
github.com/stretchr/testify v1.8.4
github.com/thanos-io/objstore v0.0.0-20231123170144-bffedaa58acb
github.com/thanos-io/promql-engine v0.0.0-20231127105941-257543af55e8
github.com/thanos-io/thanos v0.32.5-0.20231204192512-28407d61e72d
github.com/thanos-io/thanos v0.32.5-0.20231207001313-e7aecb401f54
github.com/uber/jaeger-client-go v2.30.0+incompatible
github.com/weaveworks/common v0.0.0-20221201103051-7c2720a9024d
go.etcd.io/etcd/api/v3 v3.5.10
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1509,8 +1509,8 @@ github.com/thanos-io/objstore v0.0.0-20231123170144-bffedaa58acb h1:3s/a99MWpt5Z
github.com/thanos-io/objstore v0.0.0-20231123170144-bffedaa58acb/go.mod h1:RMvJQnpB4QQiYGg1gF8mnPJg6IkIPY28Buh8f6b+F0c=
github.com/thanos-io/promql-engine v0.0.0-20231127105941-257543af55e8 h1:KX57eKPq3yzX7ENyd0+fOfPb6v9tjOCLRT8/9waWs/w=
github.com/thanos-io/promql-engine v0.0.0-20231127105941-257543af55e8/go.mod h1:vfXJv1JXNdLfHnjsHsLLJl5tyI7KblF76Wo5lZ9YC4Q=
github.com/thanos-io/thanos v0.32.5-0.20231204192512-28407d61e72d h1:GJhhQxnEENOF0ZgHfK2WAC0qwQhAnexld0Ujhe0xgrY=
github.com/thanos-io/thanos v0.32.5-0.20231204192512-28407d61e72d/go.mod h1:tADvTBUwVH/yjWymDztTV8/bErQVIGFlHIZqXc07QQo=
github.com/thanos-io/thanos v0.32.5-0.20231207001313-e7aecb401f54 h1:DVpL9tb0ZlFXrpZnLXSGfC4E/mlLrlN3iaPvAH+WLos=
github.com/thanos-io/thanos v0.32.5-0.20231207001313-e7aecb401f54/go.mod h1:tADvTBUwVH/yjWymDztTV8/bErQVIGFlHIZqXc07QQo=
github.com/themihai/gomemcache v0.0.0-20180902122335-24332e2d58ab h1:7ZR3hmisBWw77ZpO1/o86g+JV3VKlk3d48jopJxzTjU=
github.com/themihai/gomemcache v0.0.0-20180902122335-24332e2d58ab/go.mod h1:eheTFp954zcWZXCU8d0AT76ftsQOTo4DTqkN/h3k1MY=
github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
Expand Down
16 changes: 9 additions & 7 deletions pkg/storage/tsdb/index_cache.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,41 +204,43 @@ func NewIndexCache(cfg IndexCacheConfig, logger log.Logger, registerer prometheu
iReg = prometheus.WrapRegistererWith(prometheus.Labels{"level": fmt.Sprintf("L%v", i)}, registerer)
}

var (
cache storecache.IndexCache
err error
)
switch backend {
case IndexCacheBackendInMemory:
c, err := newInMemoryIndexCache(cfg.InMemory, logger, iReg)
cache, err = newInMemoryIndexCache(cfg.InMemory, logger, iReg)
if err != nil {
return c, err
return cache, err
}
caches = append(caches, c)
enabledItems = append(enabledItems, cfg.InMemory.EnabledItems)
case IndexCacheBackendMemcached:
c, err := newMemcachedIndexCacheClient(cfg.Memcached.ClientConfig, logger, registerer)
if err != nil {
return nil, err
}
// TODO(yeya24): expose TTL
cache, err := storecache.NewRemoteIndexCache(logger, c, nil, iReg, defaultTTL)
cache, err = storecache.NewRemoteIndexCache(logger, c, nil, iReg, defaultTTL)
if err != nil {
return nil, err
}
caches = append(caches, cache)
enabledItems = append(enabledItems, cfg.Memcached.EnabledItems)
case IndexCacheBackendRedis:
c, err := newRedisIndexCacheClient(cfg.Redis.ClientConfig, logger, iReg)
if err != nil {
return nil, err
}
// TODO(yeya24): expose TTL
cache, err := storecache.NewRemoteIndexCache(logger, c, nil, iReg, defaultTTL)
cache, err = storecache.NewRemoteIndexCache(logger, c, nil, iReg, defaultTTL)
if err != nil {
return nil, err
}
caches = append(caches, cache)
enabledItems = append(enabledItems, cfg.Redis.EnabledItems)
default:
return nil, errUnsupportedIndexCacheBackend
}
caches = append(caches, storecache.NewTracingIndexCache(backend, cache))
}

return newMultiLevelCache(registerer, cfg.MultiLevel, enabledItems, caches...), nil
Expand Down
2 changes: 1 addition & 1 deletion pkg/storage/tsdb/multilevel_cache_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ func Test_MultiIndexCacheInstantiation(t *testing.T) {
cfg: IndexCacheConfig{
Backend: "inmemory",
},
expectedType: &InMemoryIndexCache{},
expectedType: &storecache.TracingIndexCache{},
},
"instantiate multiples backends - inmemory/redis": {
cfg: IndexCacheConfig{
Expand Down
4 changes: 2 additions & 2 deletions vendor/github.com/thanos-io/thanos/pkg/compact/compact.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions vendor/github.com/thanos-io/thanos/pkg/store/cache/factory.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

92 changes: 92 additions & 0 deletions vendor/github.com/thanos-io/thanos/pkg/store/cache/tracing_index_cache.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 6 additions & 1 deletion vendor/github.com/thanos-io/thanos/pkg/store/prometheus.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 6 additions & 1 deletion vendor/github.com/thanos-io/thanos/pkg/store/tsdb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion vendor/modules.txt
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading