How to optimize compression within Tempo.

[wkshare]

What is meant below? Please help me. how to fix issues.

Aug 09 14:23:39 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:23:39.747526423Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:29:39 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:29:39.75205824Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:30:09 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:30:09.749287422Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:30:39 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:30:39.747262209Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:31:09 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:31:09.748184887Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:31:39 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:31:39.747265393Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:32:09 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:32:09.748364727Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:32:39 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:32:39.748066537Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:33:09 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:33:09.747943817Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Aug 09 14:33:39 ech-10-157-130-81 grafana_tempo[534813]: level=warn ts=2022-08-09T06:33:39.747582551Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"

Tempo Configs:

ingester:
  trace_idle_period: 10s               # the length of time after a trace has not received spans to consider it complete and flush it
  max_block_bytes: 1_000_000           # cut the head block when it hits this size or ...
  max_block_duration: 5m               #   this much time passes
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
    heartbeat_period: 100ms

compactor:
  compaction:
    compaction_window: 1h              # blocks in this time window will be compacted together
    max_block_bytes: 100_000_000       # maximum size of compacted blocks
    block_retention: 1h
    compacted_block_retention: 10m

memberlist:
  abort_if_cluster_join_fails: false
  bind_port: 7947
  join_members:
{% for host in groups.tempo %}
  - {{ host }}:7947
{% endfor %}

storage:
  trace:
    backend: s3                        # backend configuration to use
    block:
      bloom_filter_false_positive: .05 # bloom filter false positive rate.  lower values create larger filters but fewer false positives
      index_downsample_bytes: 1000     # number of bytes per index record
      encoding: zstd                   # block encoding/compression.  options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd, s2
    wal:
      path: /monit/data/tempo/wal             # where to store the the wal locally
      encoding: snappy                 # wal encoding/compression.  options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd, s2
    s3:
      bucket: tempo                    # how to store data in s2
      endpoint: {{ monit_gateway_vip }}:9299
      access_key: monituser
      secret_key: monituser
      insecure: true
    pool:
      max_workers: 100                 # worker pool determines the number of parallel requests to the object store backend
      queue_depth: 10000

querier:
  frontend_worker:
    frontend_address: 127.0.0.1:23200

[joe-elliott]

I provided an answer in the issue you filed with the same question:
#1626

Now that I'm seeing your config I'd recommend adding memberlist as the kvstore for your compactors like this:

compactor:
  ring:
    kvstore:
      store: memberlist

[wkshare]

@joe-elliott Thank you, I will try to test it.