Unable to configure memcached container resources

[NRCan-LGariepy]

Describe the bug

Seemingly impossible to set values in the mimir-distributed chart to set the memory and CPU requests of the various memcached caches that mimir makes use of (chunks, results, etc). No matter the configuration, the caches always use 500m CPU.

To Reproduce

Setting chunks_chache.resources.requests with something like

  resources:
    requests:
      cpu: 100m

I also tried doing the same in memcached.resources.requests to no avail.

Expected behavior

I expected the caches (for example the chunks cache) to use the requests that I provided. Instead, the caches always reserve 500m CPU, no matter what I set.

Environment

• Infrastructure: RKE2 running on EC2s, managed by Rancher.
• Deployment tool: Helm

[56quarters]

Yes, this is currently hardcoded to 500m within the Helm chart. Seems reasonable to allow this to be configured if you'd like to attempt a PR.

[edwintye]

All the cache is named using - which is quite confusing. The hard coded 500m is in the else part of the statement https://github.com/grafana/mimir/blob/mimir-distributed-5.5.1/operations/helm/charts/mimir-distributed/templates/memcached/_memcached-statefulset.tpl#L92-L102 so you should be able to set the cpu like this (I just use a random number here so it is easier to search in the output).

chunks-cache:
  enabled: true
  resources:
    requests:
      cpu: 321m
      memory: 2480Mi
    limits:
      memory: 2480Mi
index-cache:
  enabled: true
  resources:
    requests:
      cpu: 321m
metadata-cache:
  enabled: true
  resources:
    requests:
      cpu: 321m
results-cache:
  enabled: true
  resources:
    requests:
      cpu: 321m

However, there is a trap here (which we may or may not have fallen into before) and I feel like it deserves a warning. The resources block is rendered as-is which means that memory will not be set if you choose to use override. Obviously there is an "easy fix" in that you can define memory request/limit in your override too BUT the memory allocation via the command line would still be using the allocatedMemory https://github.com/grafana/mimir/blob/mimir-distributed-5.5.1/operations/helm/charts/mimir-distributed/templates/memcached/_memcached-statefulset.tpl#L107 instead of taking the resources (and performing the corresponding calculation). Therefore it is necessary for you to make sure that the memory set via the resources block is compatible with the allocatedMemory value.

The templated out yaml above will yield (for the chunks cache)

      containers:
        - name: memcached
          image: memcached:1.6.31-alpine
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: 2480Mi
            requests:
              cpu: 321m
              memory: 2480Mi
          ports:
            - containerPort: 11211
              name: client
          args:
            - -m 8192
            - --extended=modern,track_sizes
            - -I 1m
            - -c 16384
            - -v
            - -u 11211

which means that oom kill will naturally happen as the cache fills up. An additional condition to calculate a "new allocated memory" if the resource override is used would be very nice and minimize mistakes.