Velero Service Monitor is not discovered by Prometheus Operator (kube-prometheus -stack)

[darnone]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
I have deployed velero into a namespace called velero using helmfile and I have prometheus-operator running in a namespace called monitoring. I have activated metrics with the following values:

metrics:
  enabled: true
  scrapeInterval: 30s
  scrapeTimeout: 10s

  # Pod annotations for Prometheus
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8085"
    prometheus.io/path: "/metrics"

  serviceMonitor:
    autodetect: false
    enabled: true
    annotations: 
      "helm.sh/hook": post-install,post-upgrade
    namespace: monitoring

The service monitor is created in the monitoring namespace but prometheus does not see it as a target. The service monitor produced is:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: velero
  namespace: monitoring
  annotations:
    helm.sh/hook: post-install,post-upgrade
  labels:
    app.kubernetes.io/name: velero
    app.kubernetes.io/instance: velero
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: velero-7.1.1
spec:
  namespaceSelector:
    matchNames:
      - velero
  selector:
    matchLabels:
      app.kubernetes.io/name: velero
      app.kubernetes.io/instance: velero
  endpoints:
  - port: http-monitoring
    interval: 30s
    scrapeTimeout: 10s

What did you expect to happen:

I expected prometheus to find the servicemonitor under service discovery and targets in prometheus UI and scrape metrics

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

nothing appears abnormal in the prometheus or velero logs

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

I am using helmfile v0.166.0
The kube-prometheus-stack is working for default k8s services

Environment:

• helm version (use helm version): v3.15.3
• helm chart version and app version: velero-7.1.1
• Kubernetes version: v1.28.11-eks-db838b0
• Kubernetes installer & version: asdf
• Cloud provider or hardware configuration: AWS EKS
• OS (e.g. from /etc/os-release): Amazon EC2 Linux

[darnone]

I have written to the Velero-users channel on clock but I have not received a reply.

[darnone]

I am attaching the velero pod and deployment
velero-manifests.zip

[phac008]

try adding additionalLabels to serviceMonitor values

 serviceMonitor:
    additionalLabels:
      release: kube-prometheus-stack

[darnone]

That did it. What led you find that fix?

[rverchere]

That did it. What led you find that fix?

By default, if you use the kube-prometheus-stack, prometheus uses some label for discover/scrape metrics: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusioscrape

Prometheus discovers PodMonitors and ServiceMonitors within its namespace, that are labeled with the same release tag as the prometheus-operator release

You can change this behaviour with the prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues but it's more a monitoring issue than velero one.