Can't get Grafana/Prometheus metrics to work

[EHRETic]

Hi there,

As I don't know much about Prometheus metrics, I'm struggling a bit to get Authentik Prometheus data source working in my grafana Instance.
One important thing to know, in my environement, all docker instances have a dedicated IP address (macvlan), so port mapping is not working as expected. They are also in the same subnet, so there is no firewall/routing between Grafana & Authentik IPs.

This i what I put to the data source configuration (IP is my authentik server instance - not the worker instance)

What I've noticed so far, if I comment the variable AUTHENTIK_LISTEN__METRICS (set to 9300), I get the following error:

If I make it active again, I get the following:

So variable seems to make a difference and should be active to make Authentik listen to that port but either I'm doing something wrong or there is something not working as expected.

Thanks in advance for your help 😉

PS: Authentik server version is 2023.10.6

[BeryJu]

You need an instance of prometheus which will "scrape" the metrics from authentik and then you can add that prometheus server in grafana

[EHRETic]

Thanks a lot for this super quick reply! 😉👌

The need of an instance wasn't clear when you don't know anything. So, I'm already a step further and have a basic Promotheus container, but the scrape job doesn't seems to get any data. Maybe I need something else, here is the prometheus.yml:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "authentik-test"
    static_configs:
      - targets: ["192.168.X.X:9300"]

And I see this in Prometheus GUI:

I might still miss something, would you know where?

[clyhtsuriva]

Hey @EHRETic, if it's still a current issue or if anyone passes by.
By default, for a container, the port 9300 is exposed but not published.
For example, in the ports section of your compose.yaml (of authentik), you'd need to have something like that in the last line:

    ports:
      - ${COMPOSE_PORT_HTTP:-9000}:9000
      - ${COMPOSE_PORT_HTTPS:-9443}:9443
      - 9300:9300

You can also manually try to query the metrics before testing on prometheus with a simple curl.