diff --git a/Makefile b/Makefile index e41bbe510..9ff43ca67 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ # These variables follow the naming convention from the GNU Make documentation # but their defaults correspond to the rest of the code (note that changing # libexecdir here wouldn't affect the default path for the channel plugin -# directory used by the dameon for example). +# directory used by the daemon for example). # # https://www.gnu.org/software/make/manual/html_node/Directory-Variables.html prefix ?= /usr diff --git a/README.md b/README.md index 97ea8bc60..a30cbdb9d 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,19 @@ # Icinga Notifications -> **Warning** -> This is an early preview version for you to try, but do not use this in production. There may still be severe bugs -> and incompatible changes may happen without any notice. At the moment, we don't yet provide any support for this. +> [!WARNING] +> This is an early preview version for you to try, but do not use this in production. +> There may still be severe bugs and incompatible changes may happen without any notice. +> At the moment, we don't yet provide any support for this. -Icinga Notifications is a set of components that processes received events from various sources, manages incidents and -forwards notifications to predefined contacts, consisting of: +Icinga Notifications is a set of components that processes received events from various sources, +manages incidents and forwards notifications to predefined contacts, consisting of: -* The Icinga Notifications daemon (this repository), which receives events and sends notifications -* An [Icinga Web module](https://github.com/Icinga/icinga-notifications-web) that provides graphical configuration and further processing of the data collected by the daemon -* And Icinga 2 and other custom sources that propagate state updates and acknowledgement events to the daemon +* The Icinga Notifications daemon (this repository), which receives events and sends notifications. +* An [Icinga Web module](https://github.com/Icinga/icinga-notifications-web) that provides graphical configuration and further processing of the data collected by the daemon. +* Icinga 2 or another sources that provides monitoring events leading to notifications. -## Installation - -To install Icinga Notifications and get started, you first need to clone this repository. -```bash -git clone https://github.com/Icinga/icinga-notifications.git -``` - -Next, you need to provide a `config.yml` file, similar to the [example config](config.example.yml), for the daemon. -It is required that you have created a new database and imported the [schema](schema/pgsql/schema.sql) file beforehand. -> **Note** -> At the moment **PostgreSQL** is the only database backend we support. - -Additionally, it also requires you to manually insert items into the **source** table before starting the daemon. -```sql -INSERT INTO source - (id, type, name, icinga2_base_url, icinga2_auth_user, icinga2_auth_pass, icinga2_insecure_tls) -VALUES - (1, 'icinga2', 'Local Icinga 2', 'https://localhost:5665', 'root', 'icinga', 'y'); -``` - -Then, you can launch the daemon with the following command. -```go -go run ./cmd/icinga-notifications --config config.yml -``` +For more information about how to install or use Icinga Notifications, +just follow the [documentation](https://icinga.com/docs/icinga-notifications/latest). ## License diff --git a/doc/01-About.md b/doc/01-About.md new file mode 100644 index 000000000..e3252a879 --- /dev/null +++ b/doc/01-About.md @@ -0,0 +1,41 @@ +# Icinga Notifications + +!!! warning + + This is an early preview version for you to try, but do not use this in production. + There may still be severe bugs and incompatible changes may happen without any notice. + At the moment, we don't yet provide any support for this. + +Icinga Notifications is a set of components that processes received events from various sources, manages incidents and +forwards notifications to predefined contacts, consisting of: + +* The Icinga Notifications daemon (this repository), which receives events and sends notifications. +* An [Icinga Web module](https://github.com/Icinga/icinga-notifications-web) that provides graphical configuration and further processing of the data collected by the daemon. +* Icinga 2 or another sources that provides monitoring events leading to notifications. + +## Big Picture + +Because Icinga Notifications consists of several components, +this section tries to help understand how these components relate. + +First, the Icinga Notifications configuration resides in a SQL database. +It can be conveniently tweaked via Icinga Notifications Web directly from a web browser. +The Icinga Notifications daemon uses this database to read the current configuration. + +As in any Icinga setup, all host and service checks are defined in Icinga 2. +By querying the Icinga 2 API, the Icinga Notifications daemon retrieves state changes, acknowledgements, and other events. +These events are stored in the database and are available for further inspection in the Icinga Notifications Web. + +Depending on its configuration, the daemon will take further action on these events. +This optionally includes escalations that are sent through a channel plugin. +Each such channel plugin implements a domain-specific transport, e.g., the `email` channel sends emails via SMTP. +When configured, Icinga Notifications will use channel plugins to notify end users or talk to other APIs. + +## Installation + +To install Icinga Notifications see [Installation](02-Installation.md). + +## License + +Icinga Notifications and the Icinga Notifications documentation are licensed under the terms of the +GNU General Public License Version 2. diff --git a/doc/02-Installation.md b/doc/02-Installation.md new file mode 100644 index 000000000..501e96e77 --- /dev/null +++ b/doc/02-Installation.md @@ -0,0 +1,120 @@ + +# Installing Icinga Notifications + +The recommended way to install Icinga Notifications is to use prebuilt packages +for all supported platforms from our official release repository. +Please follow the steps listed for your target operating system, +which guide you through setting up the repository and installing Icinga Notifications. + +To upgrade an existing Icinga Notifications installation to a newer version, +see the [Upgrading](04-Upgrading.md) documentation for the necessary steps. + + + +## Setting up the Database + +A MySQL (≥5.5), MariaDB (≥10.1), or PostgreSQL (≥9.6) database is required to run Icinga Notifications. +Please follow the steps listed for your target database, +which guide you through setting up the database and user and importing the schema. + +### Setting up a MySQL or MariaDB Database + +If you use a version of MySQL < 5.7 or MariaDB < 10.2, the following server options must be set: + +``` +innodb_file_format=barracuda +innodb_file_per_table=1 +innodb_large_prefix=1 +``` + +Set up a MySQL database for Icinga Notifications: + +``` +# mysql -u root -p + +CREATE DATABASE notifications; +CREATE USER 'notifications'@'localhost' IDENTIFIED BY 'CHANGEME'; +GRANT ALL ON notifications.* TO 'notifications'@'localhost'; +``` + +After creating the database, import the Icinga Notifications schema using the following command: + +``` +mysql -u root -p notifications < /usr/share/icinga-notifications/schema/mysql/schema.sql +``` + +### Setting up a PostgreSQL Database + +Set up a PostgreSQL database for Icinga Notifications: + +``` +# su -l postgres + +createuser -P notifications +createdb -E UTF8 --locale en_US.UTF-8 -T template0 -O notifications notifications +echo 'CREATE EXTENSION IF NOT EXISTS citext;' | psql notifications +``` + +The `CREATE EXTENSION` command requires the `postgresql-contrib` package. + +Edit `pg_hba.conf`, insert the following before everything else: + +``` +local all notifications md5 +host all notifications 0.0.0.0/0 md5 +host all notifications ::/0 md5 +``` + +To apply these changes, run `systemctl reload postgresql`. + +After creating the database, import the Icinga Notifications schema using the following command: + +``` +psql -U icinga-notifications notifications < /usr/share/icinga-notifications/schema/pgsql/schema.sql +``` + +## Configuring Icinga Notifications + +Icinga Notifications installs its configuration file to `/etc/icinga-notifications/config.yml`, +pre-populating most of the settings for a local setup. Before running Icinga Notifications, +adjust the database credentials and the Icinga Web 2 URL. +The configuration file explains general settings. +All available settings can be found under [Configuration](03-Configuration.md). + +## Configuring Icinga 2 + +Icinga Notifications utilizes the [Icinga 2 API](https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/) +to fetch events from an Icinga 2 instance. +Thus, if the Icinga 2 API is not already configured, +please follow the [API setup guide](https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#setting-up-the-api) now. +Furthermore, an [`ApiUser`](https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#objecttype-apiuser) +with [permissions](https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#icinga2-api-permissions) +as listed in the example below is required. + +``` +object ApiUser "notifications" { + password = "insecure" + permissions = [ "events/*", "objects/query/*" ] +} +``` + +After completing setting up Icinga Notifications Web, the Icinga 2 API can be configured with this `ApiUser` as a source. + +## Running Icinga Notifications + +The `icinga-notifications` package automatically installs the necessary systemd unit files to run Icinga Notifications. +Please run the following command to enable and start its service: + +``` +systemctl enable --now icinga-notifications +``` + +## Installing Icinga Notifications Web + +With Icinga 2, Icinga Notifications and the database fully set up, it is now time to install Icinga Notifications Web, +which connects to the database and allows configuring Icinga Notifications. + +Please follow the +[Icinga Notifications Web documentation](https://icinga.com/docs/icinga-notifications-web/latest/doc/02-Installation/). + + diff --git a/doc/02-Installation.md.d/From-Source.md b/doc/02-Installation.md.d/From-Source.md new file mode 100644 index 000000000..16aecbeb6 --- /dev/null +++ b/doc/02-Installation.md.d/From-Source.md @@ -0,0 +1,10 @@ +# Installing Icinga Notifications from Source + +Although the provided packages are highly recommended and are the only officially supported installation, +it may be necessary to build the software manually, e.g., if there are no packages available for the target platform. + +Please follow the build instructions in the [development section](../30-Development.md) +for more information about requirements and the `Makefile`-based build. + + + diff --git a/doc/03-Configuration.md b/doc/03-Configuration.md new file mode 100644 index 000000000..3530c186a --- /dev/null +++ b/doc/03-Configuration.md @@ -0,0 +1,120 @@ +# Configuration + +The configuration for Icinga Notifications is twofold. +The main configuration resides in the database, +shared between the Icinga Notifications daemon and Icinga Notifications Web. +However, as the Icinga Notifications daemon needs to know how to access this database and some further settings, +it needs its own configuration file as well. + +This configuration is stored in `/etc/icinga-notifications/config.yml`. +See [config.example.yml](../config.example.yml) for an example configuration. + +## Top Level Configuration + +### HTTP API Configuration + +The HTTP API listener can be used both for submission and for debugging purposes. + +| Option | Description | +|----------------|----------------------------------------------------------------------| +| listen | Address to bind to, port included. (Example: `localhost:5680`) | +| debug-password | Password expected via HTTP Basic Authentication for debug endpoints. | + +### Icinga Web 2 + +The `icingaweb2-url` is expected to point to the base directory of your Icinga Web 2 installation, +i.e., `https://example.com/icingaweb2/`, to be used for URL creation. + +### Channel Plugin Directory + +All available Icinga Notification Channel Plugins should reside in the `channel-plugin-dir`. +For a package installation, the default will point to the correct location. + +This directory should be `/usr/libexec/icinga-notifications/channels` on systems that follow the Filesystem Hierarchy Standard. +It may also be `/usr/lib/icinga-notifications/channels`, depending on the operating system conventions. + +## Database Configuration + +Connection configuration for the database to which Icinga Notifications synchronizes monitoring data. +This is also the database used in Icinga Notifications Web to view and work with the data. + +| Option | Description | +|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| +| type | **Optional.** Either `mysql` (default) or `pgsql`. | +| host | **Required.** Database host or absolute Unix socket path. | +| port | **Optional.** Database port. By default, the MySQL or PostgreSQL port, depending on the database type. | +| database | **Required.** Database name. | +| user | **Required.** Database username. | +| password | **Optional.** Database password. | +| tls | **Optional.** Whether to use TLS. | +| cert | **Optional.** Path to TLS client certificate. | +| key | **Optional.** Path to TLS private key. | +| ca | **Optional.** Path to TLS CA certificate. | +| insecure | **Optional.** Whether not to verify the peer. | +| options | **Optional.** List of low-level [database options](#database-options) that can be set to influence some Icinga Notifications internal default behaviours. | + +### Database Options + +Each of these configuration options are highly technical with thoroughly considered and tested default values that you +should only change when you exactly know what you are doing. You can use these options to influence the Icinga Notifications default +behaviour, how it interacts with databases, thus the defaults are usually sufficient for most users and do not need any +manual adjustments. + +!!! important + + Do not change the defaults if you do not have to! + +| Option | Description | +|--------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| +| max_connections | **Optional.** Maximum number of database connections Icinga Notifications is allowed to open in parallel if necessary. Defaults to `16`. | +| max_connections_per_table | **Optional.** Maximum number of queries Icinga Notifications is allowed to execute on a single table concurrently. Defaults to `8`. | +| max_placeholders_per_statement | **Optional.** Maximum number of placeholders Icinga Notifications is allowed to use for a single SQL statement. Defaults to `8192`. | +| max_rows_per_transaction | **Optional.** Maximum number of rows Icinga Notifications is allowed to `SELECT`,`DELETE`,`UPDATE` or `INSERT` in a single transaction. Defaults to `8192`. | +| wsrep_sync_wait | **Optional.** Enforce [Galera cluster](#galera-cluster) nodes to perform strict cluster-wide causality checks. Defaults to `7`. | + +## Logging Configuration + +Configuration of the logging component used by Icinga Notifications. + +| Option | Description | +|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| level | **Optional.** Specifies the default logging level. Can be set to `fatal`, `error`, `warn`, `info` or `debug`. Defaults to `info`. | +| output | **Optional.** Configures the logging output. Can be set to `console` (stderr) or `systemd-journald`. If not set, logs to systemd-journald when running under systemd, otherwise stderr. | +| interval | **Optional.** Interval for periodic logging defined as [duration string](#duration-string). Defaults to `"20s"`. | +| options | **Optional.** Map of component name to logging level in order to set a different logging level for each component instead of the default one. See [logging components](#logging-components) for details. | + +### Logging Components + +| Component | Description | +|-----------------|---------------------------------------------------------------------------| +| channel | Notification channels, their configuration and output. | +| database | Database connection status and queries. | +| icinga2 | Icinga 2 API communications, including the Event Stream. | +| incident | Incident management and changes. | +| listener | HTTP listener for event submission and debugging. | +| runtime-updates | Configuration changes through Icinga Notifications Web from the database. | + +## Appendix + +### Duration String + +A duration string is a sequence of decimal numbers and a unit suffix, such as `"20s"`. +Valid units are `"ms"`, `"s"`, `"m"` and `"h"`. + +### Galera Cluster + +Icinga Notifications expects a more consistent behaviour from its database than a +[Galera cluster](https://mariadb.com/kb/en/what-is-mariadb-galera-cluster/) provides by default. To accommodate this, +Icinga Notifications sets the [wsrep_sync_wait](https://mariadb.com/kb/en/galera-cluster-system-variables/#wsrep_sync_wait) system +variable for all its database connections. Consequently, strict cluster-wide causality checks are enforced before +executing specific SQL queries, which are determined by the value set in the `wsrep_sync_wait` system variable. +By default, Icinga Notifications sets this to `7`, which includes `READ, UPDATE, DELETE, INSERT, REPLACE` query types and is +usually sufficient. Unfortunately, this also has the downside that every single Icinga Notifications query will be blocked until +the cluster nodes resynchronise their states after each executed query, and may result in degraded performance. + +However, this does not necessarily have to be the case if, for instance, Icinga Notifications is only allowed to connect to a +single cluster node at a time. This is the case when a load balancer does not randomly route connections to all the +nodes evenly, but always to the same node until it fails, or if your database cluster nodes have a virtual IP address +fail over assigned. In such situations, you can set the `wsrep_sync_wait` system variable to `0` in the +`/etc/icinga-notifications/config.yml` file to disable it entirely, as Icinga Notifications doesn't have to wait for cluster +synchronisation then. diff --git a/doc/04-Upgrading.md b/doc/04-Upgrading.md new file mode 100644 index 000000000..7df38a670 --- /dev/null +++ b/doc/04-Upgrading.md @@ -0,0 +1,4 @@ +# Upgrading Icinga Notifications + +Specific version upgrades are described below. Please note that version upgrades are incremental. +If you are upgrading across multiple versions, make sure to follow the steps for each of them. diff --git a/doc/05-Distributed-Setups.md b/doc/05-Distributed-Setups.md new file mode 100644 index 000000000..4afca60d9 --- /dev/null +++ b/doc/05-Distributed-Setups.md @@ -0,0 +1,3 @@ +# Distributed Setups + +To be done. \ No newline at end of file diff --git a/doc/channel-plugin.md b/doc/10-Channels.md similarity index 71% rename from doc/channel-plugin.md rename to doc/10-Channels.md index a40dec106..319a45daf 100644 --- a/doc/channel-plugin.md +++ b/doc/10-Channels.md @@ -1,4 +1,21 @@ -## Channel Plugin +# Channels + +After Icinga Notifications decides to send a notification of any kind, it will be passed to a channel plugin. +Such a channel plugin submits the notification event to a channel, e.g., email or a chat client. + +Icinga Notifications comes packed with channel plugins, but also enables you to develop your own plugins. + +To make those plugins available to Icinga Notifications, they must be placed in the [plugin channel directory](03-Configuration.md#channel-plugin-directory), +which is being done automatically for package installations. +Afterwards they can be configured through Icinga Notifications Web. + +## Available Channels + +- _email_: Email submission via SMTP +- _rocketchat_: Rocket.Chat +- _webhook_: Configurable HTTP/HTTPS queries for your backend + +## Technical Channel Description Channel plugins are processes that run continuously and independently of each other. They receive many requests over their lifetime. They receive JSON-formatted requests on stdin and reply with JSON-formatted responses on stdout. The @@ -73,4 +90,4 @@ Currently, the channel plugin include following three methods: [(example)](../internal/channel/examples/get-info.json) - `SendNotification`: Send the notifications. The `params` key should contain the information about the contact to be notified, corresponding object, the incident and the triggered event. - [(example)](../internal/channel/examples/send-notification.json) \ No newline at end of file + [(example)](../internal/channel/examples/send-notification.json) diff --git a/doc/20-HTTP-API.md b/doc/20-HTTP-API.md new file mode 100644 index 000000000..1eac405e8 --- /dev/null +++ b/doc/20-HTTP-API.md @@ -0,0 +1,47 @@ +# HTTP API + +Icinga Notifications comes with its own HTTP API, [configurable](03-Configuration.md#http-api-configuration) +via `listen` and `debug-password`. + +## Process Event + +One possible source next to the Icinga 2 API is event submission to the Icinga Notification HTTP API listener. +After creating a source with type _Other_ in Icinga Notifications Web, +the specified credentials can be used for HTTP Basic Authentication of a JSON-encoded +[Event](https://github.com/Icinga/icinga-notifications/blob/main/internal/event/event.go). + +``` +curl -v \ + -u 'source-1:correct horse battery staple' \ + -d '{"extra_tags":{"hostgroup/app-container":null ,"hostgroup/department-dev":null,"hostgroup/env-qa":null,"hostgroup/location-rome":null,"servicegroup/app-mail":null,"servicegroup/department-nms":null,"servicegroup/env-prod":null,"servicegroup/location-berlin":null},"message":"You will be honored for contributing your time and skill to a worthy cause.","name":"dummy-809: random fortune","severity":"ok","source_id":2,"tags":{"host":"dummy-809","service":"random fortune"},"url":"http://localhost/icingaweb2/icingadb/service?name=random%20fortune&host.name=dummy-809","username":""}' \ + 'http://localhost:5680/process-event' +``` + +## Debugging Endpoints + +There are multiple endpoints for dumping specific configurations. +To use those, the `debug-password` must be set and supplied via HTTP Basic Authentication next to an arbitrary username. + +### Dump Config + +The database-stored configuration from Icinga Notifications current viewpoint can be dumped as JSON. + +``` +curl -v -u ':debug-password' 'http://localhost:5680/dump-config' +``` + +### Dump Incidents + +The current incidents can be dumped as JSON. + +``` +curl -v -u ':debug-password' 'http://localhost:5680/dump-incidents' +``` + +### Dump Schedules + +All schedules with their assignee can be dumped in a human-readable form. + +``` +curl -v -u ':debug-password' 'http://localhost:5680/dump-schedules' +``` diff --git a/doc/30-Development.md b/doc/30-Development.md new file mode 100644 index 000000000..66ee7d41e --- /dev/null +++ b/doc/30-Development.md @@ -0,0 +1,41 @@ +# Developing Icinga Notifications + +Building the Icinga Notifications daemon requires a recent version of [Go](https://go.dev/dl/) and +[GNU Make](https://www.gnu.org/software/make/) to be installed. +The required Go version depends on the specified version in the `go.mod` file after the `go` directive. + +To fetch the source code, +either an archive can be downloaded from [GitHub release](https://github.com/Icinga/icinga-notifications/releases) +or the repository can be cloned with git. +The latter is highly recommended. + +``` +git clone https://github.com/Icinga/icinga-notifications.git +cd icinga-notifications +``` + +To specify destinations for the final install, +consult the variables on top of the `Makefile` which can be overwritten by environment variables. + +In particular, the `prefix` variable should be noted, +as it can be overwritten to point to a directory in the current working directory for testing purposes. +If set to, say `./out`, all build artifacts will end up in that directory. + +``` +export prefix="./out" +``` + +Now the daemon and all its channels can be built. + +``` +make +``` + +Finally, the `Makefile` also allows to perform an installation on the system. +Depending on the installation target, this command might need extended permissions, +so it might need to be executed by the `root` user, through `sudo` or a similar mechanism. +If, however, the `prefix` was set to a directory writeable by the current user, evaluating privileges is not necessary. + +``` +make install +```