Monitoring#

The Monitoring Role is composed by the Prometheus and Grafana software. Prometheus extracts data from metrics and log files, that are then used by Grafana, which provides dashboards to monitor the Carbonio status.

Install Packages#

Warning

In case you install this Role on a Node which already features the Mesh and Directory Role, make sure you do not install the service-discover-agent package.

# apt install carbonio-prometheus service-discover-agent
# dnf install carbonio-prometheus service-discover-agent

Bootstrap Carbonio#

Execute:

# carbonio-bootstrap

The bootstrap command will execute a number of tasks and will set up the node. At the end, you will be prompted with a menu and, if you already configured all, you only need to click y for confirmation.

On RHEL 9 only, restart the node to ensure that Carbonio service management come under the control of systemd:

# reboot

During the process, you need to provide these values, which you can retrieve from the first Mesh and Directory node.

  • Ldap master host is the FQDN of the first Mesh and Directory node, (example: ldap-mstr.example.com)

  • Ldap Admin password is obtained from the first Mesh and Directory node (ldap-admin-password)

Join Carbonio Mesh#

Carbonio Mesh will install as an agent, that connects and communicates with the server. The configuration for the agent is created by launching command

# service-discover setup-wizard

This command will:

  • ask for the IP address and netmask of the current Node

  • ask for the Carbonio Mesh secret, which is stored in file /var/lib/service-discover/password on the Directory Leader node.

After the configuration has completed successfully, run the following command, again using the secret.

# pending-setups -a

To complete the installation, restart prometheus service with:

# systemctl restart carbonio-prometheus

Grafana Installation and Configuration#

As mentioned above, Grafana requires some manual interaction before and after the installation to install also some necessary plugin and to configure the data sources and import them from Carbonio.

Grafana Installation#

Before installing Grafana, download the Grafana GPG key, add it to your APT installation’s list of trusted keys and put the Grafana repository to your APT sources.

# wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
# add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
# wget -q -O - https://packages.grafana.com/gpg.key | \
gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > \
/dev/null

# echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] \
https://packages.grafana.com/oss/deb stable main" | sudo \
tee -a /etc/apt/sources.list.d/grafana.list
# wget -q -O gpg.key https://rpm.grafana.com/gpg.key
# rpm --import gpg.key

Create file /etc/yum.repos.d/grafana.repo with the following content:

[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Install Grafana packages.

# apt update
# apt install grafana
# apt update
# apt install grafana
# dnf install grafana

Start Grafana service and check Grafana status. If everything is work correctly, you can enable the service to automatically start Grafana on boot.

# systemctl start grafana-server
# systemctl status grafana-server
# systemctl enable grafana-server

Log in to Grafana#

The Grafana portal can be reached on port 3000 of the node on which you installed the Monitoring Role, we assume it is called http://srvmon.example.com:3000/.

You can log in for the first time with these credentials: user = admin / password = admin

Hint

After the first login don’t forget to change the password for the account.

Grafana Plugins Installation#

Some dasboards require a plugin to work, so we install them:

  1. Worldmap panel

    # grafana-cli plugins install grafana-worldmap-panel
    
  2. Treemap panel

    # grafana-cli plugins install marcusolsson-treemap-panel
    
  3. Restart Grafana to use the new plugins:

    # systemctl restart grafana-server
    

Grafana datasource config#

  1. Open the Grafana web interface at http://srvmon.example.com:3000/

  2. Log in to Grafana using your credentials

  3. In the left sidebar, click Connections, then select Data Sources

  4. Click Add new data source

  5. Choose Prometheus as the type. In the list of available data sources, find and select “Prometheus” and and enter a name

    Hint

    To prevent conflicts in the future, use a meningful name, for example carbonio-prometheus.

  6. Configure Prometheus connection. Fill in the required fields for Prometheus, including the URL for your Prometheus server (e.g., http://localhost:9090).

  7. Click the “Save & Test” button to verify that the Prometheus connection was configured correctly. If the connection is successful, save the data source.

Import Dashboards#

A dashboard in Grafana is a visual representation of data designed to provide a comprehensive and real-time overview of various metrics and information. Dashboards are customizable and can contain panels that display graphs, tables, heatmaps, and other visualizations based on the data queried from connected data sources. The Zextras Team has prepares some specific dashboards necessary for comfortable monitoring, that can be easily imported from the Grafana marketplace as follows.

  1. Access the Grafana web interface

  2. In the left sidebar, select “Dashboards”

  3. Click “New” → “Import“ to open the import page.

  4. Paste the dashboard ID you want to import.

  5. Click “Load” to have the system retrieve information about the dashboard.

  6. If necessary, configure import parameters such as selecting a data source and other options.

  7. Click “Import” to complete the import process.

  8. After a successful import, go to the “Dashboards” section and select the imported dashboard to view it.

These steps will allow you to quickly import a dashboard into Grafana using its unique ID.

List of Grafana dashboards and IDs

Dashboard

Description

ID

Node Exporter Full

Standard dashboard

1860

Carbonio MTA queues

Information about MTA queues: statuses, size

20029

Carbonio Mailbox

Information about the mailbox like load, protocols, counters and db

20027

NGINX exporter

Standard dashboard from Grafana

12708

Carbonio Service Discover status

Contains data about infrastructure, servers and services in the consul, allows you to monitor the health of servers and services.

20032

Carbonio PostgreSQL

Based on Grafana “PostgreSQL Database” dashboard. Fixed dashboard

20030

Carbonio Processes

Contains a memory and cpu map panel which shows latest resident memory and cpu map , it’s helpful to check top processes that consume a lot of memory and CPU immediately, also includes the information from node exporters and about processes

20034

Carbonio MariaDB

MariaDB Overview Innodb, I/O, Waits, Connections and so on

20061