Prometheus is a server monitoring service. It collects metrics (values) from configured target servers at certain preset intervals, evaluates certain rules and expressions, displays results, and can trigger alerts when certain conditions become true (excessive CPU usage, system run out of memory, run out of space on disk, etc.). Prometheus raises, however, monitoring on a much more complex level than Pinguzo and Netdata.
I must note that although I have used CentOS 7 for both the Prometheus server and the monitoring servers (nodes), the steps below can be followed for installing and configuring Prometheus on any Linux distribution with systemd.
In the tutorial I will not explain each command, because I assume that if you have come to read this article, you should already have some Linux knowledge.
Preparing the system
Prior to installing Prometheus, special care must be taken to set NTP sync – time is a sensitive value in metric analysis. Red Hat 7 (and its derivatives) no longer use the ntpd service, but chronyd – so you should install and configure the chrony package.
If you have the firewall enabled, you must open the following ports:
• 9090 for Prometheus
• 9100 for node_exporter
• 323 for chrony (this if you have not already done it yet)
Depending on the firewall, the commands will be run:
• for firewalld
# firewall-cmd –permanent –add-port=9090/tcp
# firewall-cmd –permanent –add-port=9100/tcp
# firewall-cmd –permanent –add-port=323/udp
# systemctl reload firewall
# firewall-cmd –list-all
• for iptables:
Edit the /etc/sysconfig/iptables file and restart iptables:
# vi /etc/sysconfig/iptables
Add the rules:
-A INPUT -p tcp -m states -state NEW -m tcp –port 9090 -j ACCEPT
-A INPUT -p tcp -m state -state NEW -m tcp –dport 9100 -j ACCEPT
# systemctl restart iptables
Installing the Prometheus server
We will install Prometheus in the directory /opt:
# cd /opt
Navigate to https://prometheus.io/download/ and download the latest version of Prometheus for Linux binaries:
# wget https://github.com/prometheus/prometheus/releases/download/v2.9.2/prometheus-2.9.2.linux-amd64.tar.gz
# tar -zxvf prometheus-2.9.2.linux-amd64.tar.gz
# ln -s prometheus-2.9.2.linux-amd64 prometheus
# useradd prometheus
# chown -R prometheus:prometheus /opt/prometheus-2.9.2.linux-amd64/
Create the Prometheus service
# vi /etc/systemd/system/prometheus.service
Inside the new file we will write the following:
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target
[Service]
User=root
Restart=on-failure
ExecStart=/opt/prometheus/prometheus –config.file=/opt/prometheus/prometheus.yml
[Install]
WantedBy=multi-user.target
Running the Prometheus server
Running commands:
# ln -s /opt/prometheus/prometheus.yml /etc
# systemctl daemon-reload
# systemctl enable –now prometheus
# systemctl status prometheus
View metrics
In the browser, go to http://IP_server_prometheus:9090. Click Graph; enter the expression scrape_duration_seconds and click Execute.
Installing Node Exporter
Node Exporter is installed on each server to be monitored (attention to the port 9100!). We will install Node Exporter in the /opt directory as well; we will download the latest version of Node Exporter (at the time of writing this article, the last stable Node Exporter version is 0.17.0):
# cd / opt
# wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz
# tar xvfz node_exporter -*.tar.gz
# rm node_exporter-0.17.0.linux-amd64.tar.gz
# ln -s node_exporter-0.17.0.linux-amd64 node_exporter
# chown -R prometheus:prometheus /opt/node_exporter-0.17.0.linux-amd64
Creating the Node Exporter service
We will create a new file in /etc/systemd/system:
# vi /etc/systemd/system/node_exporter.service
In this new file we will write the following (the –no-collector.disktats option has been added because these statistics do not work on virtualized systems – you are free to remove this if you want to):
[Unit]
Description=Node Exporter
[Service]
User=root
ExecStart=/opt/node_exporter/node_exporter –no-collector.diskstats
[Install]
WantedBy=default.target
Starting the service node_exporter:
# systemctl daemon-reload
# systemctl enable –now node_exporter
# systemctl status node_exporter
At this point, browsing the browser at http://IP_server: 9100, you’ll see a page that says Node Exporter and a link called Metrics that contains all metrics for the monitored system (node). Click on Metrics. Now we have to tell Prometheus to read these metrics. On the server where we installed Prometehus, we edit the /opt/prometheus/prometheus.yml file:
# vi /opt/prometheus/prometheus.yml
We add the following lines:
– job_name: ‘node’
static_configs:
– targets:[‘localhost:9100’]
This sequence is added for each server where Node Exporter is installed (localhost is replaced with the corresponding server IP); also job_name: ‘node’ turns into job_name: ‘node1’, etc. – or whatever name you want to name it. Or, the nodes/servers can be added one after the other, as in the example below:
– job_name:’nodes’
static_configs:
– targets:[‘IP_server_01:9100′,’IP_server_02:9100’]
This file is yaml type and is very sensitive to formatting! To verify if it is correctly written, use a validation tool (for example yaml validator).
After editing and saving this file, restart the prometheus service and check its status:
# systemctl restart prometheus
# systemctl status prometheus
The service must be active. If it is not active, the problem might be the configuration file (probably yaml is not properly formatted). To see exactly what’s happening, you can take a look at the /var/log/messages file:
# less /var/log/messages
Once you have added all the nodes where you have installed Node Exporter and want to track them through Prometheus, browse to http://IP_server_prometheus:9090/targets and you will be able to see the status of each server.