Working with Prometheus and Grafana — An Introduction
When we deploy software or application(s) in different environments, we want to monitor the environments, the performance of the deployed software or application(s), and other parameters. Monitoring is an important component of any technical field. There are many methods of monitoring like logs, metrics, profiling, etc. Each component of a given infrastructure has its specialized tool for monitoring, but there are very limited tools which can monitor multiple technologies and devices. One such tool is Prometheus.
Prometheus is an open-source monitoring platform primarily used for monitoring metrics. It collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. It stores data as a time-series database, which can be accessed using a query language called PromQL. Using the metrics collected, Prometheus allows us to observe various parameters both in numerical and graphical format.
Prometheus is often used in combination with Grafana, which is a multi-platform open-source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics, by using metrics stored as a time-series database in different graphical representations.
Typically, we would like to not only analyze the data in the form of metrics but also make the data collected persistent, which can be done with the help of container orchestration tools like Kubernetes. In this article, we will be looking at how we can integrate Prometheus and Grafana for metrics monitoring while making the metrics collected persistent with the help of Kubernetes.
To set up our work, we need a few pre-requisites:
- Installation of Minikube and Kubectl on the base OS.
- Two computer systems or one computer system with a Virtual Machine (Any OS can be used; I will be using RHEL8).
Creating a Docker container image for Prometheus and Grafana
Prometheus and Grafana can be downloaded directly from their official websites onto the system where we want to perform the monitoring. But it is always better to run these tools on a container themselves as it allows us to replicate the monitoring tools and their configurations in a very short time on multiple computer systems.
We first begin with the creation of the Dockerfile for Prometheus and Grafana using which we can build custom Docker container images. These images can then be used with Kubernetes to launch pods or deployments to set up the monitoring tools we need.
I have explained how to create a custom Docker container images in a previous article, and we follow the same procedure here. The Dockerfile for Prometheus uses CentOS as the base OS for launching Prometheus. Once the download is completed and the tar file is unzipped, we create a separate file to store the metrics we collect named prom_metrics. We finally launch Prometheus by defining the custom configuration file and the path for storing the metrics. We also expose the default port for Prometheus (Port 9090).
# Contents of Prometheus DockerfileFROM centos:latestRUN yum install wget sudo -y
RUN wget https://github.com/prometheus/prometheus/releases/download/v2.19.0/prometheus-2.19.0.linux-amd64.tar.gz
RUN tar -xzf prometheus-2.19.0.linux-amd64.tar.gz
RUN mkdir /prom_metrics
CMD ./prometheus-2.19.0.linux-amd64/prometheus --config.file=/prometheus-2.19.0.linux-amd64/prometheus.yml --storage.tsdb.path=/prom_metrics && tail -f /dev/null
EXPOSE 9090
Once the image is created, we build the image and push it into Docker Hub for later use.
docker built -t prometheus-os:v1 .
docker login
docker tag prometheus-os:v1 akshayab99/prometheus-os:v1
docker push akshayab99/prometheus-os:v1We then create the Dockerfile for launching Grafana. We follow a similar procedure as we did for Prometheus, but we also set the working directory as /usr/share/grafana which means that all the commands from this point onwards will be executed in this directory. We also expose the default port for Grafana (Port 3000).
# Contents of Grafana DockerfileFROM centos:latestRUN yum install wget -y
RUN wget https://dl.grafana.com/oss/release/grafana-7.0.3-1.x86_64.rpm
RUN yum install grafana-7.0.3-1.x86_64.rpm -yWORKDIR /usr/share/grafanaCMD /usr/sbin/grafana-server start && /usr/sbin/grafana-server enable && /bin/bash
EXPOSE 3000
After the Dockerfile is created, we build the image and push it to Docker Hub.
docker built -t grafana-os:v1 .
docker login
docker tag grafana-os:v1 akshayab99/grafana-os:v1
docker push akshayab99/grafana-os:v1Launching Prometheus
Once the custom images have been created, we write the YAML files for launching the deployments and services for Prometheus and Grafana. To make the data permanent, we also write a PersistentVolumeClaim (PVC) for Prometheus and Grafana.
Prometheus has the additional requirement of ConfigMap. A ConfigMap is used to store non-confidential environment variables as key-value pairs. These ConfigMaps can be mounted as volumes when launching pods or deployments. In our case, the ConfigMap will contain information on all the targets from which Prometheus will scrape metrics. Typically, the configuration file of Prometheus (named prometheus.yml) contains the information for scraping from Prometheus itself, but we need to add the other nodes or applications from which we want to scrape metrics.
The PVC and ConfigMap are written as shown below. In the ConfigMap, we add a node whose metrics we want to monitor. Prometheus itself cannot scrap metrics; rather we need the help of exporters. To monitor a node, we need a Node Exporter to run in the node when we begin scraping metrics. The Node Exporter communicates using the Port 9100.
#Contents of prom-cm-pvc.ymlapiVersion: v1
kind: ConfigMap
metadata:
name: prom-configmap
labels:
app: prom-configmap
data:
prometheus.yml: |
global:
scrape_interval: 20s
evaluation_interval: 20s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'rhelnode1'
static_configs:
- targets: ['192.168.1.26:9100'] #Give the IP Address of the node from which you are scraping metrics
---apiVersion: v1
kind: PersistentVolumeClaimmetadata:
name: prom-pvc
labels:
name: prom-pvcspec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Next, we write the YAML file for launching the deployment and the service for exposing the deployment.
#Contents of prometheus-dp.ymlapiVersion: apps/v1
kind: Deployment
metadata:
name: prom-dp
labels:
app: prom
spec:
replicas: 1
selector:
matchLabels:
app: prom
strategy:
type: Recreate
template:
metadata:
labels:
app: prom
spec:
containers:
- image: akshayab99/prometheus-os:v1
name: prom
ports:
- containerPort: 9090
name: prom
volumeMounts:
- name: prom-vol
mountPath: /prom_metrics
- name: prom-configmap
mountPath: /prometheus-2.19.0.linux-amd64/prometheus.yml
subPath: prometheus.yml
volumes:
- name: prom-configmap
configMap:
name: prom-configmap
- name: prom-vol
persistentVolumeClaim:
claimName: prom-pvc---apiVersion: v1
kind: Service
metadata:
name: prom-svc
labels:
app: prom
spec:
ports:
- port: 9090
targetPort: 9090
nodePort: 30303
selector:
app: prom
type: NodePort
Once the files have been written, we can then launch Prometheus using the command line using kubectl and minikube.
> kubectl create -f prom-cm-pvc.yml
> kubectl create -f prometheus-dp.ymlLaunching Grafana
To launch Grafana using our custom image, we need to write the YAML files for the PVC.
#Contents of grafana-pvc.ymlapiVersion: v1
kind: PersistentVolumeClaimmetadata:
name: grafana-pvc
labels:
name: grafana-pvcspec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Next, we write the YAML for launching Grafana using a deployment and exposing the deployment using a service.
#Contents of grafana-dp.ymlapiVersion: apps/v1
kind: Deployment
metadata:
name: grafana-dp
labels:
app: grafana
spec:
replicas: 1
selector:
matchLabels:
app: grafana
strategy:
type: Recreate
template:
metadata:
labels:
app: grafana
spec:
containers:
- image: akshayab99/grafana-os:v1
name: grafana
ports:
- containerPort: 3000
name: grafana
volumeMounts:
- name: grafana-pvc
mountPath: /usr/share/grafana/data
volumes:
- name: grafana-pvc
persistentVolumeClaim:
claimName: grafana-pvc---apiVersion: v1
kind: Service
metadata:
name: grafana-svc
labels:
app: grafana
spec:
ports:
- port: 3000
targetPort: 3000
nodePort: 30306
selector:
app: grafana
type: NodePort
Once the files have been written, we can then launch Prometheus using the command line using kubectl and minikube.
> kubectl create -f grafana-pvc.yml
> kubectl create -f grafana-dp.ymlOnce all the files have been launched successfully, we can see the following output.
Since all the deployments are launched on the Minikube cluster, we can access them using the Minikube IP and their respective Port numbers.
In the case of Grafana, we need to launch it and set up the admin account. We then add the source of metrics, which is Prometheus.
After integrating Prometheus and Grafana, we can now visualize metrics using pre-built dashboards or our custom versions.
