Introduction

Infrastructure monitoring is the basis for application performance management. The underlying system’s availability and health must be maximised continually. To achieve this, one has to monitor the system metrics like CPU, memory, network, and disk. Response time lag, if any must be addressed swiftly. Here we’lltake a look at how to Monitor servers (and even Docker Containers running inside the Server) using Grafana, Prometheus, Node Exporter, CAdvisor and Skedler Reports.

Core Components

Grafana- Database analytics & monitoring solution

Prometheus- Event monitoring and alerting

Node-Exporter- Monitoring Linux host metrics

Wmi-Exporter- Monitoring Windows host metrics

CAdvisor- Monitoring metrics for the running Containers.

Skedler Reports –Automating actionable reports

Grafana - Database Analytics & monitoring solution 

Grafana equips users to query, visualize,  and monitor metrics, no matter where the underlying data is stored. With Grafana, one can also set alerts for metrics that require attention, apart from creating, exploring, and sharing dashboards with their team and fostering a data-driven culture.

Prometheus -Event monitoring and alerting

Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company.

Node Exporter - Monitoring Linux host metrics

Node Exporter is a Prometheus exporter for hardware and OS metrics with pluggable metric collectors. It allows measuring various machine resources such as memory, disk, and CPU utilization

WMI Exporter -Monitoring Windows host metrics

Prometheus exporter for Windows machines, using the WMI (Windows Management Instrumentation).

CAdvisor - Monitoring metrics for the running Containers.

It Stands for Container Advisor and is used to aggregate and process all the metrics for the running Containers.

Skedler Reports — Automating actionable reports

Skedler offers the most powerful, flexible and easy-to-use data monitoring solution that companies use to exceed customer SLAs, achieve compliance, and empower internal IT and business leaders.

Prerequisite,

1. A Linux machine
2. Docker Installed
3. Docker Compose Installed

So Let’s get started.

Login to your Linux machine and update the repository and Install Docker and Docker Compose. To Update the Repository,

Create a Directory, say monitoring

[email protected]:~$ mkdir monitoring

[email protected]:~$ cd monitoring/

[email protected]:~$ vim docker-compose.yml

Now, create a Docker Compose file for Prometheus, You also need to create a Prometheus configuration file, prometheus.yml Docker Compose file for Prometheus as below,

Note: We will keep on extending the same docker file as we move forward to install other components.

version: ‘3’

services:

 prometheus:

   image: prom/prometheus:latest

   container_name: prometheus

   volumes:

     – ./prometheus.yml:/etc/prometheus/prometheus.yml

     – ./prometheus_db:/var/lib/prometheus

     – ./prometheus_db:/prometheus

     – ./prometheus_db:/etc/prometheus

     – ./alert.rules:/etc/prometheus/alert.rules

   command:

     – ‘–config.file=/etc/prometheus/prometheus.yml’

     – ‘–web.route-prefix=/’

     – ‘–storage.tsdb.retention.time=200h’

     – ‘–web.enable-lifecycle’

   restart: unless-stopped

   ports:

     – ‘9090:9090’

   networks:

     – monitor-net

Create a Prometheus configuration file and paste the config as below.

global:

 scrape_interval: 5s

 external_labels:

   monitor: ‘guidanz-monitor’

scrape_configs:

 – job_name: ‘prometheus’

   static_configs:

     – targets: [‘monitoring.guidanz.com:9090’] ## IP Address of the localhost

The compose file consists of two volume mappings to the container. One is the Prometheus configuration and the other one (prometheus_db) is to store the Prometheus database locally. Now run the docker-compose.

[email protected]:~/monitoring$ mkdir prometheus_db

[email protected]:~/monitoring$ docker-compose up -d

Access Prometheus using the IP and Port and you will see the Prometheus UI, http://ip_address:9090

Now we will setup the Node Exporter. It is one of the best components used along with the Prometheus to capture metrics from the server where the Prometheus is running. It Captures all hardware and kernel-related metrics like CPU, Memory, Disk, Disk Read/Write, etc.

To Install the Node exporter, simply append the docker-compose.ymlfile and prometheous.yml file as below.

node-exporter:

 image: prom/node-exporter

 ports:

   – ‘9100:9100’

Append the prometheus.yml as below,

– job_name: ‘node-exporter’

 static_configs:

   – targets: [‘monitoring.guidanz.com:9100’]

So now the Composite docker-compose file will looks like below,

version: ‘3’

services:

 prometheus:

   image: prom/prometheus:latest

   volumes:

     – ./prometheus.yml:/etc/prometheus/prometheus.yml

     – ./prometheus_db:/var/lib/prometheus

   command:

     – ‘–config.file=/etc/prometheus/prometheus.yml’

   ports:

     – ‘9090:9090’

 node-exporter:

 image: prom/node-exporter

 ports:

   – ‘9100:9100’

And prometheus.yml will look like below,

global:

 scrape_interval: 5s

 external_labels:

   monitor: ‘guidanz-monitor’

scrape_configs:

 – job_name: ‘prometheus’

   static_configs:

     – targets: [‘monitoring.guidanz.com:9090’]

 – job_name: ‘node-exporter’

   static_configs:

     – targets: [‘monitoring.guidanz.com:9100’]

Now Create the Node Exporter Container and Restart the Prometheus container using the below commands:

[email protected]:~/monitoring$ docker-compose start node-exporter

[email protected]:~/monitoring$ docker-compose restart prometheus

Or, One can simply do compose up and down.

[email protected]:~/monitoring$ docker-compose down

[email protected]:~/monitoring$ docker-compose up -d

Take a look at the targets in Prometheus. You will notice node exporter as well as a target.

Now, setup CAdvisor, for this append the docker compose with below code.

cadvisor:

 image: google/cadvisor:latest

 ports:

   – ‘8080:8080’

 volumes:

   – /:/rootfs:ro

   – /var/run:/var/run:rw

   – /sys:/sys:ro

   – /var/lib/docker/:/var/lib/docker:ro

Also, append the prometheus.yml with a bit code yml code. We are actually adding the CAdvisor service in Prometheus configuration.

– job_name: ‘cAdvisor’

 static_configs:

   – targets: [‘monitoring.guidanz.com:8080’]

Access CAdvisor from the URL, http://IP_Address:8080/docker/

Now eventually we will set up the grafana, where we will be using Prometheus as a data source. We can have a better Dashboard in grafana for the metrics visualization.

Append the code in the above docker compose and restart.

grafana:

 image: grafana/grafana

 user: “1000”

 environment:

   – GF_SECURITY_ADMIN_PASSWORD=secure_pass

 volumes:

   – ./grafana_db:/var/lib/grafana

 depends_on:

   – prometheus

 ports:

   – ‘3000:3000’

Access grafana UI from 3000 port, default user will be admin and the password you set in the compose file.

Now eventually we will set up the Skedler Reports, where we will be using Grafana as a data source. Skedler offers a simple and easy to add reporting and alerting solution for Elastic Stack and Grafana. Please review the documentation to install Skedler Reports.

Now, Setup Skedler Reports, for this append the docker compose with the below code.

reports:

   image: skedler/reports:latest

   container_name: reports

   privileged: true

   cap_add:

     – SYS_ADMIN

   volumes:

     – /sys/fs/cgroup:/sys/fs/cgroup:ro

     – reportdata:/var/lib/skedler

     – ./reporting.yml:/opt/skedler/config/reporting.yml

   ports:

     – 3001:3001