Monitoring Proxmox with InfluxDB & Grafana

Proxmox

Monitoring

Here I will show how I setup Grafana for visualization, with Promtail for log shipping, Loki for log aggregation, and InfluxDB for metrics. I use this to monitor my Proxmox cluster, pfSense router, and even my desktop pc. This blog will go over how I set it up to monitor my Proxmox cluster.

Screenshots

Previous slide

Next slide

Grafana LXC Configuration

Installing Ubuntu Container

Server Configurations

				
					sudo apt-get update && sudo apt upgrade -y

apt-get update – updates the package lists for upgrades and new package installations.
apt upgrade – will upgrade all currently installed packages.
-y – will automatically answer ‘yes’ to any prompts.

				
					shutdown -r now

				
					sudo adduser nesto

				
					sudo usermod -aG sudo nesto

				
					sudo apt install openssh-server

				
					sudo systemctl start ssh

Installing Docker

				
					curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

-fsSL – Is a combination of flags:
-f – tell ‘curl’ to not output anything if the HTTP status code indicates an error..
-s – tells ‘curl’ to hide the progress meter and error messages.
-S – tells ‘curl’ to show error messages if it fails.
-L – tells ‘curl’ to handle redirects.

				
					sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

deb – indicates that the repository is for binary packages.
[arch=amd64] – means the repository is for 64-bit software.
$(lsb_release -cs) – returns the codename for the Ubuntu server we installed and currently running commands on
stable tells the repository to use the latest stable version for Docker.

				
					sudo apt-get update

				
					sudo apt-get install docker-ce -y

				
					sudo docker run hello-world

				
					sudo systemctl start docker

				
					sudo systemctl enable docker

				
					sudo usermod -aG docker ${USER}

Installing Docker-Compose

				
					sudo curl -L "https://github.com/docker/compose/releases/download/v2.7.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

-L – Tells ‘curl’ to follow any redirects.
$(uname -s) and $(uname -m) – are used to automatically fill in the system’s kernel name and machine hardware name (e.g. ‘Linux’ and ‘x86_64).

				
					sudo chmod +x /usr/local/bin/docker-compose

				
					docker-compose --version

Grafana, Loki, Promtail Installation

Configuring YAML Files

				
					mkdir grafana

				
					mkdir loki

				
					mkdir promtail

				
					nano docker-compose.yml

				
					version: "3"
services:
  loki:
    image: grafana/loki:2.9.4
    volumes:
      - /home/nesto/loki:/etc/loki
    ports:
      - "3100:3100"
    restart: unless-stopped
    command: -config.file=/etc/loki/loki-config.yml
  promtail:
    image: grafana/promtail:2.9.4
    volumes:
      - /var/log:/var/log
      - /home/nesto/promtail:/etc/promtail
    restart: unless-stopped
    command: -config.file=/etc/promtail/promtail-config.yml
  grafana:
    image: grafana/grafana:latest
    user: "1000"
    volumes:
    - /home/nesto/grafana:/var/lib/grafana
    ports:
      - "3000:3000"
    restart: unless-stopped

ports: 3100:3100 – Maps port 3100 on the host to port 3100 on the container, you can change this if you’re already using port 3100 elsewhere.

I also changed the image from loki:latest, and promtail:latest because the containers kept getting stuck in a restart loop. After quite of bit of troubleshooting I found specifying the 2.9.4 version fixed it.

				
					nano loki/loki-config.yml

				
					auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

				
					nano promtail/promtail-config.yml

				
					server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

# local machine logs

- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

				
					docker-compose up -d

				
					sudo docker ps

Accessing Grafana

InfluxDB LXC Configuration

Server Configuration

Configuring YAML File

				
					mkdir influxdb

				
					nano /influxdb/docker-compose.yml

				
					version: "3"
services:
 influxdb:
   image: influxdb:latest
   container_name:  influxdb
   networks:
     - default
   security_opt:
     - no-new-privileges:true
   restart: unless-stopped
   ports:
     - "8086:8086"
   volumes:
     - $DOCKERDIR/appdata/influxdb2/config:/etc/influxdb2
     - $DOCKERDIR/appdata/influxdb2/db:/var/lib/influxdb2