serverhabit.com

Practical server tutorials and cost-conscious self-hosting for indie hackers

Monitoring Server Health with Prometheus on Ubuntu 24.04

by David Park
Monitoring Server Health with Prometheus on Ubuntu 24.04

Monitoring server health with Prometheus gives you a time-series metrics pipeline that scrapes CPU, memory, disk I/O, and network stats from every machine you own. By the end of this guide you'll have Prometheus 2.52 running as a systemd service, Node Exporter 1.8 reporting host metrics, and a working HTTP endpoint you can query or wire into Grafana. You'll run everything on a single Ubuntu 24.04 VPS — the same setup I use across my Hetzner fleet for about $0 in extra tooling cost.

Prerequisites

  • Ubuntu 24.04 LTS server (fresh install or existing)
  • A non-root user with sudo privileges
  • Ports 9090 (Prometheus) and 9100 (Node Exporter) reachable, or a firewall rule you control
  • At least 512 MB RAM and 2 GB free disk space
  • curl, tar, and systemd available (all present by default on Ubuntu 24.04)

Step 1 — Create Dedicated System Users

Running Prometheus and Node Exporter as unprivileged system users limits blast radius if either process is compromised.

1.1 Create the prometheus user:

sudo useradd --system --no-create-home --shell /bin/false prometheus

1.2 Create the node_exporter user:

sudo useradd --system --no-create-home --shell /bin/false node_exporter

Neither account gets a login shell or a home directory.

Step 2 — Install Node Exporter 1.8

Node Exporter exposes host-level metrics — CPU, memory, filesystem, network — on port 9100.

2.1 Download the Node Exporter binary:

curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter-1.8.1.linux-amd64.tar.gz

2.2 Verify the SHA256 checksum (compare against the value on the GitHub releases page):

sha256sum node_exporter-1.8.1.linux-amd64.tar.gz

2.3 Extract and install the binary:

tar xzf node_exporter-1.8.1.linux-amd64.tar.gz
sudo mv node_exporter-1.8.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

2.4 Create the systemd unit file at /etc/systemd/system/node_exporter.service:

sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<'EOF'
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

2.5 Enable and start Node Exporter:

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Expected output:

Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter.service ...

Step 3 — Install Prometheus 2.52

Prometheus scrapes Node Exporter every 15 seconds and stores the time-series data locally in /var/lib/prometheus.

3.1 Download Prometheus:

curl -LO https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz

3.2 Verify the checksum:

sha256sum prometheus-2.52.0.linux-amd64.tar.gz

3.3 Extract and install the binaries:

tar xzf prometheus-2.52.0.linux-amd64.tar.gz
sudo mv prometheus-2.52.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.52.0.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

3.4 Create the directories Prometheus needs:

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

3.5 Copy the default console templates and libraries (optional but useful for the built-in UI):

sudo cp -r prometheus-2.52.0.linux-amd64/consoles /etc/prometheus/
sudo cp -r prometheus-2.52.0.linux-amd64/console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus/consoles /etc/prometheus/console_libraries

Step 4 — Configure Prometheus

Write a minimal prometheus.yml that scrapes itself and Node Exporter.

4.1 Create /etc/prometheus/prometheus.yml:

sudo tee /etc/prometheus/prometheus.yml > /dev/null <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100"]
EOF

4.2 Set ownership:

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

4.3 Validate the configuration with promtool:

sudo -u prometheus promtool check config /etc/prometheus/prometheus.yml

Expected output:

Checking /etc/prometheus/prometheus.yml
  SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

If promtool reports an error, re-check indentation — YAML is whitespace-sensitive.

Step 5 — Create the Prometheus systemd Service

5.1 Write the unit file at /etc/systemd/system/prometheus.service:

sudo tee /etc/systemd/system/prometheus.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus Monitoring
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

The --storage.tsdb.retention.time=15d flag keeps 15 days of metrics. On a $4 Hetzner CAX11 with a 40 GB disk, Node Exporter data for one host stays well under 1 GB at this retention window.

5.2 Enable and start Prometheus:

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

Step 6 — Open Firewall Ports (UFW)

If you're running UFW, allow access to the Prometheus UI. Restrict port 9100 to localhost — Node Exporter should not be publicly reachable.

6.1 Allow Prometheus UI from your IP only (replace YOUR_IP):

sudo ufw allow from YOUR_IP to any port 9090 proto tcp

6.2 Confirm Node Exporter is bound only to localhost (it is by default). Verify:

ss -tlnp | grep 9100

Expected output:

LISTEN  0  4096  127.0.0.1:9100  0.0.0.0:*  users:(("node_exporter",...

If it shows 0.0.0.0:9100, add --web.listen-address=127.0.0.1:9100 to the Node Exporter ExecStart line and restart the service.

Verify It Works

Run these checks in order. All three must pass before you consider the setup complete.

Check 1 — Both services are active:

systemctl is-active prometheus node_exporter

Expected output:

active
active

Check 2 — Node Exporter is serving metrics:

curl -s http://localhost:9100/metrics | head -5

Expected output (first lines will vary but must start with #):

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.67e-05
...

Check 3 — Prometheus is scraping targets successfully:

curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | grep '"health"'

Expected output:

            "health": "up",
            "health": "up",

Both targets (prometheus and node_exporter) must show "up". If either shows "down", see Troubleshooting below.

Check 4 — Query a real metric:

curl -sg 'http://localhost:9090/api/v1/query?query=node_cpu_seconds_total' | python3 -m json.tool | grep resultType

Expected output:

    "resultType": "vector",

A vector result confirms Prometheus is storing and returning Node Exporter data.

Troubleshooting

Service fails to start — check the journal:

journalctl -u prometheus -n 50 --no-pager
journalctl -u node_exporter -n 50 --no-pager

Common causes: wrong file ownership on /var/lib/prometheus or a YAML syntax error in prometheus.yml.

Target shows "down" in /api/v1/targets:

Confirm the exporter port is open:

ss -tlnp | grep 9100

If nothing appears, Node Exporter is not running. Restart it:

sudo systemctl restart node_exporter

promtool check config fails with a YAML error:

Re-paste the prometheus.yml block exactly. YAML does not allow tabs — use two spaces for indentation. Verify with:

cat -A /etc/prometheus/prometheus.yml | grep -P '\t'

No output means no tabs are present.

Prometheus UI unreachable from browser:

Confirm the process is listening:

ss -tlnp | grep 9090

Then check UFW rules:

sudo ufw status numbered

Add a rule for your IP if it's missing.

High disk usage from TSDB:

Reduce retention or limit storage size. Edit the ExecStart line in /etc/systemd/system/prometheus.service and add:

--storage.tsdb.retention.size=800MB

Then reload:

sudo systemctl daemon-reload && sudo systemctl restart prometheus

Next Steps

You now have a working monitoring server health with Prometheus pipeline on Ubuntu 24.04. From here:

  • Add Grafana — point a Grafana data source at http://localhost:9090 and import dashboard ID 1860 (Node Exporter Full) for pre-built panels.
  • Add more hosts — install Node Exporter on each additional server, then add their IPs to the node_exporter job's targets list in prometheus.yml.
  • Set up Alertmanager — define alerting rules in /etc/prometheus/rules/ and route notifications to Slack or PagerDuty when CPU stays above 90% for 5 minutes.
  • Secure the endpoint — put Prometheus behind an Nginx reverse proxy with HTTP basic auth if you need to expose the UI over the internet rather than restricting by IP.

The scrape interval of 15 seconds and 15-day retention are conservative defaults that work well for a single-node setup. Tune both once you know your actual query patterns.