Monitoring Observability Setup
Introduction
My setup
Resources
| Component | CPU Request | CPU Limit | Memory Request | Memory Limit |
|---|---|---|---|---|
| Prometheus | 500m | 1–2 cores | 512Mi | 2–4 Gi |
| Grafana | 100m | 500m | 128Mi | 512Mi–1 Gi |
| Loki | 300m | 1 core | 512Mi | 2 Gi |
| OTEL Collector | 200m | 500m | 256Mi | 512Mi–1 Gi |
Persistent Data
| Component | Needs Persistence? | What It Stores | Notes |
|---|---|---|---|
| Prometheus | Yes | Time-series metrics (TSDB) | Use a PVC for `/prometheus` or `/data`. Retention defaults to 15d; tune via `--storage.tsdb.retention.time`. |
| Grafana | Optional | Dashboards, users, config | If using SQLite (default), persist `/var/lib/grafana`. For Postgres/MySQL, persist the DB. |
| Loki | Yes | Log chunks, index, metadata | Persist `/loki` or use object storage (e.g., MinIO, S3). Index and chunk retention are configurable. |
| OTEL Collector | No | N/A | Stateless by design. Doesn’t store data unless you add a file exporter or buffering. |
Backups
| Component | Persistence Required? | Suggested Setup | Retention Strategy |
|---|---|---|---|
| Prometheus | Minimal | Use emptyDir or hostPath | Retention: ~6h to 24h. Set `--storage.tsdb.retention.time=6h` |
| Grafana | Optional | No volume needed unless storing user dashboards | Use provisioning (ConfigMaps) for dashboards |
| Loki | Yes-ish | Use emptyDir or hostPath | Set retention via config (`table_manager`, `index`, `chunks`) |
| OTEL Collector | No | Stateless | No action needed |
Migration
| Step | Action | Notes |
|---|---|---|
| 1 | Install MicroK8s | `sudo snap install microk8s --classic` — enable `dns`, `hostpath-storage`, `helm3` |
| 2 | Export kubeconfig | `microk8s config > ~/.kube/config` — allows kubectl/helm access |
| 3 | Prepare Helm charts | Use official charts for Prometheus, Grafana, Loki, OTEL Collector |
| 4 | Convert local configs | Translate local YAML/configs into Helm `values.yaml` or Kubernetes manifests |
| 5 | Deploy stack | `helm install` each component into `observability` namespace |
| 6 | Wire services | Use `Service`, `Ingress`, or `NodePort` to expose Grafana/Prometheus |
| 7 | Validate data flow | Confirm metrics/logs/traces are flowing via OTEL Collector |