Grafana

Introduction

This is a quick page to note about setup of Grafana

Installing

I install using helm in microk8s. I use PersistentVolume and PersistentVolumeClaim. I originally had Grafana installed locally so had to override the security context. (could have deleted but choose not to).
The PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /var/lib/grafana

And the PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  namespace: bibble-grafana
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: manual

Amend the Service to a Node Port

apiVersion: v1
kind: Service
metadata:
  name: grafana-nodeport
  namespace: bibble-grafana
spec:
  type: NodePort
  selector:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/name: grafana
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30930

What the Amended pod would look like - we do not run this

volumeMounts:
  - name: grafana-storage
    mountPath: /var/lib/grafana
volumes:
  - name: grafana-storage
    persistentVolumeClaim:
      claimName: grafana-pvc

Installing with helm

helm install grafana grafana/grafana --namespace bibble-grafana \
  --set securityContext.runAsUser=120 \
  --set securityContext.runAsGroup=122 \
  --set securityContext.fsGroup=122
  --set persistence.enabled=true \
  --set persistence.existingClaim=grafana-pvc

CA Cert

The CA Cert is configured against the datasource. You cut and paste the rootCA.pem. Otherwise you get the error

client: failed to call resources: error querying resource: Post \"https://192.168.1.220:9090/api/v1/labels\": tls: failed to verify certificate: x509: certificate signed by unknown authority

Data Frames

Gosh it is all lingo. This is what grafana refers to for it internal layout.

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "caddy_requests_by_5min",
          "bucket": "1752712500",
          "instance": "192.168.1.220:8080",
          "job": "bibble-caddy-exporter",
          "site": "wiki.bibble.co.nz"
        },
        "values": [
          [
            1752719700,
            "123"
          ]
        ]
      },
      {
        "metric": {
          "__name__": "caddy_requests_by_5min",
          "bucket": "1752712800",
          "instance": "192.168.1.220:8080",
          "job": "bibble-caddy-exporter",
          "site": "wiki.bibble.co.nz"
        },
        "values": [
          [
            1752719700,
            "118"
          ]
        ]
      },
    ]
  }
}

View a data frame this would look like this. Note it is the prometheus generated timestamp in the value which makes these data frames. Here they are not unique because the scrape time was the same.

Time (Scrape TS)	Value	Bucket Label	Site	Instance	Job	Frame Explanation
1752719700	123	1752712500	wiki.bibble.co.nz	192.168.1.220:8080	bibble-caddy-exporter	Not Unique as Scrap time the same
1752719700	118	1752712800	wiki.bibble.co.nz	192.168.1.220:8080	bibble-caddy-exporter	Not Unique as Scrap time the same)

This matters a you cannot use organize fields in transformations

Getting My Graph to Work

So to get this to work using the data above I used Labels to Fields which gives

Site	Bucket	Timestamp	Value
wiki.bibble.co.nz	1752712500	1752719700	123
wiki.bibble.co.nz	1752712800	1752719700	118

Then merge which gives

Time	Site	Bucket	Requests
1752719700	wiki.bibble.co.nz	1752712500	123
1752719700	wiki.bibble.co.nz	1752712800	118

Than Add field from Calculation with row index

Row Index	Time	Site	Bucket	Requests
0	1752719700	wiki.bibble.co.nz	1752712500	123
1	1752719700	wiki.bibble.co.nz	1752712800	118

From there you can make the index a string to use as the label and you get this