Lightweight cluster metrics for FreeLens on Talos

I wanted FreeLens to show CPU and memory usage in my Talos cluster without deploying the whole kube-prometheus-stack extended universe. In the past I did run it and it surprised me how much it wrecks power usage on my server.

Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands.

Despite being an engineer, I didn’t feel like putting down the effort so I took Codex on a 2 hour trip to make a setup that barely works in hopes of a lighter solution.

This ended up a bit more complex than I would like, but was an overall success with around 0.5% CPU usage total.

TL;DR

metrics-server is enough for the quick glance columns in modern Freelens versions, but not for the whole UI.

The setup:

• metrics-server for Kubernetes’ metrics.k8s.io API
• VictoriaMetrics single as the Prometheus-ish endpoint
• prometheus-node-exporter for host CPU/memory/filesystem counters
• kube-state-metrics for requests, limits, capacity and allocatable values

Then Freelens needs to be configured like this:

• Provider type: Helm
• Prometheus path: vmsingle/prometheus:9090

metrics-server

Pod list CPU and memory columns come from:

/apis/metrics.k8s.io/v1beta1/pods

metrics-server/Chart.yaml:

apiVersion: v2
name: metrics-server
version: 1.0.0
dependencies:
  - name: metrics-server
    version: 3.13.1
    repository: https://kubernetes-sigs.github.io/metrics-server/

metrics-server/values.yaml:

metrics-server:
  args:
    - --kubelet-insecure-tls

  resources:
    requests:
      cpu: 20m
      memory: 64Mi
    limits:
      memory: 128Mi

The Talos-specific bit is --kubelet-insecure-tls. The cleaner version is kubelet server certificate rotation plus a CSR approver. I did not want to solve PKI for a UI sidebar.

Validation:

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl top nodes
kubectl top pods -A
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods

If kubectl top pods -A works, Freelens has enough data for the pod table.

VictoriaMetrics

This data is near-disposable. One day retention and an emptyDir suffice.

vmsingle/Chart.yaml:

apiVersion: v2
name: vmsingle
version: 1.0.0
dependencies:
  - name: victoria-metrics-single
    version: 0.35.0
    repository: https://victoriametrics.github.io/helm-charts/
  - name: prometheus-node-exporter
    version: 4.55.0
    repository: https://prometheus-community.github.io/helm-charts
    condition: prometheus-node-exporter.enabled
  - name: kube-state-metrics
    version: 7.5.1
    repository: https://prometheus-community.github.io/helm-charts

vmsingle/values.yaml:

victoria-metrics-single:
  server:
    retentionPeriod: 1d

    persistentVolume:
      enabled: false

    emptyDir:
      sizeLimit: 1Gi

    resources:
      requests:
        cpu: 25m
        memory: 96Mi
      limits:
        memory: 256Mi

    scrape:
      enabled: true
      config:
        global:
          scrape_interval: 30s
          scrape_timeout: 10s
        scrape_configs:
          # The jobs below go here.

prometheus-node-exporter:
  enabled: true
  resources:
    requests:
      cpu: 10m
      memory: 24Mi
    limits:
      memory: 64Mi

kube-state-metrics:
  replicas: 1
  resources:
    requests:
      cpu: 10m
      memory: 48Mi
    limits:
      memory: 128Mi

This is where the naming gets annoying. metrics-server and kube-state-metrics sound related but do completely different jobs.

metrics-server is live usage through Kubernetes APIs. kube-state-metrics is Kubernetes object state as Prometheus metrics.

Scrapes

These go under victoria-metrics-single.server.scrape.config.scrape_configs in vmsingle/values.yaml.

Pod usage needs kubelet cAdvisor through the Kubernetes API proxy:

- job_name: kubelet-cadvisor
  scheme: https
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  kubernetes_sd_configs:
    - role: node
  relabel_configs:
    - source_labels:
        - __meta_kubernetes_node_name
      target_label: node
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels:
        - __meta_kubernetes_node_name
      target_label: __metrics_path__
      replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
  metric_relabel_configs:
    - source_labels:
        - __name__
      action: keep
      regex: ^((container|pod|node)_(cpu_usage_seconds_total|memory_working_set_bytes)|resource_scrape_error)$

This matters because Freelens queries filter on labels like image!="". The kubelet resource endpoint did not have the shape Freelens expected. cAdvisor did.

Node usage:

- job_name: node-exporter
  kubernetes_sd_configs:
    - role: endpoints
  relabel_configs:
    - source_labels:
        - __meta_kubernetes_namespace
        - __meta_kubernetes_service_name
      action: keep
      regex: vmsingle;vmsingle-prometheus-node-exporter
    - source_labels:
        - __meta_kubernetes_endpoint_node_name
      target_label: node
    - source_labels:
        - __meta_kubernetes_endpoint_node_name
      target_label: kubernetes_node
    - source_labels:
        - __meta_kubernetes_endpoint_node_name
      target_label: instance
  metric_relabel_configs:
    - source_labels:
        - __name__
      action: keep
      regex: ^node_(cpu_seconds_total|memory_(MemTotal|MemFree|Buffers|Cached)_bytes|filesystem_(size|avail)_bytes)$

The duplicated labels are deliberate. The Helm provider mostly wants node, but some Lens-derived query paths look for instance or kubernetes_node. Carrying all three is cheap and avoids a surprisingly convincing 0% CPU lie.

Object state:

- job_name: kube-state-metrics
  static_configs:
    - targets:
        - vmsingle-kube-state-metrics:8080
  metric_relabel_configs:
    - source_labels:
        - __name__
      action: keep
      regex: ^kube_(node_status_(capacity|allocatable)|pod_container_resource_(requests|limits)|pod_info)$

RBAC and PSA

VictoriaMetrics needs certain permissions to perform kubelet scraping and endpoint discovery.

vmsingle/templates/rbac.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vmsingle-kubelet-resource
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
      - pods
      - services
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vmsingle-kubelet-resource
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vmsingle-kubelet-resource
subjects:
  - kind: ServiceAccount
    name: vmsingle-victoria-metrics-single-server
    namespace: {{ .Release.Namespace }}

Talos by default carries a restrictive security policy that we need to work around for node-exporter:

vmsingle/templates/privileged-namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Release.Namespace }}
  labels:
    pod-security.kubernetes.io/enforce: privileged

Helper service

This is just a nice name for Freelens.

vmsingle/templates/prometheus-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: {{ .Release.Namespace }}
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: server
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: vmsingle
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9090
      protocol: TCP
      targetPort: http
  selector:
    app: server
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/name: victoria-metrics-single

Verify

From Kubernetes:

kubectl top nodes
kubectl top pods -A
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods

From VictoriaMetrics:

count(container_memory_working_set_bytes{image!=""})
count(kube_pod_container_resource_requests)
count(node_cpu_seconds_total)
count(node_memory_MemTotal_bytes)
sum(rate(node_cpu_seconds_total{mode=~"user|system"}[5m])) by(node)

From Freelens

it should be pretty obvious :)

The only caveat I found is that the Node view formats the CPU value as 0.0Ki for me.

The underlying PromQL queries were correct, so I am going to pretend I did not see that.