Prometheus 基于 Pod 和 Service 注解的服务发现

背景

很多应用会为 Pod 或 Service 打上一些注解用于 Prometheus 的服务发现，如 prometheus.io/scrape: "true"，这种注解并不是 Prometheus 官方支持的，而是社区的习惯性用法，要使这种注解生效，还需结合 Prometheus 的采集配置，本文介绍具体的配置方法。

注意：如果你使用 kube-prometheus-stack 部署的监控系统，默认就会对自身的一些组件创建采集规则，比如会给 kube-state-metrics 创建 ServiceMonitor，并且 kube-state-metrics 的 Service 上也有 prometheus.io/scrape: "true" 的注解，如果配置了基于 Service 注解的服务发现，就会导致重复采集。

真实案例

istio 指标采集

istio 使用了这种 Pod 注解，当 Pod 被自动注入 sidecar 的同时也会被自动注入以下注解：

代码语言：yaml复制

    prometheus.io/path: /stats/prometheus
    prometheus.io/port: "15020"
    prometheus.io/scrape: "true"

表示声明让 Prometheus 采集 Envoy Sidecar 暴露的 metrics，端口是 15020，路径是 /stats/prometheus。

除此之外，控制面组件 istiod 的 Pod 也会有类似注解：

代码语言：yaml复制

    prometheus.io/port: "15014"
    prometheus.io/scrape: "true"

Kubernetes Addon 指标采集

Kubenretes 源码仓库中的一些 addon 组件也使用了这种注解，有的是 Pod 注解，有的是 Service 注解。

• coredns 使用 Service 注解：

代码语言：yaml showLineNumbers复制

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"  

• nodelocaldns 使用 Pod 注解：

代码语言：yaml showLineNumbers复制

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    k8s-app: node-local-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
      annotations:
        prometheus.io/port: "9253"
        prometheus.io/scrape: "true"

Prometheus 采集配置

根据 Pod 注解动态采集

代码语言：yaml复制

- job_name: "kubernetes-pods"

  kubernetes_sd_configs:
    - role: pod

  relabel_configs:
    # prometheus relabel to scrape only pods that have
    # `prometheus.io/scrape: "true"` annotation.
    - source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      action: keep
      regex: true

    # prometheus relabel to customize metric path based on pod
    # `prometheus.io/path: <metric path>` annotation.
    - source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
      action: replace
      target_label: __metrics_path__
      regex: (. )

    # prometheus relabel to scrape only single, desired port for the pod
    # based on pod `prometheus.io/port: <port>` annotation.
    - source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
      action: replace
      regex: ([^:] )(?::d )?;(d )
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(. )
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: pod

根据 Service 注解动态采集

代码语言：yaml复制

- job_name: "kubernetes-service-endpoints"

  kubernetes_sd_configs:
    - role: endpoints

  relabel_configs:
    # prometheus relabel to scrape only endpoints that have
    # `prometheus.io/scrape: "true"` annotation.
    - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      action: keep
      regex: true

    # prometheus relabel to customize metric path based on endpoints
    # `prometheus.io/path: <metric path>` annotation.
    - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
      action: replace
      target_label: __metrics_path__
      regex: (. )

    # prometheus relabel to scrape only single, desired port for the service based
    # on endpoints `prometheus.io/port: <port>` annotation.
    - source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
      action: replace
      regex: ([^:] )(?::d )?;(d )
      replacement: $1:$2
      target_label: __address__

    # prometheus relabel to configure scrape scheme for all service scrape targets
    # based on endpoints `prometheus.io/scheme: <scheme>` annotation.
    - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
      action: replace
      target_label: __scheme__
      regex: (https?)
    - action: labelmap
      regex: __meta_kubernetes_service_label_(. )
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: service

kube-prometheus-stack 采集配置方法

如果你使用 kube-prometheus-stack 来安装 Prometheus，需要在 additionalScrapeConfigs 里加上采集配置，示例:

代码语言：yaml复制

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      -  job_name: "kubernetes-service-endpoints"
         ...
      -  job_name: "kubernetes-pods"
         ...

参考资料

• kube-prometheus-stack: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
• istio: https://istio.io/

prometheus

0 人点赞