1
0
Fork 0
mirror of https://github.com/prometheus-operator/prometheus-operator.git synced 2025-04-16 01:06:27 +00:00

Merge pull request #1383 from brancz/update-kops-docs

kube-prometheus: Update kops docs and add Ingress docs
This commit is contained in:
Frederic Branczyk 2018-05-25 17:14:00 +02:00 committed by GitHub
commit 4b62f2d48e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
15 changed files with 260 additions and 59 deletions

View file

@ -2,3 +2,4 @@ tmp/
minikube-manifests/
jsonnetfile.lock.json
vendor/
./auth

View file

@ -1,3 +1,5 @@
JSONNET_FMT := jsonnet fmt -n 2 --max-blank-lines 2 --string-style s --comment-style s
image:
docker build -f ../../scripts/jsonnet/Dockerfile -t po-jsonnet ../../
@ -11,10 +13,14 @@ crdtojsonnet:
cat ../../example/prometheus-operator-crd/servicemonitor.crd.yaml | gojsontoyaml -yamltojson > jsonnet/kube-prometheus/prometheus-operator/servicemonitor-crd.libsonnet
cat ../../example/prometheus-operator-crd/rulefile.crd.yaml | gojsontoyaml -yamltojson > jsonnet/kube-prometheus/prometheus-operator/rulefile-crd.libsonnet
generate-raw: crdtojsonnet
generate-raw: crdtojsonnet fmt
jb install
./build.sh
fmt:
find . -name 'vendor' -prune -o -name '*.libsonnet' -o -name '*.jsonnet' -print | \
xargs -n 1 -- $(JSONNET_FMT) -i
test: image
@echo ">> Compiling assets and generating Kubernetes manifests"
docker run --rm -u=$(shell id -u $(USER)):$(shell id -g $(USER)) -v $(shell dirname $(dir $(abspath $(dir $$PWD)))):/go/src/github.com/coreos/prometheus-operator/ --workdir /go/src/github.com/coreos/prometheus-operator/contrib/kube-prometheus po-jsonnet make test-raw
@ -23,4 +29,4 @@ test-raw: crdtojsonnet
jb install
./test.sh
.PHONY: image generate crdtojsonnet generate-raw test
.PHONY: image generate crdtojsonnet generate-raw test test-raw fmt

View file

@ -17,6 +17,19 @@ Components included in this package:
This stack is meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components. In addition to that it delivers a default set of dashboards and alerting rules. Many of the useful dashboards and alerts come from the [kubernetes-mixin project](https://github.com/kubernetes-monitoring/kubernetes-mixin), similar to this project it provides composable jsonnet as a library for users to customize to their needs.
## Table of contents
* [Prerequisites](#prerequisites)
* [minikube](#minikube)
* [Quickstart](#quickstart)
* [Usage](#usage)
* [Compiling](#compiling)
* [Configuration](#configuration)
* [Customization](#customization)
* [Customizing Prometheus alerting/recording rules and Grafana dashboards](#customizing-prometheus-alertingrecording-rules-and-grafana-dashboards)
* [Exposing Prometheus/Alermanager/Grafana via Ingress](#exposing-prometheusalermanagergrafana-via-ingress)
* [Minikube Example](#minikube-example)
## Prerequisites
You will need a Kubernetes cluster, that's it! By default it is assumed, that the kubelet uses token authN and authZ, as otherwise Prometheus needs a client certificate, which gives it full access to the kubelet, rather than just the metrics. Token authN and authZ allows more fine grained and easier access control.
@ -151,39 +164,47 @@ kubeadm:
[embedmd]:# (examples/jsonnet-snippets/kubeadm.jsonnet)
```jsonnet
(import "kube-prometheus/kube-prometheus.libsonnet") +
(import "kube-prometheus/kube-prometheus-kubeadm.libsonnet")
(import 'kube-prometheus/kube-prometheus.libsonnet') +
(import 'kube-prometheus/kube-prometheus-kubeadm.libsonnet')
```
bootkube:
[embedmd]:# (examples/jsonnet-snippets/bootkube.jsonnet)
```jsonnet
(import "kube-prometheus/kube-prometheus.libsonnet") +
(import "kube-prometheus/kube-prometheus-bootkube.libsonnet")
(import 'kube-prometheus/kube-prometheus.libsonnet') +
(import 'kube-prometheus/kube-prometheus-bootkube.libsonnet')
```
kops:
[embedmd]:# (examples/jsonnet-snippets/kops.jsonnet)
```jsonnet
(import 'kube-prometheus/kube-prometheus.libsonnet') +
(import 'kube-prometheus/kube-prometheus-kops.libsonnet')
```
Another mixin that may be useful for exploring the stack is to expose the UIs of Prometheus, Alertmanager and Grafana on NodePorts:
[embedmd]:# (examples/jsonnet-snippets/node-ports.jsonnet)
```jsonnet
(import "kube-prometheus/kube-prometheus.libsonnet") +
(import "kube-prometheus/kube-prometheus-node-ports.libsonnet")
(import 'kube-prometheus/kube-prometheus.libsonnet') +
(import 'kube-prometheus/kube-prometheus-node-ports.libsonnet')
```
For example the name of the `Prometheus` object provided by this library can be overridden:
[embedmd]:# (examples/prometheus-name-override.jsonnet)
```jsonnet
((import "kube-prometheus/kube-prometheus.libsonnet") + {
prometheus+: {
prometheus+: {
metadata+: {
name: "my-name",
}
}
}
}).prometheus.prometheus
((import 'kube-prometheus/kube-prometheus.libsonnet') + {
prometheus+: {
prometheus+: {
metadata+: {
name: 'my-name',
},
},
},
}).prometheus.prometheus
```
Standard Kubernetes manifests are all written using [ksonnet-lib](https://github.com/ksonnet/ksonnet-lib/), so they can be modified with the mixins supplied by ksonnet-lib. For example to override the namespace of the node-exporter DaemonSet:
@ -203,9 +224,13 @@ local daemonset = k.apps.v1beta2.daemonSet;
### Customizing Prometheus alerting/recording rules and Grafana dashboards
See [developing alerts and dashboards](docs/developing-prometheus-rules-and-grafana-dashboards.md) guide.
See [developing Prometheus rules and Grafana dashboards](docs/developing-prometheus-rules-and-grafana-dashboards.md) guide.
## Example
### Exposing Prometheus/Alermanager/Grafana via Ingress
See [exposing Prometheus/Alertmanager/Grafana](docs/exposing-prometheus-alertmanager-grafana-ingress.md) guide.
## Minikube Example
To use an easy to reproduce example, let's take the minikube setup as demonstrated in [prerequisites](#Prerequisites). It is a kubeadm cluster (as we use the kubeadm bootstrapper) and because we would like easy access to our Prometheus, Alertmanager and Grafana UI we want the services to be exposed as NodePort type services:

View file

@ -1,20 +0,0 @@
# Adding kube-prometheus to [KOPS](https://github.com/kubernetes/kops) on AWS 1.5.x
## Prerequisites
A running Kubernetes cluster created with [KOPS](https://github.com/kubernetes/kops).
These instructions have currently been tested with **topology=public** on AWS with KOPS 1.7.1 and Kubernetes 1.7.x
Following the instructions in the [README](https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/README.md):
Example:
```bash
git clone -b master https://github.com/coreos/prometheus-operator.git prometheus-operator-temp;
cd prometheus-operator-temp/contrib/kube-prometheus
./hack/cluster-monitoring/self-hosted-deploy
cd -
rm -rf prometheus-operator-temp
```

View file

@ -0,0 +1,91 @@
# Exposing Prometheus, Alertmanager and Grafana UIs via Ingress
In order to access the web interfaces via the Internet [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) is a popular option. This guide explains, how Kubernetes Ingress can be setup, in order to expose the Prometheus, Alertmanager and Grafana UIs, that are included in the [kube-prometheus](https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus) project.
Note: before continuing, it is recommended to first get familiar with the [kube-prometheus](https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus) stack by itself.
## Prerequisites
Apart from a running Kubernetes cluster with a running [kube-prometheus](https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus) stack, a Kubernetes Ingress controller must be installed and functional. This guide was tested with the [nginx-ingress-controller](https://github.com/kubernetes/ingress-nginx). If you wish to reproduce the exact result in as depicted in this guide we recommend using the nginx-ingress-controller.
## Setting up Ingress
The setup of Ingress objects is the same for Prometheus, Alertmanager and Grafana. Therefore this guides demonstrates it in detail for Prometheus as it can easily be adapted for the other applications.
As monitoring data may contain sensitive data, this guide describes how to setup Ingress with basic auth as an example of minimal security. Of course this should be adapted to the preferred authentication mean of any particular organization, but we feel it is important to at least provide an example with a minimum of security.
In order to setup basic auth, a secret with the `htpasswd` formatted file needs to be created. To do this, first install the [`htpasswd`](https://httpd.apache.org/docs/2.4/programs/htpasswd.html) tool.
To create the `htpasswd` formatted file called `auth` run:
```
htpasswd -c auth <username>
```
In order to use this a secret needs to be created containing the name of the `htpasswd`, and with annotations on the Ingress object basic auth can be configured.
[embedmd]:# (../examples/ingress.jsonnet)
```jsonnet
local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';
local secret = k.core.v1.secret;
local ingress = k.extensions.v1beta1.ingress;
local ingressTls = ingress.mixin.spec.tlsType;
local ingressRule = ingress.mixin.spec.rulesType;
local httpIngressPath = ingressRule.mixin.http.pathsType;
local kp =
(import 'kube-prometheus/kube-prometheus.libsonnet') +
{
_config+:: {
namespace: 'monitoring',
},
ingress+:: {
'prometheus-k8s':
ingress.new() +
ingress.mixin.metadata.withName('prometheus-k8s') +
ingress.mixin.metadata.withNamespace($._config.namespace) +
ingress.mixin.metadata.withAnnotations({
'nginx.ingress.kubernetes.io/auth-type': 'basic',
'nginx.ingress.kubernetes.io/auth-secret': 'basic-auth',
'nginx.ingress.kubernetes.io/auth-realm': 'Authentication Required',
}) +
ingress.mixin.spec.withRules(
ingressRule.new() +
ingressRule.withHost('prometheus.example.com') +
ingressRule.mixin.http.withPaths(
httpIngressPath.new() +
httpIngressPath.mixin.backend.withServiceName('prometheus-k8s') +
httpIngressPath.mixin.backend.withServicePort('web')
),
),
},
} + {
ingress+:: {
'basic-auth-secret':
secret.new('basic-auth', { auth: std.base64(importstr 'auth') }) +
secret.mixin.metadata.withNamespace($._config.namespace),
},
};
k.core.v1.list.new([
kp.ingress['prometheus-k8s'],
kp.ingress['basic-auth-secret'],
])
```
In order to expose Alertmanager and Grafana, simply create additional fields containing an ingress object, but simply pointing at the `alertmanager` or `grafana` instead of the `prometheus-k8s` Service. Make sure to also use the correct port respectively, for Alertmanager it is also `web`, for Grafana it is `http`.
In order to render the ingress objects similar to the other objects use as demonstrated in the [main readme](../README.md#usage):
```
{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['ingress-' + name]: kp.ingress[name] for name in std.objectFields(kp.ingress) }
```
Note, that in comparison only the last line was added, the rest is identical to the original.

View file

@ -0,0 +1,2 @@
# This file should not ever be used, it's just a mock.
dontusethis:$apr1$heg6VIp7$1PSzJ/Z6fYboQ5pYrbgSy.

View file

@ -0,0 +1,45 @@
local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';
local secret = k.core.v1.secret;
local ingress = k.extensions.v1beta1.ingress;
local ingressTls = ingress.mixin.spec.tlsType;
local ingressRule = ingress.mixin.spec.rulesType;
local httpIngressPath = ingressRule.mixin.http.pathsType;
local kp =
(import 'kube-prometheus/kube-prometheus.libsonnet') +
{
_config+:: {
namespace: 'monitoring',
},
ingress+:: {
'prometheus-k8s':
ingress.new() +
ingress.mixin.metadata.withName('prometheus-k8s') +
ingress.mixin.metadata.withNamespace($._config.namespace) +
ingress.mixin.metadata.withAnnotations({
'nginx.ingress.kubernetes.io/auth-type': 'basic',
'nginx.ingress.kubernetes.io/auth-secret': 'basic-auth',
'nginx.ingress.kubernetes.io/auth-realm': 'Authentication Required',
}) +
ingress.mixin.spec.withRules(
ingressRule.new() +
ingressRule.withHost('prometheus.example.com') +
ingressRule.mixin.http.withPaths(
httpIngressPath.new() +
httpIngressPath.mixin.backend.withServiceName('prometheus-k8s') +
httpIngressPath.mixin.backend.withServicePort('web')
),
),
},
} + {
ingress+:: {
'basic-auth-secret':
secret.new('basic-auth', { auth: std.base64(importstr 'auth') }) +
secret.mixin.metadata.withNamespace($._config.namespace),
},
};
k.core.v1.list.new([
kp.ingress['prometheus-k8s'],
kp.ingress['basic-auth-secret'],
])

View file

@ -1,2 +1,2 @@
(import "kube-prometheus/kube-prometheus.libsonnet") +
(import "kube-prometheus/kube-prometheus-bootkube.libsonnet")
(import 'kube-prometheus/kube-prometheus.libsonnet') +
(import 'kube-prometheus/kube-prometheus-bootkube.libsonnet')

View file

@ -0,0 +1,2 @@
(import 'kube-prometheus/kube-prometheus.libsonnet') +
(import 'kube-prometheus/kube-prometheus-kops.libsonnet')

View file

@ -1,2 +1,2 @@
(import "kube-prometheus/kube-prometheus.libsonnet") +
(import "kube-prometheus/kube-prometheus-kubeadm.libsonnet")
(import 'kube-prometheus/kube-prometheus.libsonnet') +
(import 'kube-prometheus/kube-prometheus-kubeadm.libsonnet')

View file

@ -1,2 +1,2 @@
(import "kube-prometheus/kube-prometheus.libsonnet") +
(import "kube-prometheus/kube-prometheus-node-ports.libsonnet")
(import 'kube-prometheus/kube-prometheus.libsonnet') +
(import 'kube-prometheus/kube-prometheus-node-ports.libsonnet')

View file

@ -1,9 +1,9 @@
((import "kube-prometheus/kube-prometheus.libsonnet") + {
prometheus+: {
prometheus+: {
metadata+: {
name: "my-name",
}
}
}
}).prometheus.prometheus
((import 'kube-prometheus/kube-prometheus.libsonnet') + {
prometheus+: {
prometheus+: {
metadata+: {
name: 'my-name',
},
},
},
}).prometheus.prometheus

View file

@ -0,0 +1,25 @@
{
prometheus+:: {
serviceMonitorKubelet+:
{
spec+: {
endpoints: [
{
port: 'http-metrics',
scheme: 'http',
interval: '30s',
bearerTokenFile: '/var/run/secrets/kubernetes.io/serviceaccount/token',
},
{
port: 'http-metrics',
scheme: 'http',
path: '/metrics/cadvisor',
interval: '30s',
honorLabels: true,
bearerTokenFile: '/var/run/secrets/kubernetes.io/serviceaccount/token',
},
],
},
},
},
}

View file

@ -0,0 +1,23 @@
local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';
local service = k.core.v1.service;
local servicePort = k.core.v1.service.mixin.spec.portsType;
{
prometheus+:: {
kubeControllerManagerPrometheusDiscoveryService:
service.new('kube-controller-manager-prometheus-discovery', { 'k8s-app': 'kube-controller-manager' }, servicePort.newNamed('http-metrics', 10252, 10252)) +
service.mixin.metadata.withNamespace('kube-system') +
service.mixin.metadata.withLabels({ 'k8s-app': 'kube-controller-manager' }) +
service.mixin.spec.withClusterIp('None'),
kubeSchedulerPrometheusDiscoveryService:
service.new('kube-scheduler-prometheus-discovery', { 'k8s-app': 'kube-scheduler' }, servicePort.newNamed('http-metrics', 10251, 10251)) +
service.mixin.metadata.withNamespace('kube-system') +
service.mixin.metadata.withLabels({ 'k8s-app': 'kube-scheduler' }) +
service.mixin.spec.withClusterIp('None'),
kubeDnsPrometheusDiscoveryService:
service.new('kube-dns-prometheus-discovery', { 'k8s-app': 'kube-dns' }, [servicePort.newNamed('http-metrics-skydns', 10055, 10055), servicePort.newNamed('http-metrics-dnsmasq', 10054, 10054)]) +
service.mixin.metadata.withNamespace('kube-system') +
service.mixin.metadata.withLabels({ 'k8s-app': 'kube-dns' }) +
service.mixin.spec.withClusterIp('None'),
},
}

View file

@ -253,13 +253,14 @@ data:
\"API server is erroring for {{ $value }}% of requests.\"\n \"expr\": |\n sum(rate(apiserver_request_count{job=\"kube-apiserver\",code=~\"^(?:5..)$\"}[5m]))
without(instance, pod)\n /\n sum(rate(apiserver_request_count{job=\"kube-apiserver\"}[5m]))
without(instance, pod) * 100 > 5\n \"for\": \"10m\"\n \"labels\": \n \"severity\":
\"warning\"\n - \"alert\": \"KubeCertificateExpiration\"\n \"annotations\":
\"warning\"\n - \"alert\": \"KubeClientCertificateExpiration\"\n \"annotations\":
\n \"message\": \"Kubernetes API certificate is expiring in less than 7 days.\"\n
\ \"expr\": |\n sum(apiserver_client_certificate_expiration_seconds_bucket{job=\"kube-apiserver\",le=\"604800\"})
> 0\n \"labels\": \n \"severity\": \"warning\"\n - \"alert\": \"KubeCertificateExpiration\"\n
\ \"expr\": |\n histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"kube-apiserver\"}[5m])))
< 604800\n \"labels\": \n \"severity\": \"warning\"\n - \"alert\": \"KubeClientCertificateExpiration\"\n
\ \"annotations\": \n \"message\": \"Kubernetes API certificate is expiring
in less than 1 day.\"\n \"expr\": |\n sum(apiserver_client_certificate_expiration_seconds_bucket{job=\"kube-apiserver\",le=\"86400\"})
> 0\n \"labels\": \n \"severity\": \"warning\""
in less than 1 day.\"\n \"expr\": |\n histogram_quantile(0.01, sum by
(job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=\"kube-apiserver\"}[5m])))
< 86400\n \"labels\": \n \"severity\": \"critical\""
kind: ConfigMap
metadata:
labels: