mirrors/prometheus-operator
mirrors/prometheus-operator
1
0
Fork 0
mirror of https://github.com/prometheus-operator/prometheus-operator.git synced 2025-04-21 11:48:53 +00:00
Code Activity

pkg/alertmanager: Use lower value for --cluster.reconnect-timeout

Browse source 
Alertmanager in cluster mode resolves the DNS name of each peer and
caches its IP address which uses on regular intervals to 'refresh'
the connection.

In high-dynamic environment like kubernetes, it's possible that
alertmanager pods come and go on frequent intervals. The default timeout
value of 6h is not suitable in that case as alertmanager will keep
trying to reconnect to a non-existing pod over and over until it gives
up and remove that peer from the member list. During this period of
time, the cluster is reported to be in a degraded state due to the
missing member.

As such, it's best to use a lower value which will allow the
alertmanager to remove the pod from the list of peers soon
after it disappears.

Related: https://github.com/prometheus/alertmanager/issues/2250
This commit is contained in:
Markos Chandras 2020-08-24 14:45:27 +03:00
parent 08a964702c
commit 86102e73e9
No known key found for this signature in database
GPG key ID: 1F10ED4B81DEBF19
1 changed files with 6 additions and 0 deletions

6
pkg/alertmanager/statefulset.go
View file

@ -384,6 +384,12 @@ func makeStatefulSetSpec(a *monitoringv1.Alertmanager, config Config) (*appsv1.S
// below Alertmanager v0.15.0 high availability flags are prefixed with 'mesh' instead of 'cluster'
amArgs[i] = strings.Replace(amArgs[i], "--cluster.", "--mesh.", 1)
}
} else {
// reconnect-timeout was added in 0.15 (https://github.com/prometheus/alertmanager/pull/1384)
// Override default 6h value to allow AlertManager cluster to
// quickly remove a cluster member after its pod restarted or during a
// regular rolling update.
amArgs = append(amArgs, "--cluster.reconnect-timeout=5m")
}
if version.Minor < 13 {
for i := range amArgs {