Reboot Sentinel File Period¶

By default kured checks for the existence of /var/run/reboot-required every sixty minutes; you can override these values with --reboot-sentinel and --period. Each replica of the daemon uses a random offset derived from the period on startup so that nodes don't all contend for the lock simultaneously.

Setting a schedule¶

By default, kured will reboot any time it detects the sentinel, but this may cause reboots during odd hours. While service disruption does not normally occur, anything is possible and operators may want to restrict reboots to predictable schedules. Use --reboot-days, --start-time, --end-time, and --time-zone to set a schedule. For example, business hours on the west coast USA can be specified with:

	--reboot-days mon,tue,wed,thu,fri
	--start-time 9am
	--end-time 5pm
	--time-zone America/Los_Angeles

Times can be formatted in numerous ways, including 5pm, 5:00pm 17:00, and 17. --time-zone represents a Go time.Location, and can be UTC, Local, or any entry in the standard Linux tz database.

Note that when using smaller time windows, you should consider shortening the sentinel check period (--period).

Blocking Reboots via Alerts¶

You may find it desirable to block automatic node reboots when there are active alerts - you can do so by providing the URL of your Prometheus server:

--prometheus-url=http://prometheus.monitoring.svc.cluster.local

By default the presence of any active (pending or firing) alerts will block reboots, however you can ignore specific alerts:

--alert-filter-regexp=^(RebootRequired|AnotherBenignAlert|...$

See the section on Prometheus metrics for an important application of this filter.

Blocking Reboots via Pods¶

You can also block reboots of an individual node when specific pods are scheduled on it:

--blocking-pod-selector=runtime=long,cost=expensive

Since label selector strings use commas to express logical 'and', you can specify this parameter multiple times for 'or':

--blocking-pod-selector=runtime=long,cost=expensive
--blocking-pod-selector=name=temperamental

In this case, the presence of either an (appropriately labelled) expensive long running job or a known temperamental pod on a node will stop it rebooting.

Prometheus Metrics¶

Each kured pod exposes a single gauge metric (:8080/metrics) that indicates the presence of the sentinel file:

# HELP kured_reboot_required OS requires reboot due to software updates.
# TYPE kured_reboot_required gauge
kured_reboot_required{node="ip-xxx-xxx-xxx-xxx.ec2.internal"} 0

The purpose of this metric is to power an alert which will summon an operator if the cluster cannot reboot itself automatically for a prolonged period:

# Alert if a reboot is required for any machines. Acts as a failsafe for the
# reboot daemon, which will not reboot nodes if there are pending alerts save
# this one.
ALERT RebootRequired


  IF          max(kured_reboot_required) != 0


  FOR         24h


  LABELS      { severity="warning" }


  ANNOTATIONS {


    summary = "Machine(s) require being rebooted, and the reboot daemon has failed to do so for 24 hours",


    impact = "Cluster nodes more vulnerable to security exploits. Eventually, no disk space left.",


    description = "Machine(s) require being rebooted, probably due to kernel update.",


  }

If you choose to employ such an alert and have configured kured to probe for active alerts before rebooting, be sure to specify --alert-filter-regexp=^RebootRequired$ to avoid deadlock!

Slack Notifications¶

If you specify a Slack hook via --slack-hook-url, kured will notify you immediately prior to rebooting a node:

We recommend setting --slack-username to be the name of the environment, e.g. dev or prod.

Overriding Lock Configuration¶

The --ds-name and --ds-namespace arguments should match the name and namespace of the daemonset used to deploy the reboot daemon - the locking is implemented by means of an annotation on this resource. The defaults match the daemonset YAML provided in the repository.

Similarly --lock-annotation can be used to change the name of the annotation kured will use to store the lock, but the default is almost certainly safe.

Testing¶

You can test your configuration by provoking a reboot on a node:

sudo touch /var/run/reboot-required

Disabling Reboots¶

If you need to temporarily stop kured from rebooting any nodes, you can take the lock manually:

kubectl -n kube-system annotate ds kured weave.works/kured-node-lock='{"nodeID":"manual"}'

Don't forget to release it afterwards!

Manual Unlock¶

In exceptional circumstances, such as a node experiencing a permanent failure whilst rebooting, manual intervention may be required to remove the cluster lock:

kubectl -n kube-system annotate ds kured weave.works/kured-node-lock-

Why is there no latest tag on Docker Hub?¶

Use of latest for production deployments is bad practice - see here ⟨https://kubernetes.io/docs/concepts/configuration/overview⟩ for details. The manifest on master refers to latest for local development testing with minikube only; for production use choose a versioned manifest from the release page ⟨https://github.com/weaveworks/kured/releases/⟩.