diff --git a/docs/_config.yml b/docs/_config.yml index ac52476e4..13b044a6d 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -44,6 +44,14 @@ remote_theme: rundocs/jekyll-rtd-theme@v2.0.10 version: master display_version_list: true +# Customize page width (make it wider) for large screens +scss: | + @media(min-width: 1280px){ + .content-wrap{ + max-width: 1200px + } + } + # Release is the full released version number. Used to make external links to # point to the correct blobs in the Github repo. This is also the version shown # in the sidebar (top left corner of the page) diff --git a/docs/advanced/customization-guide.md b/docs/advanced/customization-guide.md index f8a8a824f..b401942e6 100644 --- a/docs/advanced/customization-guide.md +++ b/docs/advanced/customization-guide.md @@ -2,7 +2,6 @@ title: "Customization guide" layout: default sort: 5 -published: false --- # Customization guide @@ -16,8 +15,1029 @@ published: false --- -***WORK IN PROGRESS.*** +## Overview + +NFD provides multiple extension points for vendor and application specific +labeling: + +- [`NodeFeatureRule`](#nodefeaturerule-custom-resource) objects provide a way to + deploy custom labeling rules via the Kubernetes API +- [`local`](#local-feature-source) feature source of nfd-worker creates + labels by executing hooks and reading files +- [`custom`](#custom-feature-source) feature source of nfd-worker creates + labels based on user-specified rules + +## NodeFeatureRule custom resource + +`NodeFeatureRule` objects provide an easy way to create vendor or application +specific labels. It uses a flexible rule-based mechanism for creating labels +based on node feature. + +### A NodeFeatureRule example + +Consider the following referential example: + +```yaml +apiVersion: nfd.k8s-sigs.io/v1alpha1 +kind: NodeFeatureRule +metadata: + name: my-sample-rule-object +spec: + rules: + - name: "my sample rule" + labels: + "my-sample-feature": "true" + matchFeatures: + - feature: kernel.loadedmodule + matchExpressions: + dummy: {op: Exists} + - feature: kernel.config + matchExpressions: + X86: {op: In, value: ["y"]} +``` + +It specifies one rule which creates node label +`feature.node.kubenernetes.io/my-sample-feature=true` if both of the following +conditions are true (`matchFeatures` implements a logical AND over the +matchers): + +- The `dummy` network driver module has been loaded +- X86 option in kernel config is set to `=y` + +Create a `NodeFeatureRule` with a yaml file: + +```bash +kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/node-feature-discovery/{{ site.release }}/examples/nodefeaturerule.yaml +``` + +Now, on X86 platforms the feature label appears after doing `modprobe dummy` on +a system and correspondingly the label is removed after `rmmod dummy`. Note a +re-labeling delay up to the sleep-interval of nfd-worker (1 minute by default). + +### NodeFeatureRule controller + +NFD-Master acts as the controller for `NodeFeatureRule` objects. It applies these +rules on raw feature data received from nfd-worker instances and creates node +labels, accordingly. + +**NOTE** nfd-master is stateless and (re-)labelling only happens when a request +is received from nfd-worker. That is, in practice rules are evaluated and +labels for each node are created on intervals specified by the +[`core.sleepInterval`](worker-configuration-reference#coresleepinterval) +configuration option (or +[`-sleep-interval`](worker-commandline-reference#-sleep-interval) command line +flag) of nfd-worker instances. This means that modification or creation of +`NodeFeatureRule` objects does not instantly cause the node labels to be updated. +Instead, the changes only come visible in node labels as nfd-worker instances +send their labelling requests. + +## Local feature source + +NFD-Worker has a special feature source named `local` which is an integration +point for external feature detectors. It provides a mechanism for pluggable +extensions, allowing the creation of new user-specific features and even +overriding built-in labels. + +The `local` feature source has two methods for detecting features, hooks and +feature files. The features discovered by the `local` source can further be +used in label rules specified in +[`NodeFeatureRule`](#nodefeaturerule-custom-resource) objects and the +[`custom`](#custom-feature-source) feature source. + +**NOTE:** Be careful when creating and/or updating hook or feature files while +NFD is running. In order to avoid race conditions you should write into a +temporary file (outside the `source.d` and `features.d` directories), and, +atomically create/update the original file by doing a filesystem move +operation. + +### A hook example + +Consider a shell script +`/etc/kubernetes/node-feature-discovery/source.d/my-hook.sh` having the +following stdout output, or alternatively, a plaintext file +`/etc/kubernetes/node-feature-discovery/features.d/my-features` having the +following contents: + +```plaintext +my-feature.1 +my-feature.2=myvalue +my.namespace/my-feature.3=456 +``` + +This will translate into the following node labels: + +```yaml +feature.node.kubernetes.io/my-feature.1: "true" +feature.node.kubernetes.io/my-feature.2: "myvalue" +my.namespace/my-feature.3: "456" +``` + +Note that in the example above `-extra-label-ns=my.namespace` must be specified +on the nfd-master command line. + +### Hooks + +The `local` source executes hooks found in +`/etc/kubernetes/node-feature-discovery/source.d/`. The hook files must be +executable and they are supposed to print all discovered features in `stdout`. +With ELF binaries static linking is recommended as the selection of system +libraries available in the NFD release image is very limited. Other runtimes +currently supported by the NFD image are bash and perl. + +`stderr` output of hooks is propagated to NFD log so it can be used for +debugging and logging. + +NFD tries to execute any regular files found from the hooks directory. +Any additional data files the hook might need (e.g. a configuration file) +should be placed in a separate directory in order to avoid NFD unnecessarily +trying to execute them. A subdirectory under the hooks directory can be used, +for example `/etc/kubernetes/node-feature-discovery/source.d/conf/`. + +**NOTE:** NFD will blindly run any executables placed/mounted in the hooks +directory. It is the user's responsibility to review the hooks for e.g. +possible security implications. + +**NOTE:** The [minimal](deployment-and-usage#minimal) image variant only +supports running statically linked binaries. + +### Feature files + +The `local` source reads files found in +`/etc/kubernetes/node-feature-discovery/features.d/`. + +### Input format + +The hook stdout and feature files are expected to contain features in simple +key-value pairs, separated by newlines: + +```plaintext +[=] +``` + +The label value defaults to `true`, if not specified. + +Label namespace may be specified with `/[=]`. The +namespace must be explicitly allowed with the `-extra-label-ns` command line +flag of nfd-master if using something else than +`[.]feature.node.kubernetes.io` or +`[.]profile.node.kubernetes.io`. + +### Mounts + +The standard NFD deployments contain `hostPath` mounts for +`/etc/kubernetes/node-feature-discovery/source.d/` and +`/etc/kubernetes/node-feature-discovery/features.d/`, making these directories +from the host available inside the nfd-worker container. + +#### Injecting labels from other pods + +One use case for the hooks and/or feature files is detecting features in other +Pods outside NFD, e.g. in Kubernetes device plugins. By using the same +`hostPath` mounts for `/etc/kubernetes/node-feature-discovery/source.d/` and +`/etc/kubernetes/node-feature-discovery/features.d/` in the side-car (e.g. +device plugin) creates a shared area for deploying hooks and feature files to +NFD. NFD will periodically scan the directories and run any hooks and read any +feature files it finds. + +## Custom feature source + +The `custom` feature source in nfd-worker provides a rule-based mechanism for +label creation, similar to the +[`NodeFeatureRule`](#nodefeaturerule-custom-resource) objects. The difference is +that the rules are specified in the worker configuration instead of a +Kubernetes API object. + +See [worker configuration](deployment-and-usage.md#worker-configuration) for +instructions how to set-up and manage the worker configuration. + +### An example custom feature source configuration + +Consider the following referential configuration for nfd-worker: + +```yaml +core: + labelSources: ["custom"] +sources: + custom: + - name: "my sample rule" + labels: + "my-sample-feature": "true" + matchFeatures: + - feature: kernel.loadedmodule + matchExpressions: + dummy: {op: Exists} + - feature: kernel.config + matchExpressions: + X86: {op: In, value: ["y"]} +``` + +It specifies one rule which creates node label +`feature.node.kubenernetes.io/my-sample-feature=true` if both of the following +conditions are true (`matchFeatures` implements a logical AND over the +matchers): + +- The `dummy` network driver module has been loaded +- X86 option in kernel config is set to `=y` + +In addition, the configuration only enables the `custom` source, disabling all +built-in labels. + +Now, on X86 platforms the feature label appears after doing `modprobe dummy` on +a system and correspondingly the label is removed after `rmmod dummy`. Note a +re-labeling delay up to the sleep-interval of nfd-worker (1 minute by default). + +### Additional configuration directory + +In addition to the rules defined in the nfd-worker configuration file, the +`custom` feature source can read more configuration files located in the +`/etc/kubernetes/node-feature-discovery/custom.d/` directory. This makes more +dynamic and flexible configuration easier. + +As an example, consider having file +`/etc/kubernetes/node-feature-discovery/custom.d/my-rule.yaml` with the +following content: + +```yaml +- name: "my e1000 rule" + labels: + "e1000.present": "true" + matchFeatures: + - feature: kernel.loadedmodule + matchExpressions: + e1000: {op: Exists} +``` + +This simple rule will create `feature.node.kubenernetes.io/e1000.present=true` +label if the `e1000` kernel module has been loaded. + +The +[`samples/custom-rules`](https://github.com/kubernetes-sigs/node-feature-discovery/blob/{{site.release}}/deployment/overlays/samples/custom-rules) +kustomize overlay sample contains an example for deploying a custom rule from a +ConfigMap. + +## Node labels + +Feature labels have the following format: + +```plaintext +/ = +``` + +The namespace part (i.e. prefix) of the labels is controlled by nfd: + +- All built-in labels use `feature.node.kubernetes.io`. This is also + the default for user defined features that don't specify any namespace. +- User-defined labels are allowed to use: + - `feature.node.kubernetes.io` and `profile.node.kubernetes.io` plus their + sub-namespaces (e.g. `vendor.profile.node.kubernetes.io` and + `sub.ns.profile.node.kubernetes.io`) by default + - Additional namespaces may be enabled with the + [`-extra-label-ns`](../advanced/master-commandline-reference#-extra-label-ns) + command line flag of nfd-master + +## Label rule format + +This section describes the rule format used in +[`NodeFeatureRule`](#nodefeaturerule-custom-resource) objects and in the +configuration of the [`custom`](#custom-feature-source) feature source. + +It is based on a generic feature matcher that covers all features discovered by +nfd-worker. The rules rely on a unified data model of the available features +and a generic expression-based format. Features that can be used in the rules +are described in detail in [available features](#available-features) below. + +Take this rule as a referential example: + +```yaml + - name: "my feature rule" + labels: + "my-special-feature": "my-value" + matchFeatures: + - feature: cpu.cpuid + matchExpressions: + AVX512F: {op: Exists} + - feature: kernel.version + matchExpressions: + major: {op: In, value: ["5"]} + minor: {op: Gt, value: ["1"]} + - feature: pci.device + matchExpressions: + vendor: {op: In, value: ["8086"]} + class: {op: In, value: ["0200"]} +``` + +This will yield `feature.node.kubenernetes.io/my-special-feature=my-value` node +label if all of these are true (`matchFeatures` implements a logical AND over +the matchers): + +- the CPU has AVX512F capability +- kernel version is 5.2 or later (must be v5.x) +- an Intel network controller is present + +### Fields + +#### Name + +The `.name` field is required and used as an identifier of the rule. + +#### Labels + +The `.labels` is a map of the node labels to create if the rule matches. + +#### Labels template + +The `.labelsTemplate` field specifies a text template for dynamically creating +labels based on the matched features. See [templating](#templating) for +details. + +**NOTE** The `labels` field has priority over `labelsTemplate`, i.e. +labels specified in the `labels` field will override anything +originating from `labelsTemplate`. + +#### Vars + +The `.vars` field is a map of values (key-value pairs) to store for subsequent +rules to use. In other words, these are variables that are not advertised as +node labels. See [backreferences](#backreferences) for more details on the +usage of vars. + +#### Vars template + +The `.varsTemplate` field specifies a text template for dynamically creating +vars based on the matched features. See [templating](#templating) for details +on using templates and [backreferences](#backreferences) for more details on +the usage of vars. + +**NOTE** The `vars` field has priority over `varsTemplate`, i.e. +vars specified in the `vars` field will override anything originating from +`varsTemplate`. + +#### MatchFeatures + +The `.matchFeatures` field specifies a feature matcher, consisting of a list of +feature matcher terms. It implements a logical AND over the terms i.e. all +of them must match in order for the rule to trigger. + +```yaml + matchFeatures: + - feature: + matchExpressions: + : + op: + value: + - + - ... +``` + +The `.matchFeatures[].feature` field specifies the feature against which to +match. + +The `.matchFeatures[].matchExpressions` field specifies a map of expressions +which to evaluate against the elements of the feature. + +In each MatchExpression `op` specifies the operator to apply. Valid values are +described below. + +| Operator | Number of values | Matches when +| --------------- | ---------------- | ----------- +| `In` | 1 or greater | Input is equal to one of the values +| `NotIn` | 1 or greater | Input is not equal to any of the values +| `InRegexp` | 1 or greater | Values of the MatchExpression are treated as regexps and input matches one or more of them +| `Exists` | 0 | The key exists +| `DoesNotExist` | 0 | The key does not exists +| `Gt` | 1 | Input is greater than the value. Both the input and value must be integer numbers. +| `Lt` | 1 | Input is less than the value. Both the input and value must be integer numbers. +| `GtLt` | 2 | Input is between two values. Both the input and value must be integer numbers. +| `IsTrue` | 0 | Input is equal to "true" +| `IsFalse` | 0 | Input is equal "false" + +The `value` field of MatchExpression is a list of string arguments to the +operator. + +The behavior of MatchExpression depends on the [feature type](#feature-types): +for *flag* and *attribute* features the MatchExpression operates on the feature +element whose name matches the ``. However, for *instance* features all +MatchExpressions are evaluated against the attributes of each instance +separately. + +A special case of an empty `matchExpressions` field matches everything, i.e. +matches/returns all elements of the feature. This makes it possible to write +[templates](#templating) that run over all discovered features. + +#### MatchAny + +The `.matchAny` field is a list of of [`matchFeatures`](#matchfeatures) +matchers. A logical OR is applied over the matchers, i.e. at least one of them +must match in order for the rule to trigger. + +Consider the following example: + +```yaml + matchAny: + - matchFeatures: + - feature: kernel.loadedmodule + matchExpressions: + kmod-1: {op: Exists} + - feature: pci.device + matchExpressions: + vendor: {op: In, value: ["0eee"]} + class: {op: In, value: ["0200"]} + - matchFeatures: + - feature: kernel.loadedmodule + matchExpressions: + kmod-2: {op: Exists} + - feature: pci.device + matchExpressions: + vendor: {op: In, value: ["0fff"]} + class: {op: In, value: ["0200"]} +``` + +This matches if kernel module kmod-1 is loaded and a network controller from +vendor 0eee is present, OR, if kernel module kmod-2 has been loaded and a +network controller from vendor 0fff is present (OR both of these conditions are +true). + +### Available features + +#### Feature types + +Features are divided into three different types: + +- **flag** features: a set of names without any associated values, e.g. CPUID + flags or loaded kernel modules +- **attribute** features: a set of names each of which has a single value + associated with it (essentially a map of key-value pairs), e.g. kernel config + flags or os release information +- **instance** features: a list of instances, each of which has multiple + attributes (key-value pairs of their own) associated with it, e.g. PCI or USB + devices + +#### List of features + +The following features are available for matching: + +| Feature | Feature type | Elements | Value type | Description +| ---------------- | ------------ | -------- | ---------- | ----------- +| **`cpu.cpuid`** | flag | | | Supported CPU capabilities +| | | **``** | | CPUID flag is present +| **`cpu.cstate`** | attribute | | | Status of cstates in the intel_idle cpuidle driver +| | | **`enabled`** | bool | 'true' if cstates are set, otherwise 'false'. Does not exist of intel_idle driver is not active. +| **`cpu.pstate`** | attribute | | | State of the Intel pstate driver. Does not exist if the driver is not enabled. +| | | **`status`** | string | Status of the driver, possible values are 'active' and 'passive' +| | | **`turbo`** | bool | 'true' if turbo frequencies are enabled, otherwise 'false' +| | | **`scaling`** | string | Active scaling_governor, possible values are 'powersave' or 'performance'. +| **`cpu.rdt`** | flag | | | Intel RDT capabilities supported by the system +| | | **``** | | RDT capability is supported, see [RDT flags](../get-started/features#intel-rdt-flags) for details +| **`cpu.sgx`** | attribute | | | Intel SGX (Software Guard Extensions) capabilities +| | | **`enabled`** | bool | `true` if Intel SGX has been enabled, otherwise does not exist +| **`cpu.sst`** | attribute | | | Intel SST (Speed Select Technology) capabilities +| | | **`bf.enabled`** | bool | `true` if Intel SST-BF (Intel Speed Select Technology - Base frequency) has been enabled, otherwise does not exist +| **`cpu.topology`** | attribute | | | CPU topology related features +| | | **`hardware_multithreading`** | bool | Hardware multithreading, such as Intel HTT, is enabled +| **`kernel.config`** | attribute | | | Kernel configuration options +| | | **``** | string | Value of the kconfig option +| **`kernel.loadedmodule`** | flag | | | Loaded kernel modules +| | | **`mod-name`** | | Kernel module `` is loaded +| **`kernel.selinux`** | attribute | | | Kernel SELinux related features +| | | **`enabled`** | bool | `true` if SELinux has been enabled and is in enforcing mode, otherwise `false` +| **`kernel.version`** | attribute | | | Kernel version information +| | | **`full`** | string | Full kernel version (e.g. ‘4.5.6-7-g123abcde') +| | | **`major`** | int | First component of the kernel version (e.g. ‘4') +| | | **`minor`** | int | Second component of the kernel version (e.g. ‘5') +| | | **`revision`** | int | Third component of the kernel version (e.g. ‘6') +| **`local.label`** | attribute | | | Features from hooks and feature files, i.e. labels from the [*local* feature source](#local-feature-source) +| | | **``** | string | Label `` created by the local feature source, value equals the value of the label +| **`memory.nv`** | instance | | | NVDIMM devices present in the system +| | | **``** | string | Value of the sysfs device attribute, available attributes: `devtype`, `mode` +| **`memory.numa`** | attribute | | | NUMA nodes +| | | **`is_numa`** | bool | `true` if NUMA architecture, `false` otherwise +| | | **`node_count`** | int | Number of NUMA nodes +| **`network.device`** | instance | | | Physical (non-virtual) network interfaces present in the system +| | | **``** | string | Sysfs network interface attribute, available attributes: `name`, `operstate`, `speed`, `sriov_numvfs`, `sriov_totalvfs` +| **`pci.device`** | instance | | | PCI devices present in the system +| | | **``** | string | Value of the sysfs device attribute, available attributes: `class`, `vendor`, `device`, `subsystem_vendor`, `subsystem_device`, `sriov_totalvfs` +| **`storage.device`** | instance | | | Block storage devices present in the system +| | | **``** | string | Sysfs network interface attribute, available attributes: `name`, `dax`, `rotational`, `nr_zones`, `zoned` +| **`system.osrelease`** | attribute | | | System identification data from `/etc/os-release` +| | | **``** | string | One parameter from `/etc/os-release` +| **`system.name`** | attribute | | | System name information +| | | **`nodename`** | string | Name of the kubernetes node object +| **`usb.device`** | instance | | | USB devices present in the system +| | | **``** | string | Value of the sysfs device attribute, available attributes: `class`, `vendor`, `device`, `serial` +| **`rule.matched`** | attribute | | | Previously matched rules +| | | **``** | string | Label or var from a preceding rule that matched + +### Templating + +Rules support template-based creation of labels and vars with the +`.labelsTemplate` and `.varsTemplate` fields. These makes it possible to +dynamically generate labels and vars based on the features that matched. + +The template must expand into a simple format with `=` pairs +separated by newline. + +Consider the following example: + + +```yaml + labelsTemplate: | + {{ range .pci.device }}vendor-{{ .class }}-{{ .device }}.present=true + {{ end }} + matchFeatures: + - feature: pci.device + matchExpressions: + class: {op: InRegexp, value: ["^02"]} + vendor: ["0fff"] +``` + + +The rule above will create individual labels +`feature.node.kubernetes.io/vendor--.present=true` for +each network controller device (device class starting with 02) from vendor +0ffff. + +All the matched features of each feature matcher term under `matchFeatures` +fields are available for the template engine. Matched features can be +referenced with `{%raw%}{{ . }}{%endraw%}` in the template, and +the available data could be described in yaml as follows: + +```yaml +. + : + - Name: + - ... + + : + - Name: + Value: + - ... + + : + - : + : + ... + - ... +``` + +That is, the per-feature data is a list of objects whose data fields depend on +the type of the feature: + +- for *flag* features only 'Name' is available +- for *value* features 'Name' and 'Value' are available +- for *instance* features all attributes of the matched instance are available + +A simple example of a template utilizing name and value from an *attribute* +feature: + + +```yaml + labelsTemplate: | + {{ range .system.osrelease }}system-{{ .Name }}={{ .Value }} + {{ end }} + matchFeatures: + - feature: system.osRelease + matchExpressions: + ID: {op: Exists} + VERSION_ID.major: {op: Exists} +``` + + +**NOTE** In case of matchAny is specified, the template is executed separately +against each individual `matchFeatures` field and the final set of labels will +be superset of all these separate template expansions. E.g. consider the +following: + +```yaml + - name: + labelsTemplate: