</code></pre></div></div><p>The object targets node named <codeclass="language-plaintext highlighter-rouge notranslate">node-1</code>. It lists two "flag type" features under the <codeclass="language-plaintext highlighter-rouge notranslate">vendor.flags</code> domain, two "attribute type" features and under the <codeclass="language-plaintext highlighter-rouge notranslate">vendor.config</code> domain and two "instance type" features under the <codeclass="language-plaintext highlighter-rouge notranslate">vendor.devices</code> domain. These features will not be directly affecting the node labels but they will be used as input when the <ahref="#nodefeaturerule-custom-resource"><codeclass="language-plaintext highlighter-rouge notranslate">NodeFeatureRule</code></a> objects are evaluated.</p><p>In addition, the example requests directly the <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubenernetes.io/vendor-feature.enabled=true</code> node label to be created.</p><p>The <codeclass="language-plaintext highlighter-rouge notranslate">nfd.node.kubernetes.io/node-name=<node-name></code> must be in place for each NodeFeature object as NFD uses it to determine the node which it is targeting.</p><h3id="feature-types">Feature types</h3><p>Features are divided into three different types:</p><ul><li><strong>flag</strong> features: a set of names without any associated values, e.g. CPUID flags or loaded kernel modules</li><li><strong>attribute</strong> features: a set of names each of which has a single value associated with it (essentially a map of key-value pairs), e.g. kernel config flags or os release information</li><li><strong>instance</strong> features: a list of instances, each of which has multiple attributes (key-value pairs of their own) associated with it, e.g. PCI or USB devices</li></ul><h2id="nodefeaturerule-custom-resource">NodeFeatureRule custom resource</h2><p><codeclass="language-plaintext highlighter-rouge notranslate">NodeFeatureRule</code> objects provide an easy way to create vendor or application specific labels and taints. It uses a flexible rule-based mechanism for creating labels and optionally taints based on node features.</p><h3id="a-nodefeaturerule-example">A NodeFeatureRule example</h3><p>Consider the following referential example:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">apiVersion</span><spanclass="pi">:</span><spanclass="s">nfd.k8s-sigs.io/v1alpha1</span>
</code></pre></div></div><p>It specifies one rule which creates node label <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubenernetes.io/my-sample-feature=true</code> if both of the following conditions are true (<codeclass="language-plaintext highlighter-rouge notranslate">matchFeatures</code> implements a logical AND over the matchers):</p><ul><li>The <codeclass="language-plaintext highlighter-rouge notranslate">dummy</code> network driver module has been loaded</li><li>X86 option in kernel config is set to <codeclass="language-plaintext highlighter-rouge notranslate">=y</code></li></ul><p>Create a <codeclass="language-plaintext highlighter-rouge notranslate">NodeFeatureRule</code> with a yaml file:</p><divclass="language-bash highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code>kubectl apply <spanclass="nt">-f</span> https://raw.githubusercontent.com/kubernetes-sigs/node-feature-discovery/master/examples/nodefeaturerule.yaml
</code></pre></div></div><p>Now, on X86 platforms the feature label appears after doing <codeclass="language-plaintext highlighter-rouge notranslate">modprobe dummy</code> on a system and correspondingly the label is removed after <codeclass="language-plaintext highlighter-rouge notranslate">rmmod dummy</code>. Note a re-labeling delay up to the sleep-interval of nfd-worker (1 minute by default).</p><p>See <ahref="#label-rule-format">Label rule format</a> for detailed description of available fields and how to write labeling rules.</p><h3id="nodefeaturerule-tainting-feature">NodeFeatureRule tainting feature</h3><p>This feature is experimental.</p><p>In some circumstances, it is desirable to keep nodes with specialized hardware away from running general workload and instead leave them for workloads that need the specialized hardware. One way to achieve it is to taint the nodes with the specialized hardware and add corresponding toleration to pods that require the special hardware. NFD offers node tainting functionality which is disabled by default. User can define one or more custom taints via the <codeclass="language-plaintext highlighter-rouge notranslate">taints</code> field of the NodeFeatureRule CR. The same rule-based mechanism is applied here and the NFD taints only rule matching nodes.</p><p>To enable the tainting feature, <codeclass="language-plaintext highlighter-rouge notranslate">--enable-taints</code> flag needs to be set to <codeclass="language-plaintext highlighter-rouge notranslate">true</code>. If the flag <codeclass="language-plaintext highlighter-rouge notranslate">--enable-taints</code> is set to <codeclass="language-plaintext highlighter-rouge notranslate">false</code> (i.e. disabled), taints defined in the NodeFeatureRule CR have no effect and will be ignored by the NFD master.</p><blockquote><p><strong>NOTE:</strong> Before enabling any taints, make sure to edit nfd-worker daemonset to tolerate the taints to be created. Otherwise, already running pods that do not tolerate the taint are evicted immediately from the node including the nfd-worker pod.</p></blockquote><p>Example NodeFeatureRule with custom taints:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">apiVersion</span><spanclass="pi">:</span><spanclass="s">nfd.k8s-sigs.io/v1alpha1</span>
</code></pre></div></div><p>In this example, if the <codeclass="language-plaintext highlighter-rouge notranslate">my sample taint rule</code> rule is matched, <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/pci-0300_1d0f.present=true:NoExecute</code> and <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/cpu-cpuid.ADX:NoExecute</code> taints are set on the node.</p><p>There are some limitations to the namespace part (i.e. prefix/) of the taint key:</p><ul><li><codeclass="language-plaintext highlighter-rouge notranslate">kubernetes.io/</code> and its sub-namespaces (like <codeclass="language-plaintext highlighter-rouge notranslate">sub.ns.kubernetes.io/</code>) cannot generally be used</li><li>the only exception is <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/</code> and its sub-namespaces (like <codeclass="language-plaintext highlighter-rouge notranslate">sub.ns.feature.node.kubernetes.io</code>)</li><li>unprefixed keys (like <codeclass="language-plaintext highlighter-rouge notranslate">foo</code>) keys are disallowed</li></ul><h2id="local-feature-source">Local feature source</h2><p>NFD-Worker has a special feature source named <codeclass="language-plaintext highlighter-rouge notranslate">local</code> which is an integration point for external feature detectors. It provides a mechanism for pluggable extensions, allowing the creation of new user-specific features and even overriding built-in labels.</p><p>The <codeclass="language-plaintext highlighter-rouge notranslate">local</code> feature source has two methods for detecting features, feature files and hooks (deprecated). The features discovered by the <codeclass="language-plaintext highlighter-rouge notranslate">local</code> source can further be used in label rules specified in <ahref="#nodefeaturerule-custom-resource"><codeclass="language-plaintext highlighter-rouge notranslate">NodeFeatureRule</code></a> objects and the <ahref="#custom-feature-source"><codeclass="language-plaintext highlighter-rouge notranslate">custom</code></a> feature source.</p><blockquote><p><strong>NOTE:</strong> Be careful when creating and/or updating hook or feature files while NFD is running. In order to avoid race conditions you should write into a temporary file, and atomically create/update the original file by doing a file rename operation. NFD ignores dot files, so temporary file can be written to the same directory and renamed (<codeclass="language-plaintext highlighter-rouge notranslate">.my.feature</code> -><codeclass="language-plaintext highlighter-rouge notranslate">my.feature</code>) once file is complete. Both file names should (obviously) be unique for the given application.</p></blockquote><h3id="an-example">An example</h3><p>Consider a plaintext file <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/features.d/my-features</code> having the following contents (or alternatively a shell script <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/source.d/my-hook.sh</code> having the following stdout output):</p><divclass="language-plaintext highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code>my-feature.1
</code></pre></div></div><p>This will translate into the following node labels:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="s">feature.node.kubernetes.io/my-feature.1</span><spanclass="pi">:</span><spanclass="s2">"</span><spanclass="s">true"</span>
</code></pre></div></div><h3id="feature-files">Feature files</h3><p>The <codeclass="language-plaintext highlighter-rouge notranslate">local</code> source reads files found in <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/features.d/</code>. File content is parsed and translated into node labels, see the <ahref="#input-format">input format below</a>.</p><h3id="hooks">Hooks</h3><p><strong>DEPRECATED</strong> The <codeclass="language-plaintext highlighter-rouge notranslate">local</code> source executes hooks found in <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/source.d/</code>. The hook files must be executable and they are supposed to print all discovered features in <codeclass="language-plaintext highlighter-rouge notranslate">stdout</code>. Since NFD v0.13 the default container image only supports statically linked ELF binaries.</p><p><codeclass="language-plaintext highlighter-rouge notranslate">stderr</code> output of hooks is propagated to NFD log so it can be used for debugging and logging.</p><p>NFD tries to execute any regular files found from the hooks directory. Any additional data files the hook might need (e.g. a configuration file) should be placed in a separate directory in order to avoid NFD unnecessarily trying to execute them. A subdirectory under the hooks directory can be used, for example <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/source.d/conf/</code>.</p><blockquote><p><strong>NOTE:</strong> Hooks are being DEPRECATED and will be removed in a future release. Starting from release v0.14 hooks are disabled by default and can be enabled via <codeclass="language-plaintext highlighter-rouge notranslate">sources.local.hooksEnabled</code> field in the worker configuration.</p></blockquote><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">sources</span><spanclass="pi">:</span>
</code></pre></div></div><blockquote><p><strong>NOTE:</strong> NFD will blindly run any executables placed/mounted in the hooks directory. It is the user's responsibility to review the hooks for e.g. possible security implications.</p><p><strong>NOTE:</strong> The <ahref="/node-feature-discovery/master/deployment/image-variants.html#full">full</a> image variant provides backwards-compatibility with older NFD versions by including a more expanded environment, supporting bash and perl runtimes.</p></blockquote><h3id="input-format">Input format</h3><p>The hook stdout and feature files are expected to contain features in simple key-value pairs, separated by newlines:</p><divclass="language-plaintext highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code># This is a comment
</code></pre></div></div><p>The label value defaults to <codeclass="language-plaintext highlighter-rouge notranslate">true</code>, if not specified.</p><p>Label namespace may be specified with <codeclass="language-plaintext highlighter-rouge notranslate"><namespace>/<name>[=<value>]</code>.</p><blockquote><p><strong>NOTE:</strong> The feature file size limit it 64kB. The feature file will be ignored if the size limit is exceeded.</p></blockquote><p>Comment lines (starting with <codeclass="language-plaintext highlighter-rouge notranslate">#</code>) are ignored.</p><p>Adding following line anywhere to feature file defines date when its content expires / is ignored:</p><divclass="language-plaintext highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code># +expiry-time=2023-07-29T11:22:33Z
</code></pre></div></div><p>Also, the expiry-time value would stay the same during the processing of the feature file until another expiry-time directive is encountered. Considering the following file:</p><divclass="language-plaintext highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code># +expiry-time=2012-07-28T11:22:33Z
</code></pre></div></div><p>After processing the above file, only <codeclass="language-plaintext highlighter-rouge notranslate">featureKey2</code> and <codeclass="language-plaintext highlighter-rouge notranslate">featureKey3</code> would be included in the list of accepted features.</p><blockquote><p><strong>NOTE:</strong> The time format that we are supporting is RFC3339. Also, the <codeclass="language-plaintext highlighter-rouge notranslate">expiry-time</code> tag is only evaluated in each re-discovery period, and the expiration of node labels is not tracked.</p></blockquote><p>To exclude specific features from the <codeclass="language-plaintext highlighter-rouge notranslate">local.feature</code> Feature, you can use the <codeclass="language-plaintext highlighter-rouge notranslate"># +no-feature</code> directive. The <codeclass="language-plaintext highlighter-rouge notranslate"># +no-label</code> directive causes the feature to be excluded from the <codeclass="language-plaintext highlighter-rouge notranslate">local.label</code> Feature and a node label not to be generated.</p><p>Considering the following file:</p><divclass="language-plaintext highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code># +no-feature
label-only=value
my-feature=value
foo=bar
# +no-label
foo=baz
</code></pre></div></div><p>Processing the above file would result in the following Features:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="s">local.features</span><spanclass="pi">:</span>
</code></pre></div></div><p>and the following labels added to the Node:</p><divclass="language-plaintext highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code>feature.node.kubernetes.io/label-only=value
feature.node.kubernetes.io/my-feature=value
</code></pre></div></div><h3id="mounts">Mounts</h3><p>The standard NFD deployments contain <codeclass="language-plaintext highlighter-rouge notranslate">hostPath</code> mounts for <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/source.d/</code> and <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/features.d/</code>, making these directories from the host available inside the nfd-worker container.</p><h4id="injecting-labels-from-other-pods">Injecting labels from other pods</h4><p>One use case for the feature files and hooks is detecting features in other Pods outside NFD, e.g. in Kubernetes device plugins. By using the same <codeclass="language-plaintext highlighter-rouge notranslate">hostPath</code> mounts for <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/source.d/</code> and <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/features.d/</code> in the side-car (e.g. device plugin) creates a shared area for deploying feature files and hooks to NFD. NFD periodically scans the directories and reads any feature files and runs any hooks it finds.</p><h2id="custom-feature-source">Custom feature source</h2><p>The <codeclass="language-plaintext highlighter-rouge notranslate">custom</code> feature source in nfd-worker provides a rule-based mechanism for label creation, similar to the <ahref="#nodefeaturerule-custom-resource"><codeclass="language-plaintext highlighter-rouge notranslate">NodeFeatureRule</code></a> objects. The difference is that the rules are specified in the worker configuration instead of a Kubernetes API object.</p><p>See <ahref="/node-feature-discovery/master/usage/nfd-worker.html#worker-configuration">worker configuration</a> for instructions how to set-up and manage the worker configuration.</p><h3id="an-example-custom-feature-source-configuration">An example custom feature source configuration</h3><p>Consider the following referential configuration for nfd-worker:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">core</span><spanclass="pi">:</span>
</code></pre></div></div><p>It specifies one rule which creates node label <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubenernetes.io/my-sample-feature=true</code> if both of the following conditions are true (<codeclass="language-plaintext highlighter-rouge notranslate">matchFeatures</code> implements a logical AND over the matchers):</p><ul><li>The <codeclass="language-plaintext highlighter-rouge notranslate">dummy</code> network driver module has been loaded</li><li>X86 option in kernel config is set to <codeclass="language-plaintext highlighter-rouge notranslate">=y</code></li></ul><p>In addition, the configuration only enables the <codeclass="language-plaintext highlighter-rouge notranslate">custom</code> source, disabling all built-in labels.</p><p>Now, on X86 platforms the feature label appears after doing <codeclass="language-plaintext highlighter-rouge notranslate">modprobe dummy</code> on a system and correspondingly the label is removed after <codeclass="language-plaintext highlighter-rouge notranslate">rmmod dummy</code>. Note a re-labeling delay up to the sleep-interval of nfd-worker (1 minute by default).</p><h3id="additional-configuration-directory">Additional configuration directory</h3><p>In addition to the rules defined in the nfd-worker configuration file, the <codeclass="language-plaintext highlighter-rouge notranslate">custom</code> feature source can read more configuration files located in the <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/custom.d/</code> directory. This makes more dynamic and flexible configuration easier.</p><p>As an example, consider having file <codeclass="language-plaintext highlighter-rouge notranslate">/etc/kubernetes/node-feature-discovery/custom.d/my-rule.yaml</code> with the following content:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="pi">-</span><spanclass="na">name</span><spanclass="pi">:</span><spanclass="s2">"</span><spanclass="s">my</span><spanclass="nv"></span><spanclass="s">e1000</span><spanclass="nv"></span><spanclass="s">rule"</span>
</code></pre></div></div><p>This simple rule will create <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubenernetes.io/e1000.present=true</code> label if the <codeclass="language-plaintext highlighter-rouge notranslate">e1000</code> kernel module has been loaded.</p><p>The <ahref="https://github.com/kubernetes-sigs/node-feature-discovery/blob/master/deployment/overlays/samples/custom-rules"><codeclass="language-plaintext highlighter-rouge notranslate">samples/custom-rules</code></a> kustomize overlay sample contains an example for deploying a custom rule from a ConfigMap.</p><h2id="node-labels">Node labels</h2><p>Feature labels have the following format:</p><divclass="language-plaintext highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><namespace>/<name> = <value>
</code></pre></div></div><p>The namespace part (i.e. prefix) of the labels is controlled by nfd:</p><ul><li>All built-in labels use <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io</code>. This is also the default for user defined features that don't specify any namespace.</li><li>Namespaces may be excluded with the <ahref="/node-feature-discovery/master/reference/master-commandline-reference.html#-deny-label-ns"><codeclass="language-plaintext highlighter-rouge notranslate">-deny-label-ns</code></a> command line flag of nfd-master <ul><li>To allow specific namespaces that were denied, you can use <ahref="/node-feature-discovery/master/reference/master-commandline-reference.html#-extra-label-ns"><codeclass="language-plaintext highlighter-rouge notranslate">-extra-label-ns</code></a> command line flag of nfd-master. e.g: <codeclass="language-plaintext highlighter-rouge notranslate">nfd-master -deny-label-ns="*" -extra-label-ns=example.com</code></li></ul></li></ul><h2id="label-rule-format">Label rule format</h2><p>This section describes the rule format used in <ahref="#nodefeaturerule-custom-resource"><codeclass="language-plaintext highlighter-rouge notranslate">NodeFeatureRule</code></a> objects and in the configuration of the <ahref="#custom-feature-source"><codeclass="language-plaintext highlighter-rouge notranslate">custom</code></a> feature source.</p><p>It is based on a generic feature matcher that covers all features discovered by nfd-worker. The rules rely on a unified data model of the available features and a generic expression-based format. Features that can be used in the rules are described in detail in <ahref="#available-features">available features</a> below.</p><p>Take this rule as a referential example:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="pi">-</span><spanclass="na">name</span><spanclass="pi">:</span><spanclass="s2">"</span><spanclass="s">my</span><spanclass="nv"></span><spanclass="s">feature</span><spanclass="nv"></span><spanclass="s">rule"</span>
</code></pre></div></div><p>This will yield <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubenernetes.io/my-special-feature=my-value</code> node label if all of these are true (<codeclass="language-plaintext highlighter-rouge notranslate">matchFeatures</code> implements a logical AND over the matchers):</p><ul><li>the CPU has AVX512F capability</li><li>kernel version is 5.2 or later (must be v5.x)</li><li>an Intel network controller is present</li></ul><h3id="fields">Fields</h3><h4id="name">Name</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.name</code> field is required and used as an identifier of the rule.</p><h4id="labels">Labels</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.labels</code> is a map of the node labels to create if the rule matches.</p><p>Take this rule as a referential example:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">apiVersion</span><spanclass="pi">:</span><spanclass="s">nfd.k8s-sigs.io/v1alpha1</span>
</code></pre></div></div><p>Label <codeclass="language-plaintext highlighter-rouge notranslate">linux-lsm-enabled</code> uses the <codeclass="language-plaintext highlighter-rouge notranslate">@</code> notation for dynamic values. The value of the label will be the value of the attribute <codeclass="language-plaintext highlighter-rouge notranslate">LSM</code> of the feature <codeclass="language-plaintext highlighter-rouge notranslate">kernel.config</code>.</p><p>The <codeclass="language-plaintext highlighter-rouge notranslate">@<feature-name>.<element-name></code> format can be used to inject values of detected features to the label. See <ahref="#available-features">available features</a> for possible values to use.</p><p>This will yield into the following node label:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">labels</span><spanclass="pi">:</span>
</code></pre></div></div><h4id="labels-template">Labels template</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.labelsTemplate</code> field specifies a text template for dynamically creating labels based on the matched features. See <ahref="#templating">templating</a> for details.</p><blockquote><p><strong>NOTE:</strong> The <codeclass="language-plaintext highlighter-rouge notranslate">labels</code> field has priority over <codeclass="language-plaintext highlighter-rouge notranslate">labelsTemplate</code>, i.e. labels specified in the <codeclass="language-plaintext highlighter-rouge notranslate">labels</code> field will override anything originating from <codeclass="language-plaintext highlighter-rouge notranslate">labelsTemplate</code>.</p></blockquote><h4id="node-annotations">Node Annotations</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.annotations</code> field is a list of features to be advertised as annotations.</p><p>Take this rule as a referential example:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">apiVersion</span><spanclass="pi">:</span><spanclass="s">nfd.k8s-sigs.io/v1alpha1</span>
</code></pre></div></div><p>This will yield into the following node annotations:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">annotations</span><spanclass="pi">:</span>
</code></pre></div></div><p>NFD enforces some limitations to the namespace (or prefix)/ of the annotations:</p><ul><li><codeclass="language-plaintext highlighter-rouge notranslate">kubernetes.io/</code> and its sub-namespaces (like <codeclass="language-plaintext highlighter-rouge notranslate">sub.ns.kubernetes.io/</code>) cannot generally be used</li><li>the only exception is <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/</code> and its sub-namespaces (like <codeclass="language-plaintext highlighter-rouge notranslate">sub.ns.feature.node.kubernetes.io</code>)</li><li>unprefixed names will get prefixed with <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/</code> automatically (e.g. <codeclass="language-plaintext highlighter-rouge notranslate">foo</code> becomes <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/foo</code>)</li></ul><blockquote><p><strong>NOTE:</strong> The <codeclass="language-plaintext highlighter-rouge notranslate">annotations</code> field has will only advertise features via node annotations the features won't be advertised as node labels unless they are specified in the <codeclass="language-plaintext highlighter-rouge notranslate">labels</code> field.</p></blockquote><h4id="taints">Taints</h4><p><em>taints</em> is a list of taint entries and each entry can have <codeclass="language-plaintext highlighter-rouge notranslate">key</code>, <codeclass="language-plaintext highlighter-rouge notranslate">value</code> and <codeclass="language-plaintext highlighter-rouge notranslate">effect</code>, where the <codeclass="language-plaintext highlighter-rouge notranslate">value</code> is optional. Effect could be <codeclass="language-plaintext highlighter-rouge notranslate">NoSchedule</code>, <codeclass="language-plaintext highlighter-rouge notranslate">PreferNoSchedule</code> or <codeclass="language-plaintext highlighter-rouge notranslate">NoExecute</code>. To learn more about the meaning of these effects, check out k8s <ahref="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/">documentation</a>.</p><blockquote><p><strong>NOTE:</strong> taints field is not available for the custom rules of nfd-worker and only for NodeFeatureRule objects.</p></blockquote><h4id="vars">Vars</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.vars</code> field is a map of values (key-value pairs) to store for subsequent rules to use. In other words, these are variables that are not advertised as node labels. See <ahref="#backreferences">backreferences</a> for more details on the usage of vars.</p><h4id="extended-resources">Extended resources</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.extendedResources</code> field is a list of extended resources to advertise. See <ahref="#extended-resources">extended resources</a> for more details.</p><p>Take this rule as a referential example:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">apiVersion</span><spanclass="pi">:</span><spanclass="s">nfd.k8s-sigs.io/v1alpha1</span>
</code></pre></div></div><p>The extended resource <codeclass="language-plaintext highlighter-rouge notranslate">vendor.io/dynamic</code> is defined in the form <codeclass="language-plaintext highlighter-rouge notranslate">@feature.attribute</code>. The value of the extended resource will be the value of the attribute <codeclass="language-plaintext highlighter-rouge notranslate">major</code> of the feature <codeclass="language-plaintext highlighter-rouge notranslate">kernel.version</code>.</p><p>The <codeclass="language-plaintext highlighter-rouge notranslate">@<feature-name>.<element-name></code> format can be used to inject values of detected features to the extended resource. See <ahref="#available-features">available features</a> for possible values to use. Note that the value must be eligible as a Kubernetes resource quantity.</p><p>This will yield into the following node status:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">allocatable</span><spanclass="pi">:</span>
</code></pre></div></div><p>There are some limitations to the namespace part (i.e. prefix)/ of the Extended Resources names:</p><ul><li><codeclass="language-plaintext highlighter-rouge notranslate">kubernetes.io/</code> and its sub-namespaces (like <codeclass="language-plaintext highlighter-rouge notranslate">sub.ns.kubernetes.io/</code>) cannot generally be used</li><li>the only exception is <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/</code> and its sub-namespaces (like <codeclass="language-plaintext highlighter-rouge notranslate">sub.ns.feature.node.kubernetes.io</code>)</li><li>unprefixed names will get prefixed with <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/</code> automatically (e.g. <codeclass="language-plaintext highlighter-rouge notranslate">foo</code> becomes <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/foo</code>)</li></ul><h4id="vars-template">Vars template</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.varsTemplate</code> field specifies a text template for dynamically creating vars based on the matched features. See <ahref="#templating">templating</a> for details on using templates and <ahref="#backreferences">backreferences</a> for more details on the usage of vars.</p><blockquote><p><strong>NOTE:</strong> The <codeclass="language-plaintext highlighter-rouge notranslate">vars</code> field has priority over <codeclass="language-plaintext highlighter-rouge notranslate">varsTemplate</code>, i.e. vars specified in the <codeclass="language-plaintext highlighter-rouge notranslate">vars</code> field will override anything originating from <codeclass="language-plaintext highlighter-rouge notranslate">varsTemplate</code>.</p></blockquote><h4id="matchfeatures">MatchFeatures</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.matchFeatures</code> field specifies a feature matcher, consisting of a list of feature matcher terms. It implements a logical AND over the terms i.e. all of them must match in order for the rule to trigger.</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">matchFeatures</span><spanclass="pi">:</span>
</code></pre></div></div><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.matchFeatures[].feature</code> field specifies the feature against which to match.</p><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.matchFeatures[].matchExpressions</code> field specifies a map of expressions which to evaluate against the elements of the feature.</p><p>In each MatchExpression <codeclass="language-plaintext highlighter-rouge notranslate">op</code> specifies the operator to apply. Valid values are described below.</p><table><thead><tr><th>Operator</th><th>Number of values</th><th>Matches when</th></tr></thead><tbody><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">In</code></td><td>1 or greater</td><td>Input is equal to one of the values</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">NotIn</code></td><td>1 or greater</td><td>Input is not equal to any of the values</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">InRegexp</code></td><td>1 or greater</td><td>Values of the MatchExpression are treated as regexps and input matches one or more of them</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">Exists</code></td><td>0</td><td>The key exists</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">DoesNotExist</code></td><td>0</td><td>The key does not exists</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">Gt</code></td><td>1</td><td>Input is greater than the value. Both the input and value must be integer numbers.</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">Lt</code></td><td>1</td><td>Input is less than the value. Both the input and value must be integer numbers.</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">GtLt</code></td><td>2</td><td>Input is between two values. Both the input and value must be integer numbers.</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">IsTrue</code></td><td>0</td><td>Input is equal to "true"</td></tr><tr><td><codeclass="language-plaintext highlighter-rouge notranslate">IsFalse</code></td><td>0</td><td>Input is equal "false"</td></tr></tbody></table><p>The <codeclass="language-plaintext highlighter-rouge notranslate">value</code> field of MatchExpression is a list of string arguments to the operator.</p><p>The behavior of MatchExpression depends on the <ahref="#feature-types">feature type</a>: for <em>flag</em> and <em>attribute</em> features the MatchExpression operates on the feature element whose name matches the <codeclass="language-plaintext highlighter-rouge notranslate"><key></code>. However, for <em>instance</em> features all MatchExpressions are evaluated against the attributes of each instance separately.</p><h4id="matchany">MatchAny</h4><p>The <codeclass="language-plaintext highlighter-rouge notranslate">.matchAny</code> field is a list of of <ahref="#matchfeatures"><codeclass="language-plaintext highlighter-rouge notranslate">matchFeatures</code></a> matchers. A logical OR is applied over the matchers, i.e. at least one of them must match in order for the rule to trigger.</p><p>Consider the following example:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">matchAny</span><spanclass="pi">:</span>
</code></pre></div></div><p>This matches if kernel module kmod-1 is loaded and a network controller from vendor 0eee is present, OR, if kernel module kmod-2 has been loaded and a network controller from vendor 0fff is present (OR both of these conditions are true).</p><h3id="available-features">Available features</h3><p>The following features are available for matching:</p><table><thead><tr><th>Feature</th><th><ahref="#feature-types">Feature type</a></th><th>Elements</th><th>Value type</th><th>Description</th></tr></thead><tbody><tr><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">cpu.cpuid</code></strong></td><td>flag</td><td></td><td></td><td>Supported CPU capabilities</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate"><cpuid-flag></code></strong></td><td></td><td>CPUID flag is present</td></tr><tr><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">cpu.cstate</code></strong></td><td>attribute</td><td></td><td></td><td>Status of cstates in the intel_idle cpuidle driver</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">enabled</code></strong></td><td>bool</td><td>‘true' if cstates are set, otherwise ‘false'. Does not exist of intel_idle driver is not active.</td></tr><tr><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">cpu.model</code></strong></td><td>attribute</td><td></td><td></td><td>CPU model related attributes</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">family</code></strong></td><td>int</td><td>CPU family</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">vendor_id</code></strong></td><td>string</td><td>CPU vendor ID</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">id</code></strong></td><td>int</td><td>CPU model ID</td></tr><tr><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">cpu.pstate</code></strong></td><td>attribute</td><td></td><td></td><td>State of the Intel pstate driver. Does not exist if the driver is not enabled.</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">status</code></strong></td><td>string</td><td>Status of the driver, possible values are ‘active' and ‘passive'</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">turbo</code></strong></td><td>bool</td><td>‘true' if turbo frequencies are enabled, otherwise ‘false'</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">scaling</code></strong></td><td>string</td><td>Active scaling_governor, possible values are ‘powersave' or ‘performance'.</td></tr><tr><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">cpu.rdt</code></strong></td><td>attribute</td><td></td><td></td><td>Intel RDT capabilities supported by the system</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate"><rdt-flag></code></strong></td><td></td><td>RDT capability is supported, see <ahref="#intel-rdt-flags">RDT flags</a> for details</td></tr><tr><td></td><td></td><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">RDTL3CA_NUM_CLOSID</code></strong></td><td>int</td><td>The number or available CLOSID (Class of service ID) for Intel L3 Cache Allocation Technology</td></tr><tr><td><strong><codeclass="language-plaintext highlighter-rouge notranslate">cpu.security</code></strong></td><td>attribute</td><td></td><td></td><td>Features related to security and trusted execution environments</td></tr><tr><td></td><td></td><td><str
</code></pre></div></div><!----><p>The rule above will create individual labels <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/vendor-<class-id>-<device-id>.present=true</code> for each network controller device (device class starting with 02) from vendor 0ffff.</p><p>All the matched features of each feature matcher term under <codeclass="language-plaintext highlighter-rouge notranslate">matchFeatures</code> fields are available for the template engine. Matched features can be referenced with <codeclass="language-plaintext highlighter-rouge notranslate">{{ .<feature-name> }}</code> in the template, and the available data could be described in yaml as follows:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="s">.</span>
</code></pre></div></div><p>That is, the per-feature data is a list of objects whose data fields depend on the type of the feature:</p><ul><li>for <em>flag</em> features only ‘Name' is available</li><li>for <em>value</em> features ‘Name' and ‘Value' are available</li><li>for <em>instance</em> features all attributes of the matched instance are available</li></ul><p>A simple example of a template utilizing name and value from an <em>attribute</em> feature: <!----></p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">labelsTemplate</span><spanclass="pi">:</span><spanclass="pi">|</span>
<spanclass="s">{{ range .system.osrelease }}system-{{ .Name }}={{ .Value }}</span>
</code></pre></div></div><!----><blockquote><p><strong>NOTE:</strong> In case of matchAny is specified, the template is executed separately against each individual <codeclass="language-plaintext highlighter-rouge notranslate">matchFeatures</code> field and the final set of labels will be superset of all these separate template expansions. E.g. consider the following:</p></blockquote><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="pi">-</span><spanclass="na">name</span><spanclass="pi">:</span><spanclass="s"><name></span>
</code></pre></div></div><p>In the example above (assuming the overall result is a match) the template would be executed on matcher#1 as well as on matcher#2 and/or matcher#3 (depending on whether both or only one of them match). All the labels from these separate expansions would be created, i.e. the end result would be a union of all the individual expansions.</p><p>Rule templates use the Golang <ahref="https://pkg.go.dev/text/template">text/template</a> package and all its built-in functionality (e.g. pipelines and functions) can be used. An example template taking use of the built-in <codeclass="language-plaintext highlighter-rouge notranslate">len</code> function, advertising the number of PCI network controllers from a specific vendor: <!----></p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="na">labelsTemplate</span><spanclass="pi">:</span><spanclass="pi">|</span>
</code></pre></div></div><!----><p>Imaginative template pipelines are possible, but care must be taken in order to produce understandable and maintainable rule sets.</p><h3id="backreferences">Backreferences</h3><p>Rules support referencing the output of preceding rules. This enables sophisticated scenarios where multiple rules are combined together to for more complex heuristics than a single rule can provide. The labels and vars created by the execution of preceding rules are available as a special <codeclass="language-plaintext highlighter-rouge notranslate">rule.matched</code> feature.</p><p>Consider the following configuration:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="pi">-</span><spanclass="na">name</span><spanclass="pi">:</span><spanclass="s2">"</span><spanclass="s">my</span><spanclass="nv"></span><spanclass="s">kernel</span><spanclass="nv"></span><spanclass="s">label</span><spanclass="nv"></span><spanclass="s">rule"</span>
</code></pre></div></div><p>The <codeclass="language-plaintext highlighter-rouge notranslate">feature.node.kubernetes.io/high-level-feature = true</code> label depends on the two previous rules.</p><p>Note that when referencing rules across multiple <ahref="#nodefeaturerule-custom-resource"><codeclass="language-plaintext highlighter-rouge notranslate">NodeFeatureRule</code></a> objects attention must be paid to the ordering. <codeclass="language-plaintext highlighter-rouge notranslate">NodeFeatureRule</code> objects are processed in alphabetical order (based on their <codeclass="language-plaintext highlighter-rouge notranslate">.metadata.name</code>).</p><h3id="examples">Examples</h3><p>Some more configuration examples below.</p><p>Match certain CPUID features:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="pi">-</span><spanclass="na">name</span><spanclass="pi">:</span><spanclass="s2">"</span><spanclass="s">example</span><spanclass="nv"></span><spanclass="s">cpuid</span><spanclass="nv"></span><spanclass="s">rule"</span>
</code></pre></div></div><p>Require a certain loaded kernel module and OS version:</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="pi">-</span><spanclass="na">name</span><spanclass="pi">:</span><spanclass="s2">"</span><spanclass="s">my</span><spanclass="nv"></span><spanclass="s">multi-feature</span><spanclass="nv"></span><spanclass="s">rule"</span>
</code></pre></div></div><p>Require a loaded kernel module and two specific PCI devices (both of which must be present):</p><divclass="language-yaml highlighter-rouge notranslate"><divclass="highlight"><preclass="highlight"><code><spanclass="pi">-</span><spanclass="na">name</span><spanclass="pi">:</span><spanclass="s2">"</span><spanclass="s">my</span><spanclass="nv"></span><spanclass="s">multi-device</span><spanclass="nv"></span><spanclass="s">rule"</span>
