Commit bfbc47f55e added a lot of those and
this patch tries to cover all that we missed there. Having .md suffixes
in references to internal files makes it convenient to browse the
document locally, just as text files as the references work correctly.
This PR adds a config option for setting the NFD API controller resync period.
The resync period is only activated when the NodeFeature API has been
enabled (with -enable-nodefeature-api).
Signed-off-by: AhmedGrati <ahmedgrati1999@gmail.com>
This patch add SEV ASIDs and the related (but distinct) SEV Encrypted State
(SEV-ES) IDs as two quantities to be exposed via extended resources.
In a kernel built with CONFIG_CGROUP_MISC on a suitably equipped AMD CPU, the
root control group will have a misc.capacity file that shows the number of
available IDs in each category.
The added extended resources are:
- sev.asids
- sev.encrypted_state_ids
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
This PR adds the combination of dynamic and builtin kernel modules into
one feature called `kernel.enabledmodule`. It's a superset of the
`kernel.loadedmodule` feature.
Signed-off-by: AhmedGrati <ahmedgrati1999@gmail.com>
Mark the -resource-labels flag (and the corresponding resourceLabels
config option) as deprecated. We now support managing extended resources
via NodeFeatureRule objects. This kludge deserves to go, eventually.
Add support for management of Extended Resources via the
NodeFeatureRule CRD API.
There are usage scenarios where users want to advertise features
as extended resources instead of labels (or annotations).
This patch enables the discovery of extended resources, via annotation
and patch of node.status.capacity and node.status.allocatable. By using
the NodeFeatureRule API.
Co-authored-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Co-authored-by: Markus Lehtonen <markus.lehtonen@intel.com>
Co-authored-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Disallow taints having a key with "kubernetes.io/" or "*.kubernetes.io/"
prefix. This is a precaution to protect the user from messing up with
the "official" well-known taints from Kubernetes itself. The only
exception is that the "nfd.node.kubernetes.io/" prefix is allowed.
However, there is one allowed NFD-specific namespace (and its
sub-namespaces) i.e. "feature.node.kubernetes.io" under the
kubernetes.io domain that can be used for NFD-managed taints.
Also disallow unprefixed taint keys. We don't add a default prefix to
unprefixed taints (like we do for labels) from NodeFeatureRules. This is
to prevent unpleasant surprises to users that need to manage matching
tolerations for their workloads.
Document built-in RDT labels to be deprecated and removed in a future
release. The plan is that the default built-in RDT labels would not be
created anymore, but the RDT features would still be available for
NodeFeatureRules to consume.
The RDT labels are not very useful (they don't e.g indicate if the
features are really enabled in kernel or if the resctrlfs is mounted).
Similar to the nfd-worker, in this PR we want to support the
dynamic run-time configurability through a config file for the nfd-master.
We'll use a json or yaml configuration file along with the fsnotify in
order to watch for changes in the config file. As a result, we're
allowing dynamic control of logging params, allowed namespaces,
extended resources, label whitelisting, and denied namespaces.
Signed-off-by: AhmedGrati <ahmedgrati1999@gmail.com>
The total amount of keys that can be used on a specific TDX system is
exposed via the cgroups misc.capacity. See:
```
$ cat /sys/fs/cgroup/misc.capacity
tdx 31
```
The first step to properly manage the amount of keys present in a node
is exposing it via the NFD, and that's exactly what this commit does.
An example of how it ends up being exposed via the NFD:
```
$ kubectl get node 984fee00befb.jf.intel.com -o jsonpath='{.metadata.labels}' | jq | grep tdx.total_keys
"feature.node.kubernetes.io/cpu-security.tdx.total_keys": "31",
```
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Access to the kubelet state directory may raise concerns in some setups, added an option to disable it.
The feature is enabled by default.
Signed-off-by: Talor Itzhak <titzhak@redhat.com>
Bump cpuid version to v2.2.4 in the go.mod so that WRMSRNS (
Non-Serializing Write to Model Specific Register) and MSRLIST
(Read/Write List of Model Specific Registers) instructions are
detectable.
Signed-off-by: Muyassarov, Feruzjon <feruzjon.muyassarov@intel.com>
Make distroless/base as the base image for the default image,
effectively making the minimal image as the default. Add a new "full"
image variant that corresponds the previous default image. The
"*-minimal" container image tag is provided for backwards compatibility.
The practical user impact of this change is that hook support is limited
to statically linked ELF binaries. Bash or Perl scripts are not
supported by the default image, anymore, but the new "full" image
variant can be used for backwards compatibility.
Bump cpuid to v2.2.3 which adds support for detecting Intel Sierra
Forest instructions like AVXIFMA, AVXNECONVERT, AVXVNNIINT8 and
CMPCCXADD.
Signed-off-by: Muyassarov, Feruzjon <feruzjon.muyassarov@intel.com>
Document the usage of the NodeFeature CRD API. Also re-organize the
documentation a bit, moving the description of NodeFeatureRule
controller from customization guide to nfd-master usage page.
Deprecate the '-featurerules-controller' command line flag as the name
does not describe the functionality anymore: in practice it controls the
CRD controller handling both NodeFeature and NodeFeatureRule objects.
The patch introduces a duplicate, more generally named, flag
'-crd-controller'. A warning is printed in the log if
'-featurerules-controller' flag is encountered.
Add initial support for handling NodeFeature objects. With this patch
nfd-master watches NodeFeature objects in all namespaces and reacts to
changes in any of these. The node which a certain NodeFeature object
affects is determined by the "nfd.node.kubernetes.io/node-name"
annotation of the object. When a NodeFeature object targeting certain
node is changed, nfd-master needs to process all other objects targeting
the same node, too, because there may be dependencies between them.
Add a new command line flag for selecting between gRPC and NodeFeature
CRD API as the source of feature requests. Enabling NodeFeature API
disables the gRPC interface.
-enable-nodefeature-api enable NodeFeature CRD API for incoming
feature requests, will disable the gRPC
interface (defaults to false)
It is not possible to serve gRPC and watch NodeFeature objects at the
same time. This is deliberate to avoid labeling races e.g. by nfd-worker
sending gRPC requests but NodeFeature objects in the cluster
"overriding" those changes (labels from the gRPC requests will get
overridden when NodeFeature objects are processed).
Support the new NodeFeatures object of the NFD CRD api. Add two new
command line options to nfd-worker:
-kubeconfig specifies the kubeconfig to use for
connecting k8s api (defaults to empty which
implies in-cluster config)
-enable-nodefeature-api enable the NodeFeature CRD API for
communicating node features to nfd-master,
will also automatically disable gRPC
(defgault to false)
No config file option for selecting the API is available as there should
be no need for dynamically selecting between gRPC and CRD. The
nfd-master configuration must be changed in tandem and it is safer (and
avoid awkward configuration races) to configure the whole NFD deployment
at once.
Default behavior of nfd-worker is not changed i.e. NodeFeatures object
creation is not enabled by default (but must be enabled with the command
line flag).
The patch also updates the kustomize and Helm deployment, adding RBAC
rules for nfd-worker and updating the example worker configuration.
Drop the gRPC communication to nfd-master and connect to the Kubernetes
API server directly when updating NodeResourceTopology objects.
Topology-updater already has connection to the API server for listing
Pods so this is not that dramatic change. It also simplifies the code
a lot as there is no need for the NFD gRPC client and no need for
managing TLS certs/keys.
This change aligns nfd-topology-updater with the future direction of
nfd-worker where the gRPC API is being dropped and replaced by a
CRD-based API.
This patch also update deployment files and documentation to reflect
this change.
Add a reference to the label rule format in the NodeFeatureRule section.
Also make it explicit in the beginning of Hooks section that hooks are
deprecated.
Add a separate page for describing the custom resources used by NFD.
Simplify the Introduction page by moving the details of
NodeResourceTopology from there. Similarly, drop long
NodeResourceTopology example from the quick-start page, making the page
shorter and simpler.
Drop the following flags that were deprecated already in v0.8.0:
-sleep-interval (replaced by core.sleepInterval config file option)
-label-whitelist (replaced by core.labelWhiteList config file option)
-sources (replaced by -label-sources flag)
Introduce two main sections "Deployment" and "Usage" and move "Developer
guide" to the top level, too. In particular, split the huge
deployment-and-usage file into multiple parts under the new main
sections. Move customization guide from "Advanced" to "Usage".
This patch also renames "Advanced" to "Reference" as only that is left
there is reference documentation.
Set `cpu-security.tdx.enable` to `true` when TDX is avialable and has
been enabled. otherwise it'll be set to `false`.
`/sys/module/kvm_intel/parameters/tdx` presence and content is used to
detect whether a CPU is Intel TDX capable.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
In some cases (CI) it is useful to run NFD e2e tests using
ephemeral clusters. To save time and bandwidth, it is also useful
to prime the ephemeral cluster with the images under test.
In these circumstances there is no risk of running a stale image,
and having a `Always` PullPolicy hardcoded actually makes
the whole exercise null.
So we add a new option, disabled by default, to make the e2e
manifest use the `IfNotPresent` pull policy, to effectively
cover this use case.
Signed-off-by: Francesco Romani <fromani@redhat.com>
Move existing security/trusted-execution related features (i.e. SGX and
SE) under the same "security" feature, deprecating the old features. The
motivation for the change is to keep the source code and user interface
more organized as we experience a constant inflow of similar security
related features. This change will affect the user interface so it is
less painful to do it early on.
New feature labels will be:
feature.node.kubernetes.io/cpu-security.se.enabled
feature.node.kubernetes.io/cpu-security.sgx.enabled
and correspondingly new "cpu.security" feature with "se.enabled" and
"sgx.enabled" elements will be available for custom rules, for example:
- name: "sample sgx rule"
labels:
sgx.sample.feature: "true"
matchFeatures:
- feature: cpu.security
matchExpressions:
"sgx.enabled": {op: IsTrue}
At the same time deprecate old labels "cpu-sgx.enabled" and
"cpu-se.enabled" feature labels and the corresponding features for
custom rules. These will be removed in the future causing an effective
change in NFDs user interface.
Update the partial list of x86 cpuid features that are presented in the
NFD documentation. In particular, the following instructions were left
out of the list: AVXSLOW, CETIBT, CETSS, CLDEMOTE, HLE, MPX, RTM,
RTM_ALWAYS_ABORT, SERIALIZE, SHA, TSXLDTRK.
Let the documentation follow the latest release name. Even if it's just
referential here it would look odd in the future if we refer to some
ancient version.
Set `cpu.se-enabled` to `true` when IBM Secure Execution for Linux
(IBM Z & LinuxONE) is available and has been enabled.
Uses `/sys/firmware/uv/prot_virt_host`, which is available in kernels
>=5.12 + backports. For simplicity, skip more complicated facility &
kernel cmdline lookups.
This patch changes a rare corner case of custom label rules with an
empty set of matchexpressions. The patch removes a special case where an
empty match expression set matched everything and returned all feature
elements for templates to consume. With this patch the match expression
set logically evaluates all expressions in the set and returns all
matches - if there are no expressions there are no matches and no
matched features are returned. However, the overall match result
(determining if "non-template" labels will be created) in this special
case will be "true" as before as none of the zero match expressions
failed.
The former behavior was somewhat illogical and counterintuitive: having
1 to N expressions matched and returned 1 to N features (at most), but,
having 0 expressions always matched everything and returned all
features. This was some leftover proof-of-concept functionality (for
some possible future extensions) that should have been removed before
merging.
Change the default K8S_NAMESPACE to node-feature-discovery from
kube-system. The default was changed in the Makefile in commit
5d4484a1d9, but the docs were not updated
to correspond with that.
Discover "iommu/intel-iommu/version" sysfs attribute for pci devices.
This information is available for custom label rules.
An example custom rule:
- name: "iommu version rule"
labels:
iommu.version_1: "true"
matchFeatures:
- feature: pci.device
matchExpressions:
"iommu/intel-iommu/version": {op: In, value: ["1:0"]}
Add "iommu_group/type" to the list of PCI device attributes that are
discovered. The value is the raw value from sysfs (i.e DMA, DMA-FQ or
identity).
No built-in (automatic) labels are generated based on this, but, the
attribute is available for custom label rules to use. Examples of custom
rules:
- name: "iommu enabled rule"
labels:
iommu.enabled: "true"
matchFeatures:
- feature: pci.device
matchExpressions:
"iommu_group/type": {op: NotIn, value: ["unknown"]}
- name: "iommu passthrough rule"
labels:
iommu.passthrough: "true"
matchFeatures:
- feature: pci.device
matchExpressions:
"iommu_group/type": {op: In, value: ["identity"]}
Add cross-referencing links to the helm deployment and configuration
sections. Use correct names for the tls related helm options
(tls.enabled and tls.certManager).
Add a separate customization guide. Move documentation of the custom and
local sources there. Also, cover the new NodeFeatureRules custom
resource and the new expression-based label rule format.
This patch also simplifies the "Feature labels" page, describing
built-in labels. Reformat the tables describing feature labels.
Change the helm chart so that the NodeFeatureRule controller will be
disabled for other than the default deployment (i.e. all deployments
where master.instance is non-empty), unless explicitly set to true. With
this we try to ensure that there is only on controller instance for the
CR, avoiding contention and conflicts.
Move top-level serviceAccount and rbac fields under master, making the
Helm chart more coherent.
Also, drop unused rbac.serviceAccountName and
rbac.serviceAccountAnnotations from values.yaml.
Implicitly injecting the filename of the hook/featurefile into the name
of the label is confusing, counter-intuitive and unnecessarily complex
to understand. It's much clearer to advertise features and labels as
presented in the feature file / output of the hook.
NOTE: this breaks backwards compatibility with usage scenarios that rely
on prefixing the label with the filename.
Add a configuration option for controlling the enabled "raw" feature
sources. This is useful e.g. in testing and development, plus it also
allows fully shutting down discovery of features that are not needed in
a deployment. Supplements core.labelSources which controls the
enablement of label sources.
Use the single-dash (i.e. '-option' instead of '--option') format
consistently accross log messages and documentation. This is the format
that was mostly used, already, and shown by command line help of the
binaries, for example.
Add a new command line flag for disabling/enabling the controller for
NodeFeatureRule objects. In practice, disabling the controller disables
all labels generated from rules in NodeFeatureRule objects.
Implement a simple controller stub that operates on NodeFeatureRule
objects. The controller does not yet have any functionality other than
logging changes in the (NodeFeatureRule) objecs it is watching.
Also update the documentation on the -no-publish flag to match the new
functionality.
Add a cluster-scoped Custom Resource Definition for specifying labeling
rules. Nodes (node features, node objects) are cluster-level objects and
thus the natural and encouraged setup is to only have one NFD deployment
per cluster - the set of underlying features of the node stays the same
independent of how many parallel NFD deployments you have. Our extension
points (hooks, feature files and now CRs) can be be used by multiple
actors (depending on us) simultaneously. Having the CRD cluster-scoped
hopefully drives deployments in this direction. It also should make
deployment of vendor-specific labeling rules easy as there is no need to
worry about the namespace.
This patch virtually replicates the source.custom.FeatureSpec in a CRD
API (located in the pkg/apis/nfd/v1alpha1 package) with the notable
exception that "MatchOn" legacy rules are not supported. Legacy rules
are left out in order to keep the CRD simple and clean.
The duplicate functionality in source/custom will be dropped by upcoming
patches.
This patch utilizes controller-gen (from sigs.k8s.io/controller-tools)
for generating the CRD and deepcopy methods. Code can be (re-)generated
with "make generate". Install controller-gen with:
go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.7.0
Update kustomize and helm deployments to deploy the CRD.
The NodeResourceTopology API has been made cluster
scoped as in the current context a CR corresponds to
a Node and since Node is a cluster scoped resource it
makes sense to make NRT cluster scoped as well.
Ref: https://github.com/k8stopologyawareschedwg/noderesourcetopology-api/pull/18
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
Prior to this feature, NFD consisted of only software components namely
nfd-master and nfd-worker. We have introduced another software component
called nfd-topology-updater.
NFD-Topology-Updater is a daemon responsible for examining allocated resources
on a worker node to account for allocatable resources on a per-zone basis (where
a zone can be a NUMA node). It then communicates the information to nfd-master
which does the CRD creation corresponding to all the nodes in the cluster. One
instance of nfd-topology-updater is supposed to be running on each node of the
cluster.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
* Simplify NFD worker service configuration in Helm
Signed-off-by: Elias Koromilas <elias.koromilas@gmail.com>
* Update docs/get-started/deployment-and-usage.md
Co-authored-by: Markus Lehtonen <markus.lehtonen@intel.com>
Co-authored-by: Markus Lehtonen <markus.lehtonen@intel.com>
Make table of contents in the pages cleaner and more readable by
dropping the main heading (H1 level) from TOCs. This was the original
intention with the usage of "no_toc" kramdown magic, which was broken,
however. The kramdown class magic needs to be specified on the line
immediately following the headinds, otherwise it has no effect. We need
to disable MD022 rule of mdlint as it does not understand this magic.
Add a separate label namespace for profile labels, intended for
user-specified higher level "meta features". Also sub-namespaces of this
(i.e. <sub-ns>.profile.node.kubernetes.io) are allowed.
Allow <sub-ns>.feature.node.kubernetes.io label namespaces. Makes it
possible to have e.g. vendor specific label ns without the need to user
-extra-label-ns.
The naming was changed in when with cpuid v2
(github.com/klauspost/cpuid/v2) and we didn't catch this in NFD. No
issue reports of the inadvertent naming change so let's just adapt to
the updated naming in NFD configuration. The SSE4* labels are disabled
by default so they're not widely used, if at all.
Change the sidebar customization so that the full version (e.g. v0.8.2)
is displayed, instead of the truncated "releae branch version" (e.g.
v0.8) in the top left corner of the page. The items in the version menu
are unchanged and will still show the shorter form.
In my homelab, I have different FTDI serial converters connected to
several utility meters. They all have identical vendor/device, but
different serials.
In order to detect a specific FTDI unit (eg. the one connected to my
electricity meter), I'd like feature labels triggered by a specific USB
serial.
Signed-off-by: Jorik Jonker <jorik@kippendief.biz>
* Add support for configurable runtime full and minimal images.
* Fixups and renamings.
* Change variables *_IMG_* to *_IMAGE_*
* Fix args in Dockerfile also.
cert-manager can be used to automate TLS certificate management for
nfd-master and the nfd-worker pod(s).
Add a template to deploy cert-manager CA Issuer and Certificates and
document steps how to use them.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Take the default value displayed for the image.repository Helm chart
parameter dynamically from the Jekyll configuration. This way, we
get the correct value for releases/release branches.
This adds additional cpu features:
- pstate status from status of intel_pstate driver
- pstate scaling settings from scaling_governor
- cstate enable from max_cstates in intel_idle driver
After moving to the flags package for command line argument parsing NFD
accepts command line arguments (flags) starting with a single dash (e.g.
-no-publish in addition to --no-publish). Even if double-dash can be
used the single-dash version is printed e.g. in the usage string (from
-h, -help) so align documentation with that.
There are cases when the only available metadata for discovering
features is the node's name. The "nodename" rule extends the custom
source and matches when the node's name matches one of the given
nodename regexp patterns.
It is also possible now to set an optional "value" on custom rules,
which overrides the default "true" label value in case the rule matches.
In order to allow more dynamic configurations without having to modify
the complete worker configuration, custom rules are additionally read
from a "custom.d" directory now. Typically that directory will be filled
by mounting one or more ConfigMaps.
Signed-off-by: Marc Sluiter <msluiter@redhat.com>
This commit adds Helm chart for node-feature-discovery
Signed-off-by: Adrian Chiris <adrianc@nvidia.com>
Signed-off-by: Ivan Kolodiazhnyi <ikolodiazhny@nvidia.com>
Add a config file option for controlling the enabled feature sources,
aimed at replacing the --sources command line flag which is now marked
as deprecated. The command line flag takes precedence over the config
file option.
Add a config file option for label whitelisting. Deprecate the
--label-whitelist command line flag. Note that the command line flag has
higher priority than the config file option.
Add a new config file option for (dynamically) controlling the sleep
interval. At the same time, deprecate the --sleep-interval command line
flag. The command line flag takes precedence over the config file option.
This can be used to help running multiple parallel NFD deployments in
the same cluster. The flag changes the node annotation namespace to
<instance>.nfd.node.kubernetes.io allowing different nfd-master intances
to store metadata in separate annotations.
Add a virtually empty ConfigMap that is mounted inside the workers.
Makes it easier to start customizing the worker deployment e.g. with just:
$ kubectl -n ${NFD_NS} edit configmap nfd-worker-conf
Create a new 'templates' make target for inserting the content of
nfd-worker.conf.example into the configmap spec of the templates. Thus,
'make templates' should be run whenever the example config is update.
Update the verify.sh prow script to check that the templates are up to
date.
This patch also streamlines the documentation about configuration
management, reflecting the changes.
A new special value 'all' is a shortcut for enabling all feature
sources. It should be the only name specified -- if any other names are
specified 'all' does not take effect, but, we only enable the listed
feature sources. E.g.
--sources=all enables all sources, but
--sources=all,cpu only enables the cpu source
Also, print a warning if unknown sources are specified.
Only show the dynamic version menu in production environment where it's
available. Local (development) builds only show a stub version menu with
one item.
Also, fix the theme customization. Append version list items to the
correct element in html so that they are displayed correctly. Drop
unnecessary <div/>.
Make it hidden, by default. Unify style with other items in the navbar
(making version list horizontal, for example). Show current version next
to the caret mark that opens the menu.
Describe NFD deployment via the operator.
Add minimal documentation for uninnstalling NFD. Add instructions and
template spec for running "nfd-master --prune". Also modify the RBAC
rules in nfd-master spec template to make it possible doing --prune
(allow nfd-master to list nodes in the cluster).
Use the existing content as a base but with heave editing. Move local
examples involving make to the developers guide.
Drop the really hackish label-nodes.sh. Just replace it with command
line examples in the documentation. If somebody really is dying for this
write it from scratch and put under scripts/hack.
Fix markdown syntax and style for content that was moved from README.md
to docs/:
- get-started/introduction
- examples-and-demos
- get-started/features
- contributing
Unify the spelling of master and worker in headings and beginning of
senctences.
Also, env variable for container name in developers-guide.
Move all content from README.md to the Jekyll site under docs/. Also
re-organize it into multiple sub-pages.
Populate README with fresh content turning it into virtually a big link
to the html-based documentation site. Spiced up with super-quick-start
instructions.
Make the list of available versions dynamic. The items displayed is now
generated with javascript. The parent directory of the site is supposed
to contain versions.js providing getVersionListItems() that returns the
available versions.
The update-gh-pages.sh script is modified to update/create versions.js
on every invocation. It simply lists all directories in the root
directory and adds them to the version list.
Sync our customizations under _includes/ with v2.0.9 of
jekyll-rtd-theme. Also, pin the theme version in order to avoid abrupt
build failures in the future (caused by our customization becoming
incompatible with the rolling latest version).
Also, drop jekyll-rtd-theme from the Gemfile. It's useless there as it's
being mandated by remote-theme setting in _config.yml.
Add rules for building and serving the html documentation inside a
container.
- 'make site-build' will build the html content unders docs/_site.
- 'make site-serve' builds the site content and serves it at
http://localhost:4000/
