1
0
Fork 0
mirror of https://github.com/kubernetes-sigs/node-feature-discovery.git synced 2025-03-05 08:17:04 +00:00
Commit graph

368 commits

Author SHA1 Message Date
Viktor Oreshkin
6fd12a2da7 apis/nfd: fix templates with MatchAny only
Signed-off-by: Viktor Oreshkin <imselfish@stek29.rocks>
2022-08-23 18:00:44 +03:00
Markus Lehtonen
2c92e1dcff logging: do not use %w with klog.Errorf
It is not recognized (and does not work like with fmt.Errorf) so use %v
instead.
2022-08-22 14:39:52 +03:00
Viktor Oreshkin
4375e08e39 apis/nfd: add more tests for templates
test that NodeFeatureRule templates work with empty MatchFeatures, but
with MatchAny.

this test would fail, higligting an issue which is fixed in next commit.

see #864.

Signed-off-by: Viktor Oreshkin <imselfish@stek29.rocks>
2022-08-22 02:27:55 +03:00
Markus Lehtonen
889e4c1351 nfd-master: more fixes to log messages
Use correct name for the CR (NodeFeatureRule) object. Also, the resource
is cluster-scoped so don't print the namespace.
2022-08-17 10:07:26 +03:00
Markus Lehtonen
f5ee836bbf nfd-master: fix incorrect log messages in crd controller 2022-08-16 16:39:27 +03:00
Markus Lehtonen
38e763e36c Refresh auto-generated files 2022-08-10 14:24:33 +03:00
Markus Lehtonen
345e9bf72c apis/nfd: revert the type hack
Revert the hack that was a workaround for issues with k8s deepcopy-gen.
New deepcopy-gen is able to generate code correctly without issues so
this is not needed anymore.

Also, removing this hack solves issues with object validation when
creating NodeFeatureRules programmatically with nfd go-client. This is
needed later with NodeFeatureRules e2e-tests.

Logically reverts f3cc109f99.
2022-08-10 14:24:33 +03:00
Markus Lehtonen
ac3030ce48 Re-generate files
Refresh auto-generated files using the new conainerized approach.
2022-08-10 09:47:23 +03:00
Markus Lehtonen
b7658c25fd generate: update mockery to latest version
In order to be able to run it on Go v1.18.
2022-08-10 09:47:23 +03:00
Markus Lehtonen
136c036d4d Drop the iommu source
It was deprecated in v0.10.0.
2022-06-14 15:00:29 +03:00
Markus Lehtonen
36341bf4c7 apis/nfd: empty match expression set returns no features for templates
This patch changes a rare corner case of custom label rules with an
empty set of matchexpressions. The patch removes a special case where an
empty match expression set matched everything and returned all feature
elements for templates to consume. With this patch the match expression
set logically evaluates all expressions in the set and returns all
matches - if there are no expressions there are no matches and no
matched features are returned. However, the overall match result
(determining if "non-template" labels will be created) in this special
case will be "true" as before as none of the zero match expressions
failed.

The former behavior was somewhat illogical and counterintuitive: having
1 to N expressions matched and returned 1 to N features (at most), but,
having 0 expressions always matched everything and returned all
features. This was some leftover proof-of-concept functionality (for
some possible future extensions) that should have been removed before
merging.
2022-03-24 11:43:42 +02:00
Tuomas Katila
2ceafe83b7 topologyupdater: Prevent crash with incorrect node id
It's possible for device plugins to advertise non-existent
numa node ids that cause topology updater to crash.

Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>
2022-03-15 11:16:02 +02:00
Dipto Chakrabarty
19a57789ad
Additional Lint Fixes in Codebase (#779)
* fix comments and conditonals to fix lint issues

* more linter fixes and spelling fixes

* fix linter issues based on feedback
2022-03-02 17:12:46 -08:00
Markus Lehtonen
f9b4ba87a8 tls: require min TLS version 1.3
Deny deprecated TLS versions (1.0 and 1.1). We don't really excpect
other clients than NFD itself so we can just request the latest version.
2022-02-25 10:08:37 +02:00
Kubernetes Prow Robot
0c330b1a35
Merge pull request #736 from marquiz/devel/grpc-stop
nfd-master: do graceful stop of gRPC server
2022-01-21 03:05:59 -08:00
Markus Lehtonen
e53d053475 nfd-master: do graceful stop of gRPC server 2022-01-21 12:03:07 +02:00
Markus Lehtonen
e95a4dd460 nfd-master: print gRPC server error correctly 2022-01-21 11:56:28 +02:00
Mohammed Naser
cf1bc4a34d Increase timeout in test setups
This patch increases the timeout when setting up the NFD master
to 5 seconds instead of 1 second to allow for running tests in
slow environments.
2022-01-20 18:59:30 -05:00
Dipto Chakrabarty
755294184c
Fix GoLinter Issues in the files (#711)
* fix linter issues for few files

* fix linter issue of exported const Name should have comment or be unexported

* fix name lint issue and resolve lints

* add changes to comments
2022-01-18 23:12:06 -08:00
Markus Lehtonen
838a375f85 source/iommu: deprecate and disable by default
Deprecate the iommu source and disable it by default.
2021-12-20 10:21:29 +02:00
Markus Lehtonen
a6eddbab4f source: rename TestSource to SupplementalSource
Just widen the scope in terms of naming, to cover deprecated and/or
experimental sources too, for example.
2021-12-20 10:05:00 +02:00
Markus Lehtonen
bf01875368 nfd-worker: drop 'custom-' prefix from matchFeatures custom rules
Do not prefix label names from the new matchFeatures/matchAny custom
rules with "custom-". We want to have the same result (set of labels)
from a rule independent of whether it has been specified in worker
config or in a NodeFeatureRule CRs. Legacy matchOn rules (not available
in NodeFeatureRule CRs) are intact, i.e. still prefixed, in order to
retain backwards compatibility.
2021-12-09 21:52:40 +02:00
Markus Lehtonen
82e14300a4 source/fake: implement FeatureSource
Makes it possible to create fake features for custom rules, enabling
testing.
2021-12-07 10:34:41 +02:00
Markus Lehtonen
58e1461d90 nfd-worker: add -feature-sources command line flag
Allows controlling (enable/disable) the "raw" feature detection.
Especially useful for development and testing.
2021-12-03 09:42:35 +02:00
Markus Lehtonen
df6909ed5e nfd-worker: add core.featureSources config option
Add a configuration option for controlling the enabled "raw" feature
sources. This is useful e.g. in testing and development, plus it also
allows fully shutting down discovery of features that are not needed in
a deployment. Supplements core.labelSources which controls the
enablement of label sources.
2021-12-03 09:42:35 +02:00
Markus Lehtonen
2c3a4d1588 nfd-worker: rename nfdWorker.enabledSources to labelSources
Refactoring in head of adding new config option for feature sources.
2021-12-02 21:08:46 +02:00
Markus Lehtonen
8cd58af613 nfd-worker: disable sources more easily
Make it easier to disable single sources by prefixing the source name
with a dash ('-') in the core.sources config option (or -sources cmdline
flag).
2021-12-02 10:36:51 +02:00
Markus Lehtonen
773280de65 nfd-worker: provide deprecated core.sources config option
Provide backwards compatibility via a deprecated 'core.sources' config
file option. This will override 'core.labelSources'. A warning is
printed in the log if this option is detected.
2021-12-01 17:11:49 +02:00
Markus Lehtonen
ad9c7dfa1e nfd-worker: rename config option 'sources' to 'labelSources'
The goal is to make the name more descriptive. Also keeping in mind a
possible future addition a 'featureSources' option (or similar) for
controlling the feature discovery.
2021-12-01 17:11:49 +02:00
Kubernetes Prow Robot
86bfe74cd7 Merge pull request #671 from marquiz/fixes/single-dash-flags
Use single-dash format of cmdline flags
2021-12-01 06:45:15 -08:00
Markus Lehtonen
1765a37c6a pkg/apis/nfd: drop unnecessary else statements 2021-12-01 10:55:50 +02:00
Markus Lehtonen
3f225be081 pkg/apis/nfd: use consistent receiver name for methods of templateHelper 2021-12-01 10:51:47 +02:00
Markus Lehtonen
d07400206f pkg/apis/nfd/v1alpha1: document exported symbols
Add missing comments and fix some existing ones.
2021-12-01 10:46:56 +02:00
Markus Lehtonen
c4f7ab0abe pkg/api/feature: document exported functions 2021-12-01 10:30:17 +02:00
Markus Lehtonen
a57a25f63c Use single-dash format of cmdline flags
Use the single-dash (i.e. '-option' instead of '--option') format
consistently accross log messages and documentation. This is the format
that was mostly used, already, and shown by command line help of the
binaries, for example.
2021-11-25 18:03:54 +02:00
Markus Lehtonen
b648d005e1 pkg/apis/nfd: support templating of "vars"
Support templating of var names in a similar manner as labels. Add
support for a new 'varsTemplate' field to the feature rule spec which is
treated similarly to the 'labelsTemplate' field. The value of the field
is processed through the golang "text/template" template engine and the
expanded value must contain variables in <key>=<value> format, separated
by newlines i.e.:

  - name: <rule-name>
    varsTemplate: |
      <label-1>=<value-1>
      <label-2>=<value-2>
      ...

Similar rules as for 'labelsTemplate' apply, i.e.

1. In case of matchAny is specified, the template is executed separately
   against each individual matchFeatures matcher.
2. 'vars' field has priority over 'varsTemplate'
2021-11-25 12:50:47 +02:00
Markus Lehtonen
f75303ce43 pkg/apis/nfd: add variables to rule spec and support backreferences
Support backreferencing of output values from previous rules. Enables
complex rule setups where custom features are further combined together
to form even more sophisticated higher level labels. The labels created
by preceding rules are available as a special 'rule.matched' feature
(for matchFeatures to use).

If referencing rules accross multiple configs/CRDs care must be taken
with the ordering. Processing order of rules in nfd-worker:

1. Static rules
2. Files from /etc/kubernetes/node-feature-discovery/custom.d/
   in alphabetical order. Subdirectories are processed by reading their
   files in alphabetical order.
3. Custom rules from main nfd-worker.conf

In nfd-master, NodeFeatureRule objects are processed in alphabetical
order (based on their metadata.name).

This patch also adds new 'vars' fields to the rule spec. Like 'labels',
it is a map of key-value pairs but no labels are generated from these.
The values specified in 'vars' are only added for backreferencing into
the 'rules.matched' feature. This may by desired in schemes where the
output of certain rules is only used as intermediate variables for other
rules and no labels out of these are wanted.

An example setup:

  - name: "kernel feature"
    labels:
      kernel-feature:
    matchFeatures:
      - feature: kernel.version
        matchExpressions:
          major: {op: Gt, value: ["4"]}

  - name: "intermediate var feature"
    vars:
      nolabel-feature: "true"
    matchFeatures:
      - feature: cpu.cpuid
        matchExpressions:
          AVX512F: {op: Exists}
      - feature: pci.device
        matchExpressions:
          vendor: {op: In, value: ["8086"]}
          device: {op: In, value: ["1234", "1235"]}

  - name: top-level-feature
    matchFeatures:
      - feature: rule.matched
        matchExpressions:
          kernel-feature: "true"
          nolabel-feature: "true"
2021-11-25 12:50:47 +02:00
Markus Lehtonen
8a4d3161cf pkg/apis/nfd: stricter format checking for template labels
Require that the expanded LabelsTemplate has values. That is, the
(expanded) template must consist of key=value pairs separated by
newlines. No default value will be assigned and we now return an error
if a (non-empty) line not conforming with the key=value format is
encountered.

Commit c8d73666d described that the value defaults to "true" if not
specified. That was not the case and we defaulted to an empty string,
instead.

An example:

  - name: "my rule"
    labelsTemplate: |
      my.label.1=foo
      my.label.2=

Would create these labels:

  "my.label.1": "foo"
  "my.label.2": ""

Further, the following:

  - name: "my failing rule"
    labelsTemplate: |
      my.label.3

will cause an error in the rule processing.
2021-11-24 21:31:35 +02:00
Markus Lehtonen
c8d73666d6 pkg/apis/nfd: support label name templating
Support templating of label names in feature rules. It is available both
in NodeFeatureRule CRs and in custom rule configuration of nfd-worker.

This patch adds a new 'labelsTemplate' field to the rule spec, making it
possible to dynamically generate multiple labels per rule based on the
matched features. The feature relies on the golang "text/template"
package.  When expanded, the template must contain labels in a raw
<key>[=<value>] format (where 'value' defaults to "true"), separated by
newlines i.e.:

  - name: <rule-name>
    labelsTemplate: |
      <label-1>[=<value-1>]
      <label-2>[=<value-2>]
      ...

All the matched features of 'matchFeatures' directives are available for
templating engine in a nested data structure that can be described in
yaml as:

.
  <domain-1>:
      <key-feature-1>:
        - Name: <matched-key>
        - ...

      <value-feature-1:
        - Name: <matched-key>
          Value: <matched-value>
        - ...

      <instance-feature-1>:
        - <attribute-1-name>: <attribute-1-value>
          <attribute-2-name>: <attribute-2-value>
          ...
        - ...

  <domain-2>:
     ...

That is, the per-feature data available for matching depends on the type
of feature that was matched:

- "key features": only 'Name' is available
- "value features": 'Name' and 'Value' can be used
- "instance features": all attributes of the matched instance are
   available

NOTE: In case of matchAny is specified, the template is executed
separately against each individual matchFeatures matcher and the
eventual set of labels is a superset of all these expansions.  Consider
the following:

  - name: <name>
    labelsTemplate: <template>
    matchAny:
      - matchFeatures: <matcher#1>
      - matchFeatures: <matcher#2>
    matchFeatures: <matcher#3>

In the example above (assuming the overall result is a match) the
template would be executed on matcher#1 and/or matcher#2 (depending on
whether both or only one of them match), and finally on matcher#3, and
all the labels from these separate expansions would be created (i.e. the
end result would be a union of all the individual expansions).

NOTE 2: The 'labels' field has priority over 'labelsTemplate', i.e.
labels specified in the 'labels' field will override any labels
originating from the 'labelsTemplate' field.

A special case of an empty match expression set matches everything (i.e.
matches/returns all existing keys/values). This makes it simpler to
write templates that run over all values. Also, makes it possible to
later implement support for templates that run over all _keys_ of a
feature.

Some example configurations:

  - name: "my-pci-template-features"
    labelsTemplate: |
      {{ range .pci.device }}intel-{{ .class }}-{{ .device }}=present
      {{ end }}
    matchFeatures:
      - feature: pci.device
        matchExpressions:
          class: {op: InRegexp, value: ["^06"]}
          vendor: ["8086"]

  - name: "my-system-template-features"
    labelsTemplate: |
      {{ range .system.osrelease }}system-{{ .Name }}={{ .Value }}
      {{ end }}
    matchFeatures:
      - feature: system.osRelease
        matchExpressions:
          ID: {op: Exists}
          VERSION_ID.major: {op: Exists}

Imaginative template pipelines are possible, of course, but care must be
taken in order to produce understandable and maintainable rule sets.
2021-11-23 21:03:22 +02:00
Markus Lehtonen
085af7c2c7 pkg/apis/nfd: helpers for handling templates in Rule names
Implement a private helper type (nameTemplateHelper) for handling
(executing and caching) of templated names. DeepCopy methods are
manually implemented as controller-gen is not able to help with that.
2021-11-23 15:08:53 +02:00
Markus Lehtonen
33fdf75190 nfd-master: process labeling rules from CRs
Enable Custom Resource based label creation in nfd-master. This extends
the previously implemented controller stub for watching NodeFeatureRule
objects. NFD-master watches NodeFeatureRule objects in the cluster and
processes the rules on every incoming labeling request from workers.
The functionality relies on the "raw features" (identical to how
nfd-worker handles custom rules) submitted by nfd-worker, making it
independent of the label source configuration of the worker. This means
that the labeling functions as expected even if all sources in the
worker are disabled.

NOTE: nfd-master is stateless and re-labeling only happens on the
reception of SetLabelsRequest from the worker – i.e. on intervals
specified by the core.sleepInterval configuration option (or
-sleep-interval cmdline flag) of each nfd-worker instance. This means
that modification/creation of NodeFeatureRule objects does not
automatically update the node labels. Instead, the changes only come
visible when workers send their labeling requests.
2021-11-23 09:18:07 +02:00
Markus Lehtonen
e8872462dc nfd-master: add -featurerules-controller flag
Add a new command line flag for disabling/enabling the controller for
NodeFeatureRule objects. In practice, disabling the controller disables
all labels generated from rules in NodeFeatureRule objects.
2021-11-22 16:57:42 +02:00
Markus Lehtonen
e6e32a88c3 nfd-master: implement controller for NodeFeatureRule CRs
Implement a simple controller stub that operates on NodeFeatureRule
objects. The controller does not yet have any functionality other than
logging changes in the (NodeFeatureRule) objecs it is watching.

Also update the documentation on the -no-publish flag to match the new
functionality.
2021-11-22 16:57:42 +02:00
Markus Lehtonen
237c4f7824 pkg/apihelpers: split out loading of kubeconfig to a separate function
Make kubeconfig loading and parsing re-usable for multiple clients.
2021-11-22 16:57:42 +02:00
NHM Tanveer Hossain Khan
856dfdd8b4 Remove fatal logging to error based on the feedback 2021-11-19 16:57:21 -05:00
Markus Lehtonen
6624ab312b pkg/generated: add code for interacting with CRD API
Add auto-generated code for interfacing our CRD API. On top of this, a
CR controller can be implemented. This patch uses k8s/code-generator
for code generation. Run "make generate" in order to (re-)generate
everything. Path to the code-generator repository may need to be
specified:

  K8S_CODE_GENERATOR=path/to/code-generator make apigen

Code-generator version 0.20.7 was used to create this patch. Install
k8s code-generator tools and clone the repo with:

  git clone https://github.com/kubernetes/code-generator -b v0.20.7 <path/to/code-generator>
  go install k8s.io/code-generator/cmd/...(at)v0.20.7
2021-11-17 18:51:34 +02:00
Markus Lehtonen
b96b86bc6c pkg/apis/nfd: drop excess field from the CRD
Drop stale leftover "LabelsTemplate" field from the rule spec.
2021-11-17 16:40:28 +02:00
Markus Lehtonen
8b9df3cf31 source/custom: move rule matching to pkg/apis/nfd
Move the rule processing of matchFeatures and matchAny from
source/custom package over to pkg/apis/nfd, aiming for better integrity
and re-usability of the code. Does not change the CRD API as such, just
adds more supportive functions.
2021-11-17 14:02:00 +02:00
Markus Lehtonen
3765ae24d6 pkg/apis/nfd: specify a dedicated type for regexp cache
Having a dedicated type makes it possible to specify deepcopy functions
for it. We need to do this manually as deepcopy-gen doesn't know how to
create copies of regexps.
2021-11-17 13:40:43 +02:00
Markus Lehtonen
f3cc109f99 pkg/apis/nfd: work around issues with k8s deepcopy-gen
Without this hack the generated code does not compile.
2021-11-17 13:40:43 +02:00