1
0
Fork 0
mirror of https://github.com/kubernetes-sigs/node-feature-discovery.git synced 2024-12-14 11:57:51 +00:00

Merge pull request #290 from adrianchiris/custom_features

Support custom features
This commit is contained in:
Kubernetes Prow Robot 2020-03-24 08:26:48 -07:00 committed by GitHub
commit 7c4ff52a3c
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
12 changed files with 562 additions and 62 deletions

123
README.md
View file

@ -153,6 +153,7 @@ for up-to-date information about the required volume mounts.
The current set of feature sources are the following:
- CPU
- Custom
- IOMMU
- Kernel
- Memory
@ -184,6 +185,7 @@ feature logically has sub-hierarchy, e.g. `sriov.capable` and
```json
{
"feature.node.kubernetes.io/cpu-<feature-name>": "true",
"feature.node.kubernetes.io/custom-<feature-name>": "true",
"feature.node.kubernetes.io/iommu-<feature-name>": "true",
"feature.node.kubernetes.io/kernel-<feature name>": "<feature value>",
"feature.node.kubernetes.io/memory-<feature-name>": "true",
@ -254,6 +256,127 @@ capability might be supported but not enabled.
| JSCVT | Perform Conversion to Match Javascript
| DCPOP | Persistent Memory Support
### Custom Features
The Custom feature source allows the user to define features based on a mix of predefined rules.
A rule is provided input witch affects its process of matching for a defined feature.
To aid in making Custom Features clearer, we define a general and a per rule nomenclature, keeping things as
consistent as possible.
#### General Nomenclature & Definitions
```
Rule :Represents a matching logic that is used to match on a feature.
Rule Input :The input a Rule is provided. This determines how a Rule performs the match operation.
Matcher :A composition of Rules, each Matcher may be composed of at most one instance of each Rule.
```
#### Custom Features Format (using the Nomenclature defined above)
```yaml
- name: <feature name>
matchOn:
- <Rule-1>: <Rule-1 Input>
[<Rule-2>: <Rule-2 Input>]
- <Matcher-2>
- ...
- ...
- <Matcher-N>
- <custom feature 2>
- ...
- ...
- <custom feature M>
```
#### Matching process
Specifying Rules to match on a feature is done by providing a list of Matchers.
Each Matcher contains one or more Rules.
Logical _OR_ is performed between Matchers and logical _AND_ is performed between Rules
of a given Matcher.
#### Rules
##### PciId Rule
###### Nomenclature
```
Attribute :A PCI attribute.
Element :An identifier of the PCI attribute.
```
The PciId Rule allows matching the PCI devices in the system on the following Attributes: `class`,`vendor` and
`device`. A list of Elements is provided for each Attribute.
###### Format
```yaml
pciId :
class: [<class id>, ...]
vendor: [<vendor id>, ...]
device: [<device id>, ...]
```
Matching is done by performing a logical _OR_ between Elements of an Attribute and logical _AND_ between the specified Attributes for
each PCI device in the system.
At least one Attribute must be specified. Missing attributes will not partake in the matching process.
##### LoadedKMod Rule
###### Nomenclature
```
Element :A kernel module
```
The LoadedKMod Rule allows matching the loaded kernel modules in the system against a provided list of Elements.
###### Format
```yaml
loadedKMod : [<kernel module>, ...]
```
Matching is done by performing logical _AND_ for each provided Element, i.e the Rule will match if all provided Elements (kernel modules) are loaded
in the system.
#### Example
```yaml
custom:
- name: "my.kernel.feature"
matchOn:
- loadedKMod: ["kmod1", "kmod2"]
- name: "my.pci.feature"
matchOn:
- pciId:
vendor: ["15b3"]
device: ["1014", "1017"]
- name: "my.combined.feature"
matchOn:
- loadedKMod : ["vendor_kmod1", "vendor_kmod2"]
pciId:
vendor: ["15b3"]
device: ["1014", "1017"]
- name: "my.accumulated.feature"
matchOn:
- loadedKMod : ["some_kmod1", "some_kmod2"]
- pciId:
vendor: ["15b3"]
device: ["1014", "1017"]
```
__In the example above:__
- A node would contain the label: `feature.node.kubernetes.io/custom-my.kernel.feature=true`
if the node has `kmod1` _AND_ `kmod2` kernel modules loaded.
- A node would contain the label: `feature.node.kubernetes.io/custom-my.pci.feature=true`
if the node contains a PCI device with a PCI vendor ID of `15b3` _AND_ PCI device ID of `1014` _OR_ `1017`.
- A node would contain the label: `feature.node.kubernetes.io/custom-my.combined.feature=true`
if `vendor_kmod1` _AND_ `vendor_kmod2` kernel modules are loaded __AND__ the node contains a PCI device
with a PCI vendor ID of `15b3` _AND_ PCI device ID of `1014` _or_ `1017`.
- A node would contain the label: `feature.node.kubernetes.io/custom-my.accumulated.feature=true`
if `some_kmod1` _AND_ `some_kmod2` kernel modules are loaded __OR__ the node contains a PCI device
with a PCI vendor ID of `15b3` _AND_ PCI device ID of `1014` _OR_ `1017`.
#### Statically defined features
Some feature labels which are common and generic are defined statically in the `custom` feature source.
A user may add additional Matchers to these feature labels by defining them in the `nfd-worker` configuration file.
| Feature | Attribute | Description |
| ------- | --------- | -----------|
| rdma | capable | The node has an RDMA capable Network adapter |
| rdma | enabled | The node has the needed RDMA modules loaded to run RDMA traffic |
### IOMMU Features
| Feature name | Description |

View file

@ -91,7 +91,7 @@ func argsParse(argv []string) (worker.Args, error) {
in testing
[Default: ]
--sources=<sources> Comma separated list of feature sources.
[Default: cpu,iommu,kernel,local,memory,network,pci,storage,system]
[Default: cpu,custom,iommu,kernel,local,memory,network,pci,storage,system]
--no-publish Do not publish discovered features to the
cluster-local Kubernetes API server.
--label-whitelist=<pattern> Regular expression to filter label names to

View file

@ -23,6 +23,8 @@ import (
. "github.com/smartystreets/goconvey/convey"
)
var allSources = []string{"cpu", "custom", "iommu", "kernel", "local", "memory", "network", "pci", "storage", "system"}
func TestArgsParse(t *testing.T) {
Convey("When parsing command line arguments", t, func() {
Convey("When --no-publish and --oneshot flags are passed", func() {
@ -32,7 +34,7 @@ func TestArgsParse(t *testing.T) {
So(args.SleepInterval, ShouldEqual, 60*time.Second)
So(args.NoPublish, ShouldBeTrue)
So(args.Oneshot, ShouldBeTrue)
So(args.Sources, ShouldResemble, []string{"cpu", "iommu", "kernel", "local", "memory", "network", "pci", "storage", "system"})
So(args.Sources, ShouldResemble, allSources)
So(len(args.LabelWhiteList), ShouldEqual, 0)
So(err, ShouldBeNil)
})
@ -56,7 +58,7 @@ func TestArgsParse(t *testing.T) {
Convey("args.labelWhiteList is set to appropriate value and args.sources is set to default value", func() {
So(args.NoPublish, ShouldBeFalse)
So(args.Sources, ShouldResemble, []string{"cpu", "iommu", "kernel", "local", "memory", "network", "pci", "storage", "system"})
So(args.Sources, ShouldResemble, allSources)
So(args.LabelWhiteList, ShouldResemble, ".*rdt.*")
So(err, ShouldBeNil)
})

View file

@ -44,3 +44,22 @@
# - "device"
# - "subsystem_vendor"
# - "subsystem_device"
# custom:
# - name: "my.kernel.feature"
# matchOn:
# - loadedKMod: ["example_kmod1", "example_kmod2"]
# - name: "my.pci.feature"
# matchOn:
# - pciId:
# class: ["0200"]
# vendor: ["15b3"]
# device: ["1014", "1017"]
# - pciId :
# vendor: ["8086"]
# device: ["1000", "1100"]
# - name: "my.combined.feature"
# matchOn:
# - pciId:
# vendor: ["15b3"]
# device: ["1014", "1017"]
# loadedKMod : ["vendor_kmod1", "vendor_kmod2"]

View file

@ -36,6 +36,7 @@ import (
"sigs.k8s.io/node-feature-discovery/pkg/version"
"sigs.k8s.io/node-feature-discovery/source"
"sigs.k8s.io/node-feature-discovery/source/cpu"
"sigs.k8s.io/node-feature-discovery/source/custom"
"sigs.k8s.io/node-feature-discovery/source/fake"
"sigs.k8s.io/node-feature-discovery/source/iommu"
"sigs.k8s.io/node-feature-discovery/source/kernel"
@ -61,6 +62,7 @@ type NFDConfig struct {
Cpu *cpu.NFDConfig `json:"cpu,omitempty"`
Kernel *kernel.NFDConfig `json:"kernel,omitempty"`
Pci *pci.NFDConfig `json:"pci,omitempty"`
Custom *custom.NFDConfig `json:"custom,omitempty"`
} `json:"sources,omitempty"`
}
@ -234,6 +236,7 @@ func configParse(filepath string, overrides string) error {
config.Sources.Cpu = &cpu.Config
config.Sources.Kernel = &kernel.Config
config.Sources.Pci = &pci.Config
config.Sources.Custom = &custom.Config
data, err := ioutil.ReadFile(filepath)
if err != nil {
@ -276,6 +279,7 @@ func configureParameters(sourcesWhiteList []string, labelWhiteListStr string) (e
pci.Source{},
storage.Source{},
system.Source{},
custom.Source{},
// local needs to be the last source so that it is able to override
// labels from other sources
local.Source{},

93
source/custom/custom.go Normal file
View file

@ -0,0 +1,93 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package custom
import (
"fmt"
"log"
"sigs.k8s.io/node-feature-discovery/source"
"sigs.k8s.io/node-feature-discovery/source/custom/rules"
)
// Custom Features Configurations
type MatchRule struct {
PciId *rules.PciIdRule `json:"pciId,omitempty""`
LoadedKMod *rules.LoadedKModRule `json:"loadedKMod,omitempty""`
}
type CustomFeature struct {
Name string `json:"name"`
MatchOn []MatchRule `json:"matchOn"`
}
type NFDConfig []CustomFeature
var Config = NFDConfig{}
// Implements FeatureSource Interface
type Source struct{}
// Return name of the feature source
func (s Source) Name() string { return "custom" }
// Discover features
func (s Source) Discover() (source.Features, error) {
features := source.Features{}
allFeatureConfig := append(getStaticFeatureConfig(), Config...)
log.Printf("INFO: Custom features: %+v", allFeatureConfig)
// Iterate over features
for _, customFeature := range allFeatureConfig {
featureExist, err := s.discoverFeature(customFeature)
if err != nil {
return features, fmt.Errorf("failed to discover feature: %s. %s", customFeature.Name, err.Error())
}
if featureExist {
features[customFeature.Name] = true
}
}
return features, nil
}
// Process a single feature by Matching on the defined rules.
// A feature is present if all defined Rules in a MatchRule return a match.
func (s Source) discoverFeature(feature CustomFeature) (bool, error) {
for _, rule := range feature.MatchOn {
// PCI ID rule
if rule.PciId != nil {
match, err := rule.PciId.Match()
if err != nil {
return false, err
}
if !match {
continue
}
}
// Loaded kernel module rule
if rule.LoadedKMod != nil {
match, err := rule.LoadedKMod.Match()
if err != nil {
return false, err
}
if !match {
continue
}
}
return true, nil
}
return false, nil
}

View file

@ -0,0 +1,61 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package rules
import (
"fmt"
"io/ioutil"
"strings"
)
// Rule that matches on loaded kernel modules in the system
type LoadedKModRule []string
const kmodProcfsPath = "/proc/modules"
// Match loaded kernel modules on provided list of kernel modules
func (kmods *LoadedKModRule) Match() (bool, error) {
loadedModules, err := kmods.getLoadedModules()
if err != nil {
return false, fmt.Errorf("failed to get loaded kernel modules. %s", err.Error())
}
for _, kmod := range *kmods {
if _, ok := loadedModules[kmod]; !ok {
// kernel module not loaded
return false, nil
}
}
return true, nil
}
func (kmods *LoadedKModRule) getLoadedModules() (map[string]struct{}, error) {
out, err := ioutil.ReadFile(kmodProcfsPath)
if err != nil {
return nil, fmt.Errorf("failed to read file %s: %s", kmodProcfsPath, err.Error())
}
loadedMods := make(map[string]struct{})
for _, line := range strings.Split(string(out), "\n") {
// skip empty lines
if len(line) == 0 {
continue
}
// append loaded module
loadedMods[strings.Fields(line)[0]] = struct{}{}
}
return loadedMods, nil
}

View file

@ -0,0 +1,87 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package rules
import (
"fmt"
pciutils "sigs.k8s.io/node-feature-discovery/source/internal"
)
// Rule that matches on the following PCI device attributes: <class, vendor, device>
// each device attribute will be a list elements(strings).
// Match operation: OR will be performed per element and AND will be performed per attribute.
// An empty attribute will not be included in the matching process.
type PciIdRuleInput struct {
Class []string `json:"class,omitempty"`
Vendor []string `json:"vendor,omitempty"`
Device []string `json:"device,omitempty"`
}
type PciIdRule struct {
PciIdRuleInput
}
// Match PCI devices on provided PCI device attributes
func (r *PciIdRule) Match() (bool, error) {
devAttr := map[string]bool{}
for _, attr := range []string{"class", "vendor", "device"} {
devAttr[attr] = true
}
allDevs, err := pciutils.DetectPci(devAttr)
if err != nil {
return false, fmt.Errorf("failed to detect PCI devices: %s", err.Error())
}
for _, classDevs := range allDevs {
for _, dev := range classDevs {
// match rule on a single device
if r.matchDevOnRule(dev) {
return true, nil
}
}
}
return false, nil
}
func (r *PciIdRule) matchDevOnRule(dev pciutils.PciDeviceInfo) bool {
if len(r.Class) == 0 && len(r.Vendor) == 0 && len(r.Device) == 0 {
return false
}
if len(r.Class) > 0 && !in(dev["class"], r.Class) {
return false
}
if len(r.Vendor) > 0 && !in(dev["vendor"], r.Vendor) {
return false
}
if len(r.Device) > 0 && !in(dev["device"], r.Device) {
return false
}
return true
}
func in(item string, arr []string) bool {
for _, val := range arr {
if val == item {
return true
}
}
return false
}

View file

@ -0,0 +1,22 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package rules
type Rule interface {
// Match on rule
Match() (bool, error)
}

View file

@ -0,0 +1,46 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package custom
import (
"sigs.k8s.io/node-feature-discovery/source/custom/rules"
)
// getStaticFeatures returns statically configured custom features to discover
// e.g RMDA related features. NFD configuration file may extend these custom features by adding rules.
func getStaticFeatureConfig() []CustomFeature {
return []CustomFeature{
CustomFeature{
Name: "rdma.capable",
MatchOn: []MatchRule{
MatchRule{
PciId: &rules.PciIdRule{
rules.PciIdRuleInput{Vendor: []string{"15b3"}},
},
},
},
},
CustomFeature{
Name: "rdma.available",
MatchOn: []MatchRule{
MatchRule{
LoadedKMod: &rules.LoadedKModRule{"ib_uverbs", "rdma_ucm"},
},
},
},
}
}

View file

@ -0,0 +1,98 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package pciutils
import (
"fmt"
"io/ioutil"
"log"
"path"
"strings"
)
type PciDeviceInfo map[string]string
var DefaultPciDevAttrs = []string{"class", "vendor", "device", "subsystem_vendor", "subsystem_device"}
var ExtraPciDevAttrs = []string{"sriov_totalvfs"}
// Read a single PCI device attribute
// A PCI attribute in this context, maps to the corresponding sysfs file
func readSingleAttribute(devPath string, attrName string) (string, error) {
data, err := ioutil.ReadFile(path.Join(devPath, attrName))
if err != nil {
return "", fmt.Errorf("failed to read device attribute %s: %v", attrName, err)
}
// Strip whitespace and '0x' prefix
attrVal := strings.TrimSpace(strings.TrimPrefix(string(data), "0x"))
if attrName == "class" && len(attrVal) > 4 {
// Take four first characters, so that the programming
// interface identifier gets stripped from the raw class code
attrVal = attrVal[0:4]
}
return attrVal, nil
}
// Read information of one PCI device
func readDevInfo(devPath string, deviceAttrSpec map[string]bool) (PciDeviceInfo, error) {
info := PciDeviceInfo{}
for attr, must := range deviceAttrSpec {
attrVal, err := readSingleAttribute(devPath, attr)
if err != nil {
if must {
return info, fmt.Errorf("Failed to read device %s: %s", attr, err)
} else {
continue
}
}
info[attr] = attrVal
}
return info, nil
}
// List available PCI devices and retrieve device attributes.
// deviceAttrSpec is a map which specifies which attributes to retrieve.
// a false value for a specific attribute marks the attribute as optional.
// a true value for a specific attribute marks the attribute as mandatory.
// "class" attribute is considered mandatory.
// DetectPci() will fail if the retrieval of a mandatory attribute fails.
func DetectPci(deviceAttrSpec map[string]bool) (map[string][]PciDeviceInfo, error) {
const basePath = "/sys/bus/pci/devices/"
devInfo := make(map[string][]PciDeviceInfo)
devices, err := ioutil.ReadDir(basePath)
if err != nil {
return nil, err
}
// "class" is a mandatory attribute, inject it to spec if needed.
deviceAttrSpec["class"] = true
// Iterate over devices
for _, device := range devices {
info, err := readDevInfo(path.Join(basePath, device.Name()), deviceAttrSpec)
if err != nil {
log.Print(err)
continue
}
class := info["class"]
devInfo[class] = append(devInfo[class], info)
}
return devInfo, nil
}

View file

@ -18,16 +18,13 @@ package pci
import (
"fmt"
"io/ioutil"
"log"
"path"
"strings"
"sigs.k8s.io/node-feature-discovery/source"
"sigs.k8s.io/node-feature-discovery/source/internal"
)
type pciDeviceInfo map[string]string
type NFDConfig struct {
DeviceClassWhitelist []string `json:"deviceClassWhitelist,omitempty"`
DeviceLabelFields []string `json:"deviceLabelFields,omitempty"`
@ -38,9 +35,6 @@ var Config = NFDConfig{
DeviceLabelFields: []string{"class", "vendor"},
}
var devLabelAttrs = []string{"class", "vendor", "device", "subsystem_vendor", "subsystem_device"}
var extraDevAttrs = []string{"sriov_totalvfs"}
// Implement FeatureSource interface
type Source struct{}
@ -58,7 +52,7 @@ func (s Source) Discover() (source.Features, error) {
configLabelFields[field] = true
}
for _, attr := range devLabelAttrs {
for _, attr := range pciutils.DefaultPciDevAttrs {
if _, ok := configLabelFields[attr]; ok {
deviceLabelFields = append(deviceLabelFields, attr)
delete(configLabelFields, attr)
@ -79,14 +73,14 @@ func (s Source) Discover() (source.Features, error) {
// Read extraDevAttrs + configured or default labels. Attributes
// set to 'true' are considered must-have.
deviceAttrs := map[string]bool{}
for _, label := range extraDevAttrs {
for _, label := range pciutils.ExtraPciDevAttrs {
deviceAttrs[label] = false
}
for _, label := range deviceLabelFields {
deviceAttrs[label] = true
}
devs, err := detectPci(deviceAttrs)
devs, err := pciutils.DetectPci(deviceAttrs)
if err != nil {
return nil, fmt.Errorf("Failed to detect PCI devices: %s", err.Error())
}
@ -112,54 +106,5 @@ func (s Source) Discover() (source.Features, error) {
}
}
}
return features, nil
}
// Read information of one PCI device
func readDevInfo(devPath string, devAttrs map[string]bool) (pciDeviceInfo, error) {
info := pciDeviceInfo{}
for attr, must := range devAttrs {
data, err := ioutil.ReadFile(path.Join(devPath, attr))
if err != nil {
if must {
return info, fmt.Errorf("Failed to read device %s: %s", attr, err)
} else {
continue
}
}
// Strip whitespace and '0x' prefix
info[attr] = strings.TrimSpace(strings.TrimPrefix(string(data), "0x"))
if attr == "class" && len(info[attr]) > 4 {
// Take four first characters, so that the programming
// interface identifier gets stripped from the raw class code
info[attr] = info[attr][0:4]
}
}
return info, nil
}
// List available PCI devices
func detectPci(devAttrs map[string]bool) (map[string][]pciDeviceInfo, error) {
const basePath = "/sys/bus/pci/devices/"
devInfo := make(map[string][]pciDeviceInfo)
devices, err := ioutil.ReadDir(basePath)
if err != nil {
return nil, err
}
// Iterate over devices
for _, device := range devices {
info, err := readDevInfo(path.Join(basePath, device.Name()), devAttrs)
if err != nil {
log.Print(err)
continue
}
class := info["class"]
devInfo[class] = append(devInfo[class], info)
}
return devInfo, nil
}