usb: Add support for USB device discovery

This builds on the PCI support to enable the discovery of USB devices. This is primarily intended to be used for the discovery of Edge-based heterogeneous accelerators that are connected via USB, such as the Coral USB Accelerator and the Intel NCS2 - our main motivation for adding this capability to NFD, and as part of our work in the SODALITE H2020 project. USB devices may define their base class at either the device or interface levels. In the case where no device class is set, the per-device interfaces are enumerated instead. USB devices may furthermore have multiple interfaces, which may or may not use the identical class across each interface. We therefore report device existence for each unique class definition to enable more fine-grained labelling and node selection. The default labelling format includes the class, vendor and device (product) IDs, as follows: feature.node.kubernetes.io/usb-fe_1a6e_089a.present=true As with PCI, a subset of device classes are whitelisted for matching. By default, there are only a subset of device classes under which accelerators tend to be mapped, which is used as the basis for the whitelist. These are: - Video - Miscellaneous - Application Specific - Vendor Specific For those interested in matching other classes, this may be extended by using the UsbId rule provided through the custom source. A full list of class codes is provided by the USB-IF at: https://www.usb.org/defined-class-codes For the moment, owing to a lack of a demonstrable use case, neither the subclass nor the protocol information are exposed. If this becomes necessary, support for these attributes can be trivially added. Signed-off-by: Paul Mundt <paul.mundt@adaptant.io>
2024-12-14 11:57:51 +00:00 · 2020-05-14 22:32:55 +02:00 · 2020-05-14 22:32:55 +02:00 · c0ea69411b
commit c0ea69411b
parent 1bdf0525f9
11 changed files with 412 additions and 9 deletions
--- a/README.md
+++ b/README.md
@ -129,7 +129,7 @@ nfd-worker.
                              in testing
                              [Default: ]
  --sources=<sources>         Comma separated list of feature sources.
-                              [Default: cpu,iommu,kernel,local,memory,network,pci,storage,system]
+                              [Default: cpu,iommu,kernel,local,memory,network,pci,storage,system,usb]
  --no-publish                Do not publish discovered features to the
                              cluster-local Kubernetes API server.
  --label-whitelist=<pattern> Regular expression to filter label names to
@ -161,6 +161,7 @@ The current set of feature sources are the following:
 - PCI
 - Storage
 - System
+- USB
 - Local (hooks for user-specific features)

 ### Feature labels
@ -193,6 +194,7 @@ feature logically has sub-hierarchy, e.g. `sriov.capable` and
  "feature.node.kubernetes.io/pci-<device label>.present": "true",
  "feature.node.kubernetes.io/storage-<feature-name>": "true",
  "feature.node.kubernetes.io/system-<feature name>": "<feature value>",
+  "feature.node.kubernetes.io/usb-<device label>.present": "<feature value>",
  "feature.node.kubernetes.io/<file name>-<feature name>": "<feature value>"
 }
 ```
@ -316,6 +318,28 @@ Matching is done by performing a logical _OR_ between Elements of an Attribute a
 each PCI device in the system.
 At least one Attribute must be specified. Missing attributes will not partake in the matching process.

+##### UsbId Rule
+###### Nomenclature
+```
+Attribute   :A USB attribute.
+Element     :An identifier of the USB attribute.
+```
+
+The UsbId Rule allows matching the USB devices in the system on the following Attributes: `class`,`vendor` and
+`device`. A list of Elements is provided for each Attribute.
+
+###### Format
+```yaml
+usbId :
+  class: [<class id>, ...]
+  vendor: [<vendor id>,  ...]
+  device: [<device id>, ...]
+```
+
+Matching is done by performing a logical _OR_ between Elements of an Attribute and logical _AND_ between the specified Attributes for
+each USB device in the system.
+At least one Attribute must be specified. Missing attributes will not partake in the matching process.
+
 ##### LoadedKMod Rule
 ###### Nomenclature
 ```
@ -342,6 +366,11 @@ custom:
      - pciId:
          vendor: ["15b3"]
          device: ["1014", "1017"]
+  - name: "my.usb.feature"
+    matchOn:
+      - usbId:
+          vendor: ["1d6b"]
+          device: ["0003"]
  - name: "my.combined.feature"
    matchOn:
      - loadedKMod : ["vendor_kmod1", "vendor_kmod2"]
@ -361,6 +390,8 @@ __In the example above:__
 if the node has `kmod1` _AND_ `kmod2` kernel modules loaded. 
 - A node would contain the label: `feature.node.kubernetes.io/custom-my.pci.feature=true`
 if the node contains a PCI device with a PCI vendor ID of `15b3` _AND_ PCI device ID of `1014` _OR_ `1017`.
+- A node would contain the label: `feature.node.kubernetes.io/custom-my.usb.feature=true`
+if the node contains a USB device with a USB vendor ID of `1d6b` _AND_ USB device ID of `0003`.
 - A node would contain the label: `feature.node.kubernetes.io/custom-my.combined.feature=true`
 if `vendor_kmod1` _AND_ `vendor_kmod2` kernel modules are loaded __AND__ the node contains a PCI device
 with a PCI vendor ID of `15b3` _AND_ PCI device ID of `1014` _or_ `1017`.
@ -433,6 +464,21 @@ Also  the set of PCI device classes that the feature source detects is
 configurable. By default, device classes (0x)03, (0x)0b40 and (0x)12, i.e.
 GPUs, co-processors and accelerator cards are detected.

+### USB Features
+
+| Feature              | Attribute     | Description                               |
+| -------------------- | ------------- | ----------------------------------------- |
+| &lt;device label&gt; | present       | USB device is detected
+
+`<device label>` is composed of raw USB IDs, separated by underscores.
+The set of fields used in `<device label>` is configurable, valid fields being
+`class`, `vendor`, and `device`.
+Defaults are `class`, `vendor` and `device`. An example label using the default
+label fields:
+```
+feature.node.kubernetes.io/usb-fe_1a6e_089a.present=true
+```
+
 See [configuration options](#configuration-options)
 for more information on NFD config.

--- a/cmd/nfd-worker/main.go
+++ b/cmd/nfd-worker/main.go
@ -91,7 +91,7 @@ func argsParse(argv []string) (worker.Args, error) {
                              in testing
                              [Default: ]
  --sources=<sources>         Comma separated list of feature sources.
-                              [Default: cpu,custom,iommu,kernel,local,memory,network,pci,storage,system]
+                              [Default: cpu,custom,iommu,kernel,local,memory,network,pci,storage,system,usb]
  --no-publish                Do not publish discovered features to the
                              cluster-local Kubernetes API server.
  --label-whitelist=<pattern> Regular expression to filter label names to
--- a/cmd/nfd-worker/main_test.go
+++ b/cmd/nfd-worker/main_test.go
@ -23,7 +23,7 @@ import (
 	. "github.com/smartystreets/goconvey/convey"
 )

-var allSources = []string{"cpu", "custom", "iommu", "kernel", "local", "memory", "network", "pci", "storage", "system"}
+var allSources = []string{"cpu", "custom", "iommu", "kernel", "local", "memory", "network", "pci", "storage", "system", "usb"}

 func TestArgsParse(t *testing.T) {
 	Convey("When parsing command line arguments", t, func() {
--- a/nfd-worker.conf.example
+++ b/nfd-worker.conf.example
@ -44,6 +44,16 @@
 #      - "device"
 #      - "subsystem_vendor"
 #      - "subsystem_device"
+#  usb:
+#    deviceClassWhitelist:
+#      - "0e"
+#      - "ef"
+#      - "fe"
+#      - "ff"
+#    deviceLabelFields:
+#      - "class"
+#      - "vendor"
+#      - "device"
 #  custom:
 #    - name: "my.kernel.feature"
 #      matchOn:
@ -57,6 +67,16 @@
 #        - pciId :
 #            vendor: ["8086"]
 #            device: ["1000", "1100"]
+#    - name: "my.usb.feature"
+#      matchOn:
+#        - usbId:
+#          class: ["ff"]
+#          vendor: ["03e7"]
+#          device: ["2485"]
+#        - usbId:
+#          class: ["fe"]
+#          vendor: ["1a6e"]
+#          device: ["089a"]
 #    - name: "my.combined.feature"
 #      matchOn:
 #        - pciId:
--- a/pkg/nfd-worker/nfd-worker.go
+++ b/pkg/nfd-worker/nfd-worker.go
@ -46,6 +46,7 @@ import (
 	"sigs.k8s.io/node-feature-discovery/source/pci"
 	"sigs.k8s.io/node-feature-discovery/source/storage"
 	"sigs.k8s.io/node-feature-discovery/source/system"
+	"sigs.k8s.io/node-feature-discovery/source/usb"
 	"sigs.k8s.io/yaml"
 )

@ -62,6 +63,7 @@ type NFDConfig struct {
 		Cpu    *cpu.NFDConfig    `json:"cpu,omitempty"`
 		Kernel *kernel.NFDConfig `json:"kernel,omitempty"`
 		Pci    *pci.NFDConfig    `json:"pci,omitempty"`
+		Usb    *usb.NFDConfig    `json:"usb,omitempty"`
 		Custom *custom.NFDConfig `json:"custom,omitempty"`
 	} `json:"sources,omitempty"`
 }
@ -236,6 +238,7 @@ func configParse(filepath string, overrides string) error {
 	config.Sources.Cpu = &cpu.Config
 	config.Sources.Kernel = &kernel.Config
 	config.Sources.Pci = &pci.Config
+	config.Sources.Usb = &usb.Config
 	config.Sources.Custom = &custom.Config

 	data, err := ioutil.ReadFile(filepath)
@ -279,6 +282,7 @@ func configureParameters(sourcesWhiteList []string, labelWhiteListStr string) (e
 		pci.Source{},
 		storage.Source{},
 		system.Source{},
+		usb.Source{},
 		custom.Source{},
 		// local needs to be the last source so that it is able to override
 		// labels from other sources
--- a/source/custom/custom.go
+++ b/source/custom/custom.go
@ -27,6 +27,7 @@ import (
 // Custom Features Configurations
 type MatchRule struct {
 	PciId      *rules.PciIdRule      `json:"pciId,omitempty""`
+	UsbId      *rules.UsbIdRule      `json:"usbId,omitempty""`
 	LoadedKMod *rules.LoadedKModRule `json:"loadedKMod,omitempty""`
 }

@ -77,6 +78,16 @@ func (s Source) discoverFeature(feature CustomFeature) (bool, error) {
 				continue
 			}
 		}
+		// USB ID rule
+		if rule.UsbId != nil {
+			match, err := rule.UsbId.Match()
+			if err != nil {
+				return false, err
+			}
+			if !match {
+				continue
+			}
+		}
 		// Loaded kernel module rule
 		if rule.LoadedKMod != nil {
 			match, err := rule.LoadedKMod.Match()
--- a/source/custom/rules/usb_id_rule.go
+++ b/source/custom/rules/usb_id_rule.go
@ -0,0 +1,79 @@
+/*
+Copyright 2020 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package rules
+
+import (
+	"fmt"
+	usbutils "sigs.k8s.io/node-feature-discovery/source/internal"
+)
+
+// Rule that matches on the following USB device attributes: <class, vendor, device>
+// each device attribute will be a list elements(strings).
+// Match operation: OR will be performed per element and AND will be performed per attribute.
+// An empty attribute will not be included in the matching process.
+type UsbIdRuleInput struct {
+	Class  []string `json:"class,omitempty"`
+	Vendor []string `json:"vendor,omitempty"`
+	Device []string `json:"device,omitempty"`
+}
+
+type UsbIdRule struct {
+	UsbIdRuleInput
+}
+
+// Match USB devices on provided USB device attributes
+func (r *UsbIdRule) Match() (bool, error) {
+	devAttr := map[string]bool{}
+	for _, attr := range []string{"class", "vendor", "device"} {
+		devAttr[attr] = true
+	}
+	allDevs, err := usbutils.DetectUsb(devAttr)
+	if err != nil {
+		return false, fmt.Errorf("failed to detect USB devices: %s", err.Error())
+	}
+
+	for _, classDevs := range allDevs {
+		for _, dev := range classDevs {
+			// match rule on a single device
+			if r.matchDevOnRule(dev) {
+				return true, nil
+			}
+		}
+	}
+	return false, nil
+}
+
+func (r *UsbIdRule) matchDevOnRule(dev usbutils.UsbDeviceInfo) bool {
+	if len(r.Class) == 0 && len(r.Vendor) == 0 && len(r.Device) == 0 {
+		return false
+	}
+
+	if len(r.Class) > 0 && !in(dev["class"], r.Class) {
+		return false
+	}
+
+	if len(r.Vendor) > 0 && !in(dev["vendor"], r.Vendor) {
+		return false
+	}
+
+	if len(r.Device) > 0 && !in(dev["device"], r.Device) {
+		return false
+	}
+
+	return true
+}
+
--- a/source/internal/pci_utils.go
+++ b/source/internal/pci_utils.go
@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */

-package pciutils
+package busutils

 import (
 	"fmt"
@ -31,7 +31,7 @@ var ExtraPciDevAttrs = []string{"sriov_totalvfs"}

 // Read a single PCI device attribute
 // A PCI attribute in this context, maps to the corresponding sysfs file
-func readSingleAttribute(devPath string, attrName string) (string, error) {
+func readSinglePciAttribute(devPath string, attrName string) (string, error) {
 	data, err := ioutil.ReadFile(path.Join(devPath, attrName))
 	if err != nil {
 		return "", fmt.Errorf("failed to read device attribute %s: %v", attrName, err)
@ -48,11 +48,11 @@ func readSingleAttribute(devPath string, attrName string) (string, error) {
 }

 // Read information of one PCI device
-func readDevInfo(devPath string, deviceAttrSpec map[string]bool) (PciDeviceInfo, error) {
+func readPciDevInfo(devPath string, deviceAttrSpec map[string]bool) (PciDeviceInfo, error) {
 	info := PciDeviceInfo{}

 	for attr, must := range deviceAttrSpec {
-		attrVal, err := readSingleAttribute(devPath, attr)
+		attrVal, err := readSinglePciAttribute(devPath, attr)
 		if err != nil {
 			if must {
 				return info, fmt.Errorf("Failed to read device %s: %s", attr, err)
@ -85,7 +85,7 @@ func DetectPci(deviceAttrSpec map[string]bool) (map[string][]PciDeviceInfo, erro

 	// Iterate over devices
 	for _, device := range devices {
-		info, err := readDevInfo(path.Join(basePath, device.Name()), deviceAttrSpec)
+		info, err := readPciDevInfo(path.Join(basePath, device.Name()), deviceAttrSpec)
 		if err != nil {
 			log.Print(err)
 			continue
--- a/source/internal/usb_utils.go
+++ b/source/internal/usb_utils.go
@ -0,0 +1,138 @@
+/*
+Copyright 2020 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package busutils
+
+import (
+	"fmt"
+	"io/ioutil"
+	"log"
+	"path"
+	"path/filepath"
+	"strings"
+)
+
+type UsbDeviceInfo map[string]string
+type UsbClassMap map[string]UsbDeviceInfo
+
+var DefaultUsbDevAttrs = []string{"class", "vendor", "device"}
+
+// The USB device sysfs files do not have terribly user friendly names, map
+// these for consistency with the PCI matcher.
+var devAttrFileMap = map[string]string{
+	"class":  "bDeviceClass",
+	"device": "idProduct",
+	"vendor": "idVendor",
+}
+
+func readSingleUsbSysfsAttribute(path string) (string, error) {
+	data, err := ioutil.ReadFile(path)
+	if err != nil {
+		return "", fmt.Errorf("failed to read device attribute %s: %v", filepath.Base(path), err)
+	}
+
+	attrVal := strings.TrimSpace(string(data))
+
+	return attrVal, nil
+}
+
+// Read a single USB device attribute
+// A USB attribute in this context, maps to the corresponding sysfs file
+func readSingleUsbAttribute(devPath string, attrName string) (string, error) {
+	return readSingleUsbSysfsAttribute(path.Join(devPath, devAttrFileMap[attrName]))
+}
+
+// Read information of one USB device
+func readUsbDevInfo(devPath string, deviceAttrSpec map[string]bool) (UsbClassMap, error) {
+	classmap := UsbClassMap{}
+	info := UsbDeviceInfo{}
+
+	for attr, _ := range deviceAttrSpec {
+		attrVal, _ := readSingleUsbAttribute(devPath, attr)
+		if len(attrVal) > 0 {
+			info[attr] = attrVal
+		}
+	}
+
+	// USB devices encode their class information either at the device or the interface level. If the device class
+	// is set, return as-is.
+	if info["class"] != "00" {
+		classmap[info["class"]] = info
+	} else {
+		// Otherwise, if a 00 is presented at the device level, descend to the interface level.
+		interfaces, err := filepath.Glob(devPath + "/*/bInterfaceClass")
+		if err != nil {
+			return classmap, err
+		}
+
+		// A device may, notably, have multiple interfaces with mixed classes, so we create a unique device for each
+		// unique interface class.
+		for _, intf := range interfaces {
+			// Determine the interface class
+			attrVal, err := readSingleUsbSysfsAttribute(intf)
+			if err != nil {
+				return classmap, err
+			}
+
+			attr := UsbDeviceInfo{}
+			for k, v := range info {
+				attr[k] = v
+			}
+
+			attr["class"] = attrVal
+			classmap[attrVal] = attr
+		}
+	}
+
+	return classmap, nil
+}
+
+// List available USB devices and retrieve device attributes.
+// deviceAttrSpec is a map which specifies which attributes to retrieve.
+// a false value for a specific attribute marks the attribute as optional.
+// a true value for a specific attribute marks the attribute as mandatory.
+// "class" attribute is considered mandatory.
+// DetectUsb() will fail if the retrieval of a mandatory attribute fails.
+func DetectUsb(deviceAttrSpec map[string]bool) (map[string][]UsbDeviceInfo, error) {
+	// Unlike PCI, the USB sysfs interface includes entries not just for
+	// devices. We work around this by globbing anything that includes a
+	// valid product ID.
+	const devicePath = "/sys/bus/usb/devices/*/idProduct"
+	devInfo := make(map[string][]UsbDeviceInfo)
+
+	devices, err := filepath.Glob(devicePath)
+	if err != nil {
+		return nil, err
+	}
+
+	// "class" is a mandatory attribute, inject it to spec if needed.
+	deviceAttrSpec["class"] = true
+
+	// Iterate over devices
+	for _, device := range devices {
+		devMap, err := readUsbDevInfo(filepath.Dir(device), deviceAttrSpec)
+		if err != nil {
+			log.Print(err)
+			continue
+		}
+
+		for class, info := range devMap {
+			devInfo[class] = append(devInfo[class], info)
+		}
+	}
+
+	return devInfo, nil
+}
--- a/source/pci/pci.go
+++ b/source/pci/pci.go
@ -22,7 +22,7 @@ import (
 	"strings"

 	"sigs.k8s.io/node-feature-discovery/source"
-	"sigs.k8s.io/node-feature-discovery/source/internal"
+	pciutils "sigs.k8s.io/node-feature-discovery/source/internal"
 )

 type NFDConfig struct {
--- a/source/usb/usb.go
+++ b/source/usb/usb.go
@ -0,0 +1,105 @@
+/*
+Copyright 2020 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package usb
+
+import (
+	"fmt"
+	"log"
+	"strings"
+
+	"sigs.k8s.io/node-feature-discovery/source"
+	usbutils "sigs.k8s.io/node-feature-discovery/source/internal"
+)
+
+type NFDConfig struct {
+	DeviceClassWhitelist []string `json:"deviceClassWhitelist,omitempty"`
+	DeviceLabelFields    []string `json:"deviceLabelFields,omitempty"`
+}
+
+var Config = NFDConfig{
+	// Whitelist specific USB classes: https://www.usb.org/defined-class-codes
+	// By default these include classes where different accelerators are typically mapped:
+	// Video (0e), Miscellaneous (ef), Application Specific (fe), and Vendor Specific (ff).
+	DeviceClassWhitelist: []string{"0e", "ef", "fe", "ff"},
+	DeviceLabelFields:    []string{"class", "vendor", "device"},
+}
+
+// Implement FeatureSource interface
+type Source struct{}
+
+// Return name of the feature source
+func (s Source) Name() string { return "usb" }
+
+// Discover features
+func (s Source) Discover() (source.Features, error) {
+	features := source.Features{}
+
+	// Construct a device label format, a sorted list of valid attributes
+	deviceLabelFields := []string{}
+	configLabelFields := map[string]bool{}
+	for _, field := range Config.DeviceLabelFields {
+		configLabelFields[field] = true
+	}
+
+	for _, attr := range usbutils.DefaultUsbDevAttrs {
+		if _, ok := configLabelFields[attr]; ok {
+			deviceLabelFields = append(deviceLabelFields, attr)
+			delete(configLabelFields, attr)
+		}
+	}
+	if len(configLabelFields) > 0 {
+		keys := []string{}
+		for key := range configLabelFields {
+			keys = append(keys, key)
+		}
+		log.Printf("WARNING: invalid fields '%v' in deviceLabelFields, ignoring...", keys)
+	}
+	if len(deviceLabelFields) == 0 {
+		log.Printf("WARNING: no valid fields in deviceLabelFields defined, using the defaults")
+		deviceLabelFields = []string{"vendor", "device"}
+	}
+
+	// Read configured or default labels. Attributes set to 'true' are considered must-have.
+	deviceAttrs := map[string]bool{}
+	for _, label := range deviceLabelFields {
+		deviceAttrs[label] = true
+	}
+
+	devs, err := usbutils.DetectUsb(deviceAttrs)
+	if err != nil {
+		return nil, fmt.Errorf("Failed to detect USB devices: %s", err.Error())
+	}
+
+	// Iterate over all device classes
+	for class, classDevs := range devs {
+		for _, white := range Config.DeviceClassWhitelist {
+			if strings.HasPrefix(class, strings.ToLower(white)) {
+				for _, dev := range classDevs {
+					devLabel := ""
+					for i, attr := range deviceLabelFields {
+						devLabel += dev[attr]
+						if i < len(deviceLabelFields)-1 {
+							devLabel += "_"
+						}
+					}
+					features[devLabel+".present"] = true
+				}
+			}
+		}
+	}
+	return features, nil
+}