From 655f5c5555801d649936faf31c50aa0308fe8727 Mon Sep 17 00:00:00 2001 From: Markus Lehtonen Date: Tue, 19 Feb 2019 10:52:46 +0200 Subject: [PATCH] sources: move all cpu related features under the cpu source Remove 'cpuid', 'pstate' and 'rdt' feature sources and move their functionality under the 'cpu' source. The goal is to have a more systematic organization of feature sources and labels. After this change we now basically have one source per type of hw, one for kernel and one for userspace sw. Related feature labels are changed, correspondingly, new labels being: feature.node.k8s.io/cpu-cpuid. feature.node.k8s.io/cpu-pstate.turbo feature.node.k8s.io/cpu-rdt. --- README.md | 102 ++++++++---------- cmd/nfd-worker/main.go | 2 +- cmd/nfd-worker/main_test.go | 4 +- ...od-with-discovery.yaml.cloverleaf.template | 2 +- ...mo-pod-with-discovery.yaml.parsec.template | 2 +- pkg/nfd-worker/nfd-worker.go | 6 -- source/cpu/cpu.go | 20 ++++ source/{cpuid/cpuid.go => cpu/cpuid_amd64.go} | 13 ++- source/{cpuid => cpu}/cpuid_arm64.go | 16 +-- source/{pstate => cpu}/pstate.go | 23 ++-- source/{rdt => cpu}/rdt_amd64.go | 2 +- source/{rdt => cpu}/rdt_stub.go | 2 +- source/cpuid/cpuid_amd64.go | 32 ------ source/rdt/rdt.go | 40 ------- 14 files changed, 86 insertions(+), 180 deletions(-) rename source/{cpuid/cpuid.go => cpu/cpuid_amd64.go} (74%) rename source/{cpuid => cpu}/cpuid_arm64.go (88%) rename source/{pstate => cpu}/pstate.go (66%) rename source/{rdt => cpu}/rdt_amd64.go (99%) rename source/{rdt => cpu}/rdt_stub.go (98%) delete mode 100644 source/cpuid/cpuid_amd64.go delete mode 100644 source/rdt/rdt.go diff --git a/README.md b/README.md index b0e5a43f2..af61c5b55 100644 --- a/README.md +++ b/README.md @@ -124,7 +124,7 @@ nfd-worker. in testing [Default: ] --sources= Comma separated list of feature sources. - [Default: cpu,cpuid,iommu,kernel,local,memory,network,pci,pstate,rdt,storage,system] + [Default: cpu,iommu,kernel,local,memory,network,pci,storage,system] --no-publish Do not publish discovered features to the cluster-local Kubernetes API server. --label-whitelist= Regular expression to filter label names to @@ -149,14 +149,11 @@ for up-to-date information about the required volume mounts. The current set of feature sources are the following: - CPU -- [CPUID][cpuid] for x86/Arm64 CPU details - IOMMU - Kernel - Memory - Network - PCI -- Pstate ([Intel P-State driver][intel-pstate]) -- RDT ([Intel Resource Director Technology][intel-rdt]) - Storage - System - Local (hooks for user-specific features) @@ -166,9 +163,9 @@ The current set of feature sources are the following: The published node labels encode a few pieces of information: - Namespace, i.e. `feature.node.kubernetes.io` -- The source for each label (e.g. `cpuid`). +- The source for each label (e.g. `cpu`). - The name of the discovered feature as it appears in the underlying - source, (e.g. `AESNI` from cpuid). + source, (e.g. `cpuid.AESNI` from cpu). - The value of the discovered feature. Feature label names adhere to the following pattern: @@ -185,14 +182,11 @@ the only label value published for features is the string `"true"`._ ```json { "feature.node.kubernetes.io/cpu-": "true", - "feature.node.kubernetes.io/cpuid-": "true", "feature.node.kubernetes.io/iommu-": "true", "feature.node.kubernetes.io/kernel-": "", "feature.node.kubernetes.io/memory-": "true", "feature.node.kubernetes.io/network-": "true", "feature.node.kubernetes.io/pci-.present": "true", - "feature.node.kubernetes.io/pstate-": "true", - "feature.node.kubernetes.io/rdt-": "true", "feature.node.kubernetes.io/storage-": "true", "feature.node.kubernetes.io/system-": "", "feature.node.kubernetes.io/-": "" @@ -208,43 +202,50 @@ such as restricting discovered features with the --label-whitelist option._ ### CPU Features -The CPU feature source differs from the CPUID feature source in that it -discovers CPU related features that are actually enabled, whereas CPUID only -reports *supported* CPU capabilities (i.e. a capability might be supported but -not enabled) as reported by the `cpuid` instruction. +| Feature name | Attribute | Description | +| ----------------------- | ------------------ | ----------------------------- | +| cpuid | <cpuid flag> | CPU capability is supported +| hardware_multithreading |
| Hardware multithreading, such as Intel HTT, enabled (number of locical CPUs is greater than physical CPUs) +| power | sst_bf.enabled | Intel SST-BF ([Intel Speed Select Technology][intel-sst] - Base frequency) enabled +| [pstate][intel-pstate] | turbo | Turbo frequencies are enabled in Intel pstate driver +| [rdt][intel-rdt] | RDTMON | Intel RDT Monitoring Technology +|
| RDTCMT | Intel Cache Monitoring (CMT) +|
| RDTMBM | Intel Memory Bandwidth Monitoring (MBM) +|
| RDTL3CA | Intel L3 Cache Allocation Technology +|
| RDTL2CA | Intel L2 Cache Allocation Technology +|
| RDTMBA | Intel Memory Bandwidth Allocation (MBA) Technology -| Feature | Attribute | Description | -| ----------------------- | -------------- | ----------------------------------| -| hardware_multithreading |
| Hardware multithreading, such as Intel HTT, enabled (number of locical CPUs is greater than physical CPUs) -| power | sst_bf.enabled | Intel SST-BF ([Intel Speed Select Technology][intel-sst] - Base frequency) enabled +**NOTE** The cpuid features advertise *supported* CPU capabilities, that is, a +capability might be supported but not enabled. -### X86 CPUID Features (Partial List) -| Feature name | Description | -| :------------: | :----------------------------------------------------------: | -| ADX | Multi-Precision Add-Carry Instruction Extensions (ADX) -| AESNI | Advanced Encryption Standard (AES) New Instructions (AES-NI) -| AVX | Advanced Vector Extensions (AVX) -| AVX2 | Advanced Vector Extensions 2 (AVX2) -| BMI1 | Bit Manipulation Instruction Set 1 (BMI) -| BMI2 | Bit Manipulation Instruction Set 2 (BMI2) -| SSE4.1 | Streaming SIMD Extensions 4.1 (SSE4.1) -| SSE4.2 | Streaming SIMD Extensions 4.2 (SSE4.2) -| SGX | Software Guard Extensions (SGX) +#### X86 CPUID Attributes (Partial List) -### Arm64 CPUID Features (Partial List) +| Attribute | Description | +| --------- | ---------------------------------------------------------------- | +| ADX | Multi-Precision Add-Carry Instruction Extensions (ADX) +| AESNI | Advanced Encryption Standard (AES) New Instructions (AES-NI) +| AVX | Advanced Vector Extensions (AVX) +| AVX2 | Advanced Vector Extensions 2 (AVX2) +| BMI1 | Bit Manipulation Instruction Set 1 (BMI) +| BMI2 | Bit Manipulation Instruction Set 2 (BMI2) +| SSE4.1 | Streaming SIMD Extensions 4.1 (SSE4.1) +| SSE4.2 | Streaming SIMD Extensions 4.2 (SSE4.2) +| SGX | Software Guard Extensions (SGX) -| Feature name | Description | -| :------------: | :----------------------------------------------------------: | -| AES | Announcing the Advanced Encryption Standard -| EVSTRM | Event Stream Frequency Features -| FPHP | Half Precision(16bit) Floating Point Data Processing Instructions -| ASIMDHP | Half Precision(16bit) Asimd Data Processing Instructions -| ATOMICS | Atomic Instructions to the A64 -| ASIMRDM | Support for Rounding Double Multiply Add/Subtract -| PMULL | Optional Cryptographic and CRC32 Instructions -| JSCVT | Perform Conversion to Match Javascript -| DCPOP | Persistent Memory Support +#### Arm64 CPUID Attribute (Partial List) + +| Attribute | Description | +| --------- | ---------------------------------------------------------------- | +| AES | Announcing the Advanced Encryption Standard +| EVSTRM | Event Stream Frequency Features +| FPHP | Half Precision(16bit) Floating Point Data Processing Instructions +| ASIMDHP | Half Precision(16bit) Asimd Data Processing Instructions +| ATOMICS | Atomic Instructions to the A64 +| ASIMRDM | Support for Rounding Double Multiply Add/Subtract +| PMULL | Optional Cryptographic and CRC32 Instructions +| JSCVT | Perform Conversion to Match Javascript +| DCPOP | Persistent Memory Support ### IOMMU Features @@ -267,12 +268,6 @@ Kernel config file to use, and, the set of config options to be detected are configurable. See [configuration options](#configuration-options) for more information. -### P-State Features - -| Feature name | Description | -| :----------: | ------------------------------------------------------------- | -| turbo | Turbo frequencies are enabled in Intel pstate driver - ### Memory Features | Feature | Attribute | Description | @@ -309,17 +304,6 @@ GPUs, co-processors and accelerator cards are detected. See [configuration options](#configuration-options) for more information on NFD config. -### RDT (Intel Resource Director Technology) Features - -| Feature name | Description | -| :------------: | :---------------------------------------------------------------------------------: | -| RDTMON | Intel RDT Monitoring Technology -| RDTCMT | Intel Cache Monitoring (CMT) -| RDTMBM | Intel Memory Bandwidth Monitoring (MBM) -| RDTL3CA | Intel L3 Cache Allocation Technology -| RDTL2CA | Intel L2 Cache Allocation Technology -| RDTMBA | Intel Memory Bandwidth Allocation (MBA) Technology - ### Storage Features | Feature name | Description | @@ -629,7 +613,7 @@ spec: - image: golang name: go1 nodeSelector: - feature.node.kubernetes.io/pstate-turbo: 'true' + feature.node.kubernetes.io/cpu-pstate.turbo: 'true' ``` For more details on targeting nodes, see [node selection][node-sel]. diff --git a/cmd/nfd-worker/main.go b/cmd/nfd-worker/main.go index 15e35318b..b11b2f4c9 100644 --- a/cmd/nfd-worker/main.go +++ b/cmd/nfd-worker/main.go @@ -91,7 +91,7 @@ func argsParse(argv []string) (worker.Args, error) { in testing [Default: ] --sources= Comma separated list of feature sources. - [Default: cpu,cpuid,iommu,kernel,local,memory,network,pci,pstate,rdt,storage,system] + [Default: cpu,iommu,kernel,local,memory,network,pci,storage,system] --no-publish Do not publish discovered features to the cluster-local Kubernetes API server. --label-whitelist= Regular expression to filter label names to diff --git a/cmd/nfd-worker/main_test.go b/cmd/nfd-worker/main_test.go index 8f8dc6874..05824fa8d 100644 --- a/cmd/nfd-worker/main_test.go +++ b/cmd/nfd-worker/main_test.go @@ -32,7 +32,7 @@ func TestArgsParse(t *testing.T) { So(args.SleepInterval, ShouldEqual, 60*time.Second) So(args.NoPublish, ShouldBeTrue) So(args.Oneshot, ShouldBeTrue) - So(args.Sources, ShouldResemble, []string{"cpu", "cpuid", "iommu", "kernel", "local", "memory", "network", "pci", "pstate", "rdt", "storage", "system"}) + So(args.Sources, ShouldResemble, []string{"cpu", "iommu", "kernel", "local", "memory", "network", "pci", "storage", "system"}) So(len(args.LabelWhiteList), ShouldEqual, 0) So(err, ShouldBeNil) }) @@ -56,7 +56,7 @@ func TestArgsParse(t *testing.T) { Convey("args.labelWhiteList is set to appropriate value and args.sources is set to default value", func() { So(args.NoPublish, ShouldBeFalse) - So(args.Sources, ShouldResemble, []string{"cpu", "cpuid", "iommu", "kernel", "local", "memory", "network", "pci", "pstate", "rdt", "storage", "system"}) + So(args.Sources, ShouldResemble, []string{"cpu", "iommu", "kernel", "local", "memory", "network", "pci", "storage", "system"}) So(args.LabelWhiteList, ShouldResemble, ".*rdt.*") So(err, ShouldBeNil) }) diff --git a/demo/helper-scripts/demo-pod-with-discovery.yaml.cloverleaf.template b/demo/helper-scripts/demo-pod-with-discovery.yaml.cloverleaf.template index 82ec03f1d..78b5b7f0f 100644 --- a/demo/helper-scripts/demo-pod-with-discovery.yaml.cloverleaf.template +++ b/demo/helper-scripts/demo-pod-with-discovery.yaml.cloverleaf.template @@ -11,7 +11,7 @@ metadata: { "matchExpressions": [ { - "key": "feature.node.kubernetes.io/pstate-turbo", + "key": "feature.node.kubernetes.io/cpu-pstate.turbo", "operator": "DoesNotExist" } ] diff --git a/demo/helper-scripts/demo-pod-with-discovery.yaml.parsec.template b/demo/helper-scripts/demo-pod-with-discovery.yaml.parsec.template index 9822c6c4c..84aa315d9 100644 --- a/demo/helper-scripts/demo-pod-with-discovery.yaml.parsec.template +++ b/demo/helper-scripts/demo-pod-with-discovery.yaml.parsec.template @@ -10,5 +10,5 @@ spec: - containerPort: 3351 hostPort: 10001 nodeSelector: - feature.node.kubernetes.io/pstate-turbo: 'true' + feature.node.kubernetes.io/cpu-pstate.turbo: 'true' restartPolicy: Never diff --git a/pkg/nfd-worker/nfd-worker.go b/pkg/nfd-worker/nfd-worker.go index d2c08d695..83ddbe68e 100644 --- a/pkg/nfd-worker/nfd-worker.go +++ b/pkg/nfd-worker/nfd-worker.go @@ -36,7 +36,6 @@ import ( "sigs.k8s.io/node-feature-discovery/pkg/version" "sigs.k8s.io/node-feature-discovery/source" "sigs.k8s.io/node-feature-discovery/source/cpu" - "sigs.k8s.io/node-feature-discovery/source/cpuid" "sigs.k8s.io/node-feature-discovery/source/fake" "sigs.k8s.io/node-feature-discovery/source/iommu" "sigs.k8s.io/node-feature-discovery/source/kernel" @@ -45,8 +44,6 @@ import ( "sigs.k8s.io/node-feature-discovery/source/network" "sigs.k8s.io/node-feature-discovery/source/panic_fake" "sigs.k8s.io/node-feature-discovery/source/pci" - "sigs.k8s.io/node-feature-discovery/source/pstate" - "sigs.k8s.io/node-feature-discovery/source/rdt" "sigs.k8s.io/node-feature-discovery/source/storage" "sigs.k8s.io/node-feature-discovery/source/system" ) @@ -235,7 +232,6 @@ func configureParameters(sourcesWhiteList []string, labelWhiteListStr string) (e // Configure feature sources. allSources := []source.FeatureSource{ cpu.Source{}, - cpuid.Source{}, fake.Source{}, iommu.Source{}, kernel.Source{}, @@ -243,8 +239,6 @@ func configureParameters(sourcesWhiteList []string, labelWhiteListStr string) (e network.Source{}, panic_fake.Source{}, pci.Source{}, - pstate.Source{}, - rdt.Source{}, storage.Source{}, system.Source{}, // local needs to be the last source so that it is able to override diff --git a/source/cpu/cpu.go b/source/cpu/cpu.go index 23c8170b4..662b7df3e 100644 --- a/source/cpu/cpu.go +++ b/source/cpu/cpu.go @@ -52,6 +52,26 @@ func (s Source) Discover() (source.Features, error) { features["power.sst_bf.enabled"] = true } + // Detect CPUID + cpuidFlags := getCpuidFlags() + for _, f := range cpuidFlags { + features["cpuid."+f] = true + } + + // Detect turbo boost + turbo, err := turboEnabled() + if err != nil { + log.Printf("ERROR: %v", err) + } else if turbo { + features["pstate.turbo"] = true + } + + // Detect RDT features + rdt := discoverRDT() + for _, f := range rdt { + features["rdt."+f] = true + } + return features, nil } diff --git a/source/cpuid/cpuid.go b/source/cpu/cpuid_amd64.go similarity index 74% rename from source/cpuid/cpuid.go rename to source/cpu/cpuid_amd64.go index 43cfc6c3e..67a919223 100644 --- a/source/cpuid/cpuid.go +++ b/source/cpu/cpuid_amd64.go @@ -14,10 +14,13 @@ See the License for the specific language governing permissions and limitations under the License. */ -package cpuid +package cpu -// Source implements FeatureSource. -type Source struct{} +import ( + "github.com/klauspost/cpuid" +) -// Name returns an identifier string for this feature source. -func (s Source) Name() string { return "cpuid" } +// Discover returns feature names for all the supported CPU features. +func getCpuidFlags() []string { + return cpuid.CPU.Features.Strings() +} diff --git a/source/cpuid/cpuid_arm64.go b/source/cpu/cpuid_arm64.go similarity index 88% rename from source/cpuid/cpuid_arm64.go rename to source/cpu/cpuid_arm64.go index 3e252201b..0f6475020 100644 --- a/source/cpuid/cpuid_arm64.go +++ b/source/cpu/cpuid_arm64.go @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package cpuid +package cpu /* #include @@ -84,7 +84,7 @@ var flagNames_arm64 = map[uint64]string{ CPU_ARM64_FEATURE_SVE: "SVE", } -func getFeaturesFromHWCAP() []string { +func getCpuidFlags() []string { r := make([]string, 0, 20) hwcap := uint64(C.gethwcap()) for i := uint(0); i < 64; i++ { @@ -96,15 +96,3 @@ func getFeaturesFromHWCAP() []string { } return r } - -// Discover returns feature names for all the supported CPU features. -func (s Source) Discover() (source.Features, error) { - // Get the cpu features as strings - features := source.Features{} - - for _, f := range getFeaturesFromHWCAP() { - features[f] = true - } - - return features, nil -} diff --git a/source/pstate/pstate.go b/source/cpu/pstate.go similarity index 66% rename from source/pstate/pstate.go rename to source/cpu/pstate.go index 209cd0512..86fcb2bbc 100644 --- a/source/pstate/pstate.go +++ b/source/cpu/pstate.go @@ -14,42 +14,31 @@ See the License for the specific language governing permissions and limitations under the License. */ -package pstate +package cpu import ( "fmt" "io/ioutil" "runtime" - - "sigs.k8s.io/node-feature-discovery/source" ) -// Source implements FeatureSource. -type Source struct{} - -// Name returns an identifier string for this feature source. -func (s Source) Name() string { return "pstate" } - // Discover returns feature names for p-state related features such as turbo boost. -func (s Source) Discover() (source.Features, error) { - features := source.Features{} - +func turboEnabled() (bool, error) { // On Arm platform, the frequency boost mechanism is software-based. // So skip pstate detection on Arm. switch runtime.GOARCH { case "arm64": - return features, nil + return false, nil } // Only looking for turbo boost for now... bytes, err := ioutil.ReadFile("/sys/devices/system/cpu/intel_pstate/no_turbo") if err != nil { - return nil, fmt.Errorf("can't detect whether turbo boost is enabled: %s", err.Error()) + return false, fmt.Errorf("can't detect whether turbo boost is enabled: %s", err.Error()) } if bytes[0] == byte('0') { - // Turbo boost is enabled. - features["turbo"] = true + return true, nil } - return features, nil + return false, nil } diff --git a/source/rdt/rdt_amd64.go b/source/cpu/rdt_amd64.go similarity index 99% rename from source/rdt/rdt_amd64.go rename to source/cpu/rdt_amd64.go index 59983bd4c..5edeb95d4 100644 --- a/source/rdt/rdt_amd64.go +++ b/source/cpu/rdt_amd64.go @@ -16,7 +16,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package rdt +package cpu import ( "sigs.k8s.io/node-feature-discovery/pkg/cpuid" diff --git a/source/rdt/rdt_stub.go b/source/cpu/rdt_stub.go similarity index 98% rename from source/rdt/rdt_stub.go rename to source/cpu/rdt_stub.go index 4b82ca25c..3dc2aec29 100644 --- a/source/rdt/rdt_stub.go +++ b/source/cpu/rdt_stub.go @@ -16,7 +16,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package rdt +package cpu func discoverRDT() []string { return []string{} diff --git a/source/cpuid/cpuid_amd64.go b/source/cpuid/cpuid_amd64.go deleted file mode 100644 index 81e6d8ad9..000000000 --- a/source/cpuid/cpuid_amd64.go +++ /dev/null @@ -1,32 +0,0 @@ -/* -Copyright 2017 The Kubernetes Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package cpuid - -import ( - "github.com/klauspost/cpuid" - "sigs.k8s.io/node-feature-discovery/source" -) - -// Discover returns feature names for all the supported CPU features. -func (s Source) Discover() (source.Features, error) { - // Get the cpu features as strings - features := source.Features{} - for _, f := range cpuid.CPU.Features.Strings() { - features[f] = true - } - return features, nil -} diff --git a/source/rdt/rdt.go b/source/rdt/rdt.go deleted file mode 100644 index 122e527ee..000000000 --- a/source/rdt/rdt.go +++ /dev/null @@ -1,40 +0,0 @@ -/* -Copyright 2017 The Kubernetes Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package rdt - -import ( - "sigs.k8s.io/node-feature-discovery/source" -) - -// Source implements FeatureSource. -type Source struct{} - -// Name returns an identifier string for this feature source. -func (s Source) Name() string { return "rdt" } - -// Discover returns feature names for CMT and CAT if supported. -func (s Source) Discover() (source.Features, error) { - features := source.Features{} - - rdtFeatures := discoverRDT() - - for _, f := range rdtFeatures { - features[f] = true - } - - return features, nil -}