kyverno/DEVELOPMENT.md

Developer Instructions

This document covers basic needs to work with Kyverno codebase.

It contains instructions to build, run, and test Kyverno.

• Tools
• Building local binaries
• Building local images
  • Building local images with docker
  • Building local images with ko
• Pushing images
  • Pushing images with docker
  • Pushing images with ko
• Deploying a local build
  • Create a local cluster
  • Build and load local images
  • Deploy with helm
• Code generation
  • Generating kubernetes API client
  • Generating API deep copy functions
  • Generating CRD definitions

Tools

Building and/or testing Kyverno requires additional tooling.

We use make to simplify installing the tools we use.

Tools will be installed in the .tools folder when possible, this allows keeping installed tools local to the Kyverno repository. The .tools folder is ignored by git and binaries should not be committed.

Note

: If you don't install tools, they will be downloaded/installed as necessary when running make targets.

You can manually install tools by running:

make install-tools

To remove installed tools, run:

make clean-tools

Building local binaries

The Kyverno repository contains code for three different binaries:

• kyvernopre: Binary to update/cleanup existing resources in clusters. This is typically run as an init container before Kyverno controller starts.
• kyverno: The Kyverno controller binary.
• cli: The Kyverno command line interface.

Note

: You can build all binaries at once by running make build-all.

Building kyvernopre locally

To build kyvernopre binary on your local system, run:

make build-kyvernopre

The binary should be created at ./cmd/initContainer/kyvernopre.

Building kyverno locally

To build kyverno binary on your local system, run:

make build-kyverno

The binary should be created at ./cmd/kyverno/kyverno.

Building cli locally

To build cli binary on your local system, run:

make build-cli

The binary should be created at ./cmd/cli/kubectl-kyverno/kubectl-kyverno.

Building local images

In the same spirit as building local binaries, you can build local docker images instead of local binaries.

Currently, we are supporting two build systems:

• Building local images with docker
• Building local images with ko

Note

: We started with docker and are progressively moving to ko.

As the ko based build system matures, we will deprecate and remove docker based builds.

Choosing between docker and ko boils down to a prefix when invoking make targets. For example:

• make docker-build-kyverno creates a docker image using the docker build system
• make ko-build-kyverno creates a docker image using the ko build system

Building local images with docker

When building local images with docker you can specify the registry used to create the image names by setting the REGISTRY environment variable (default value is ghcr.io).

Note

: You can build all local images at once by running make docker-build-all.

Building kyvernopre image locally

To build kyvernopre image on your local system, run:

make docker-build-kyvernopre

The resulting image should be available locally, named ghcr.io/kyverno/kyvernopre (by default, if REGISTRY environment variable was not set).

Building kyverno image locally

To build kyverno image on your local system, run:

make docker-build-kyverno

The resulting image should be available locally, named ghcr.io/kyverno/kyverno (by default, if REGISTRY environment variable was not set).

Building cli image locally

To build cli image on your local system, run:

make docker-build-cli

The resulting image should be available locally, named ghcr.io/kyverno/kyverno-cli (by default, if REGISTRY environment variable was not set).

Building local images with ko

When building local images with ko you can't specify the registry used to create the image names. It will always be ko.local.

Note

: You can build all local images at once by running make ko-build-all.

Building kyvernopre image locally

To build kyvernopre image on your local system, run:

make ko-build-kyvernopre

The resulting image should be available locally, named ko.local/github.com/kyverno/kyverno/cmd/initcontainer.

Building kyverno image locally

To build kyverno image on your local system, run:

make ko-build-kyverno

The resulting image should be available locally, named ko.local/github.com/kyverno/kyverno/cmd/kyverno.

Building cli image locally

To build cli image on your local system, run:

make ko-build-cli

The resulting image should be available locally, named ko.local/github.com/kyverno/kyverno/cmd/cli/kubectl-kyverno.

Pushing images

Pushing images is very similar to building local images, except that built images will be published on a remote image registry.

Currently, we are supporting two build systems:

• Pushing images with docker
• Pushing images with ko

Note

: We started with docker and are progressively moving to ko.

As the ko based build system matures, we will deprecate and remove docker based builds.

When pushing images you can specify the registry you want to publish images to by setting the REGISTRY environment variable (default value is ghcr.io).

Pushing images with docker

Authenticating to the remote registry is not done automatically in the Makefile.

You need to be authenticated before invoking targets responsible for pushing images.

Note

: You can push all images at once by running make docker-publish-all or make docker-publish-all-dev.

Pushing kyvernopre image

To push kyvernopre image on a remote registry, run:

# push stable image
make docker-publish-kyvernopre

or

# push dev image
make docker-publish-kyvernopre-dev

The resulting image should be available remotely, named ghcr.io/kyverno/kyvernopre (by default, if REGISTRY environment variable was not set).

Pushing kyverno image

To push kyverno image on a remote registry, run:

# push stable image
make docker-publish-kyverno

or

# push dev image
make docker-publish-kyverno-dev

The resulting image should be available remotely, named ghcr.io/kyverno/kyverno (by default, if REGISTRY environment variable was not set).

Pushing cli image

To push cli image on a remote registry, run:

# push stable image
make docker-publish-cli

or

# push dev image
make docker-publish-cli-dev

The resulting image should be available remotely, named ghcr.io/kyverno/kyverno-cli (by default, if REGISTRY environment variable was not set).

Pushing images with ko

Authenticating to the remote registry is done automatically in the Makefile with ko login.

To allow authentication you will need to set REGISTRY_USERNAME and REGISTRY_PASSWORD environment variables before invoking targets responsible for pushing images.

Note

: You can push all images at once by running make ko-publish-all or make ko-publish-all-dev.

Pushing kyvernopre image

To push kyvernopre image on a remote registry, run:

# push stable image
make ko-publish-kyvernopre

or

# push dev image
make ko-publish-kyvernopre-dev

The resulting image should be available remotely, named ghcr.io/kyverno/kyvernopre (by default, if REGISTRY environment variable was not set).

Pushing kyverno image

To push kyverno image on a remote registry, run:

# push stable image
make ko-publish-kyverno

or

# push dev image
make ko-publish-kyverno-dev

The resulting image should be available remotely, named ghcr.io/kyverno/kyverno (by default, if REGISTRY environment variable was not set).

Pushing cli image

To push cli image on a remote registry, run:

# push stable image
make ko-publish-cli

or

# push dev image
make ko-publish-cli-dev

The resulting image should be available remotely, named ghcr.io/kyverno/kyverno-cli (by default, if REGISTRY environment variable was not set).

Deploying a local build

After building local images, it is often useful to deploy those images in a local cluster.

We use KinD to create local clusters easily.

Create a local cluster

If you already have a local KinD cluster running, you can skip this step.

To create a local KinD cluster, run:

make kind-create-cluster

You can override the k8s version by setting the KIND_IMAGE environment variable (default value is kindest/node:v1.24.0).

You can also override the KinD cluster name by setting the KIND_NAME environment variable (default value is kind).

Build and load local images

To build local images and load them on a local KinD cluster, run:

# build kyvernopre image and load it in KinD cluster
make kind-load-kyvernopre

or

# build kyverno image and load it in KinD cluster
make kind-load-kyverno

or

# build kyvernopre and kyverno images and load them in KinD cluster
make kind-load-all

You can override the KinD cluster name by setting the KIND_NAME environment variable (default value is kind).

Deploy with helm

To build local images, load them on a local KinD cluster, and deploy helm charts, run:

# build images, load them in KinD cluster and deploy kyverno helm chart
make kind-deploy-kyverno

or

# deploy kyverno-policies helm chart
make kind-deploy-kyverno-policies

or

# build images, load them in KinD cluster and deploy helm charts
make kind-deploy-all

This will build local images, load built images in every node of the KinD cluster, and deploy kyverno and/or kyverno-policies helm charts in the cluster (overriding image repositories and tags).

Note

: This actually uses ko to build local images.

You can override the KinD cluster name by setting the KIND_NAME environment variable (default value is kind).

Code generation

We are using code generation tools to create the following portions of code:

• Generating kubernetes API client
• Generating API deep copy functions
• Generating CRD definitions

Note

: You can run make codegen-all to build all generated code at once.

Generating kubernetes API client

TODO

Generating API deep copy functions

Based on the APIs golang code definitions, you can generate the corresponding deep copy functions by running:

# generate all deep copy functions
make codegen-deepcopy-all

or

# generate kyverno deep copy functions
make codegen-deepcopy-kyverno

or

# generate policy reports deep copy functions
make codegen-deepcopy-report

This will output files named zz_generated.deepcopy.go in every API package.

Generating CRD definitions

Based on the APIs golang code definitions, you can generate the corresponding CRDs manifests by running:

# generate all CRDs
make codegen-crds-all

or

# generate Kyverno CRDs
make codegen-crds-kyverno

or

# generate policy reports CRDs
make codegen-crds-report

This will output CRDs manifests /config/crds.

Building and publishing an image locally

First, make sure you install ko

Publishing to your local Docker daemon

Set the KO_DOCKER_REPO environment variable to ko.local:

KO_DOCKER_REPO=ko.local

Then build and publish an image:

ko build ./cmd/kyverno --preserve-import-paths

The image will be available locally as ko.local/github.com/kyverno/kyverno/cmd/kyverno.

Publishing to a local KinD cluster

First, create your KinD cluster:

kind create cluster

Set the KO_DOCKER_REPO environment variable to kind.local:

KO_DOCKER_REPO=kind.local

Then build and publish an image:

ko build ./cmd/kyverno --preserve-import-paths

This will build and load the image into your KinD cluster as:

kind.local/github.com/kyverno/kyverno/cmd/kyverno

If you have multiple KinD clusters, or created them with a non-default name, set KIND_CLUSTER_NAME=<your-cluster-name>.

Publishing to a remote registry

Set the KO_DOCKER_REPO environment variable to the registry you'd like to push to: For example:

KO_DOCKER_REPO=gcr.io/my-project/kyverno
KO_DOCKER_REPO=my-dockerhub-user/my-dockerhub-repo
KO_DOCKER_REPO=<ACCOUNTID>.dkr.ecr.<REGION>.amazonaws.com

Then build and publish an image:

ko build ./cmd/kyverno

The output will tell you the image name and digest of the image you just built.