turboprop/README.rst

.. vim: set et sw=2:

#########
Turboprop
#########

Problem: You have twenty or thirty Helm releases, all of which you `template semi-manually`_. Deploying new applications involves tremendous amounts of copy-pasta.

Solution: Use Nix. With Nix, you can `ensure chart integrity`_, `generate repetitive data`_ in `subroutines`_, and `easily reuse variable data`_.

Turboprop templates your Helm charts for you, making an individual Nix derivation of each one; each of these derivations is then gathered into a mega-derivation complete with Kustomizations for every namespace and service. In short, you're two commands away from full cluster reconciliation::

  nix build && kubectl diff -k ./result

.. _template semi-manually: https://github.com/kubernetes-sigs/kustomize/blob/bfb00ecb2747dc711abfc27d9cf788ca1d7c637b/examples/chart.md#best-practice
.. _ensure chart integrity: https://git.sr.ht/~goorzhel/kubernetes/tree/c2184da86fe01b1f0881f441229dfdac2ceb84ba/item/charts/intel/device-plugins-gpu/default.nix#L5
.. _generate repetitive data: https://git.sr.ht/~goorzhel/kubernetes/tree/c2184da86fe01b1f0881f441229dfdac2ceb84ba/item/services/2-gateway/svc/gateway.nix#L37-38
.. _subroutines: https://git.sr.ht/~goorzhel/kubernetes/tree/c2184da86fe01b1f0881f441229dfdac2ceb84ba/item/services/2-gateway/svc/gateway.nix#L10-12
.. _easily reuse variable data: https://git.sr.ht/~goorzhel/kubernetes/tree/c2184da86fe01b1f0881f441229dfdac2ceb84ba/item/services/5-system-services/kube-system/nfs-subdir.nix#L15

***************
Acknowledgments
***************

- `Vladimir Pouzanov`_'s "`Nix and Kubernetes: Deployments Done Right`_" (and `its notes`_) is the reason this project exists.
- Early on, I used `heywoodlh's Kubernetes flake`_ as a starting point.
- Once I discovered `Haumea`_, Turboprop *really* started coming together.

.. _Vladimir Pouzanov: https://github.com/farcaller
.. _Nix and Kubernetes\: Deployments Done Right: https://media.ccc.de/v/nixcon-2023-35290-nix-and-kubernetes-deployments-done-right
.. _its notes: https://gist.github.com/farcaller/c87c03fbb55eaeaeb840b938455f37ff
.. _heywoodlh's Kubernetes flake: https://github.com/heywoodlh/flakes/blob/aa5a52a/kube/flake.nix
.. _Haumea: https://github.com/nix-community/haumea

************************
Tutorial (short version)
************************

First, define services in ``./services``. Ensure that CRD-providing services
are evaluated first, usually with ordered directories like ``./services/01-service-mesh``.

Then, in your flake:

#. Add Turboprop to ``inputs``.
#. Get an attrset of charts, either from Nixhelm or by making your own.
#. Call, at minimum, ``turboprop.lib.${system}.mkDerivation {charts} {pname, version, src, serviceRoot}``.

********
Tutorial
********

Installation
============

Add Turboprop to your flake's inputs, along with flake-utils and nixhelm:

.. code-block:: nix

  {
    inputs = {
      flake-utils.url = "github:numtide/flake-utils";
      nixhelm.url = "github:farcaller/nixhelm";
      turboprop = {
        url = "sourcehut:~goorzhel/turboprop";
        inputs.nixhelm.follows = "nixhelm";
      };
    };
    <...>
  }


Next, put it to use in your flake's output:

.. code-block:: nix

  {
    <...>
    outputs = {self, flake-utils, nixhelm, turboprop}:
    flake-utils.lib.eachDefaultSystem (system: let
      turbo = turboprop.lib.${system};
    in {
      packages.default = let
        pname = "my-k8s-flake";
      in
        turbo.mkDerivation {
          charts = nixhelm.chartsDerivations.${system}
        } {
          inherit pname;
          version = "rolling";
          src = builtins.path {
            path = ./.;
            name = pname;
          };

          serviceRoot = ./services;
          nsMetadata = {};
        };
    }
    );
  }

Now set that aside for the time being.

Example service module
======================

This is a module that defines a *service derivation*:

.. code-block:: nix

    { charts, lib, userData, ... }: {  # I
      builder = lib.builders.helmChart; # I2; O1
      args = {  # < - - - - - - - - - - - O2
        chart = charts.jetstack.cert-manager; # I1
        values = {
          featureGates = "ExperimentalGatewayAPISupport=true";
          installCRDs = true;
          prometheus = {
            enabled = true;
            servicemonitor = {
              enabled = true;
              prometheusInstance = "monitoring";
            };
          };
          startupapicheck.podLabels."sidecar.istio.io/inject" = "false";
        };
      };
      extraObjects = [  # O3
        {
          apiVersion = "cert-manager.io/v1";
          kind = "ClusterIssuer";
          metadata.name = userData.vars.k8sCert.name; # I3
          spec.ca.secretName = userData.vars.k8sCert.name;
        }
      ];
    }

The module takes as input these attributes, any of which you may omit:

  #. A tree of *chart derivations*;
  #. the Turboprop library;
  #. the Nixpkgs for the current system (``pkgs``);
  #. the name and namespace of the service (``name``, ``namespace``); and
  #. user data specific to your flake.

The output signature is ``{builder, args, extraObjects}``:

  #. ``builder`` is the Turboprop builder that will create your service derivation. Most often, you will use ``helmChart``; other builders exist for scenarios such as deploying a `collection of Kubernetes objects`_ or a `single remote YAML file`_. You may even `define your own builder`_.
  #. ``args`` are arguments passed to the builder. Refer to each builder's signature below.
  #. ``extraObjects`` are objects to deploy alongside the chart.

.. _collection of Kubernetes objects: https://git.sr.ht/~goorzhel/kubernetes/tree/c2184da86fe01b1f0881f441229dfdac2ceb84ba/item/services/2-gateway/svc/gateway.nix#L16
.. _single remote YAML file: https://git.sr.ht/~goorzhel/kubernetes/tree/ed73bd8e872038bb5e2d7d40434c2babe5ab28c6/item/services/0-base/gateway-system/gateway-api.nix#L2
.. _define your own builder: https://git.sr.ht/~goorzhel/kubernetes/tree/main/item/services/6-services/svc/breezewiki.nix#L2

Creating a service tree
=======================

Turboprop operates on *trees* of Nix modules, both in the filesystem sense (nested directories) and the Nix sense (nested attrsets), and uses `Haumea`_ to do so. A service tree consists of

#. an arbitrarily-named root, such as ``./services``, which contains
#. zero or more intermediate directories (we'll get to this), which each contain
#. directories representing Kubernetes namespaces, which each contain
#. Nix modules representing a templated deployment.

We'll start with building a flake containing two applications:

- the `Gateway API`_, and
- `Breezewiki`_ (through `app-template`_).

.. _Gateway API: https://gateway-api.sigs.k8s.io/
.. _Breezewiki: https://gitdab.com/cadence/breezewiki
.. _app-template: https://bjw-s.github.io/helm-charts/docs/app-template/

Normally, one would also deploy a Gateway controller, but this suffices for the example.

.. code-block:: nix

  # services/gateway-system/gateway-api.nix
  {lib, ...}: {

    # Any function can be used as a builder so long as it has variable arity
    # and produces a derivation consisting of a single YAML file.
    builder = lib.fetchers.remoteYAMLFile;

    args = rec {
      version = "1.0.0";
      url = "https://github.com/kubernetes-sigs/gateway-api/releases/download/v${version}/experimental-install.yaml";
      hash = "sha256-bGAdzteHKpQNdvpmeuEmunGMtMbblw0Lq0kSjswRkqM=";
    };
  }

.. code-block:: nix

  # services/default/breezewiki.nix
  {charts, lib, name, namespace ...}: {
    builder = lib.app-template.build;
    args = {
      mainImage = "quay.io/pussthecatorg/breezewiki:latest";
      values = let
        port = 10416;
      in {
        # app-template's schema can be found here:
        # https://github.com/bjw-s/helm-charts/blob/app-template-2.3.0/charts/library/common/values.yaml
        service.main.ports.http.port = port;
        route.main = {
          enabled = true;
          hostnames = ["${name}.example.com"];
          parentRefs = [
            {
              name = "gateway";
              inherit namespace;
              sectionName = "https";
            }
          ];
          rules = [
            {backendRefs = [{inherit name namespace port;}];}
          ];
        };
      };
    };
  }

Now build the flake::

  $ nix build
  $ ls -l result/*/*
  -r--r--r--   3 root root 88 Dec 31  1969 result/default/kustomization.yaml
  -r--r--r-- 130 root root 89 Dec 31  1969 result/gateway-system/kustomization.yaml

  result/default/breezewiki:
  total 12
  -r--r--r-- 1364 root root   90 Dec 31  1969 kustomization.yaml
  -r--r--r--    5 root root 2795 Dec 31  1969 SERVICE.yaml
  lrwxrwxrwx    4 root root   74 Dec 31  1969 SERVICE.yaml.drv -> /nix/store/sijp95rfkbijnrklmrb4smb9qvl7bd4v-yaml-stream-default-breezewiki

  result/gateway-system/gateway-api:
  total 768
  -r--r--r-- 1364 root root     90 Dec 31  1969 kustomization.yaml
  -r--r--r--   14 root root 775478 Dec 31  1969 SERVICE.yaml
  lrwxrwxrwx   11 root root     87 Dec 31  1969 SERVICE.yaml.drv -> /nix/store/0yi3y3b0lrgd71yrglgi7mjaxhk8khsm-copied-drv-gateway-system-gateway-api-1.0.0
  $ sha256sum result/gateway-system/gateway-api/SERVICE.yaml
  6c601dced7872a940d76fa667ae126ba718cb4c6db970d0bab49128ecc1192a3  result/gateway-system/gateway-api/SERVICE.yaml

Pretty cool, huh? Now to install the services...

.. code-block::

  $ kubectl apply -f result/namespaces.yaml
  namespace/default configured
  namespace/gateway-system created
  $ kubectl apply -k result/gateway-system/
  <...>
  $ kubectl apply -k result/default/breezewiki
  service/breezewiki created
  deployment.apps/breezewiki created
  error: resource mapping not found for name: "breezewiki" namespace: "default" from "result/default": no matches for kind "HTTPRoute" in version "gateway.networking.k8s.io/v1alpha2"
  ensure CRDs are installed first

Wait, what? A ``v1alpha2`` HTTP Route? That API isn't even *in* Gateway API v1. What gives?

Ordering services by provided APIs
==================================

Like most things in Nix, Helm derivations are *pure functions*: they have no room for external state. This means Helm cannot `poll a Kubernetes cluster`_ for data such as supported APIs, upon which charts such as ``app-template`` `depend`_ to calculate their output::

  {{- $routeKind := $routeObject.kind | default "HTTPRoute" -}}
  {{- $apiVersion := "gateway.networking.k8s.io/v1alpha2" -}}
  {{- if $rootContext.Capabilities.APIVersions.Has (printf "gateway.networking.k8s.io/v1beta1/%s" $routeKind) }}
    {{- $apiVersion = "gateway.networking.k8s.io/v1beta1" -}}
  {{- end -}}
  {{- if $rootContext.Capabilities.APIVersions.Has (printf "gateway.networking.k8s.io/v1/%s" $routeKind) }}
    {{- $apiVersion = "gateway.networking.k8s.io/v1" -}}
  {{- end -}}

This is a problem solved by Turboprop and all of its dependencies:

#. Helm `provides`_ the flags ``--api-versions`` and ``--kube-version`` with which to declare capabilities.
#. `nix-kube-generators`_' Helm builder `offers`_ the variables ``kubeVersion`` and ``apiVersions`` with reasonable defaults.
#. Turboprop accumulates APIs as it evaluates service modules in order, providing each module with the APIs generated before it.

Which order? Well, Haumea loads and Turboprop evaluates in alphabetical order. And thus we arrive to the crux of the problem: ``gateway-api`` > ``default``. Luckily, it's trivial to solve::

  $ mkdir services/{1-gateway,2-main}
  $ mv services/gateway-system services/1-gateway
  $ mv services/default services/2-main
  $ nix build
  <...>
  $ grep -A1 'apiVersion: gateway' result/2-main/default/breezewiki/SERVICE.yaml
  apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute

And there you have it: a Helm deployment supercharged with inheritance, functional purity, integrity-checking, and all else that is great about the Nix language.

.. _poll a Kubernetes cluster: https://helm.sh/docs/chart_template_guide/builtin_objects/
.. _depend: https://github.com/bjw-s/helm-charts/blob/c30dbd313011b63fa35e26f718d3161e476fe1fa/charts/library/common/templates/classes/_route.tpl#L9-L16
.. _provides: https://helm.sh/docs/helm/helm_template/
.. _nix-kube-generators: https://github.com/farcaller/nix-kube-generators
.. _offers: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L88-L89
.. _run-parts numbering: https://askubuntu.com/q/988664


*********
Reference
*********

Main functions
==============

These functions are only available outside of service modules.

mkDerivation
------------

``{charts, userData?}
-> {pname, version, src, serviceRoot, nsMetadata?, kubeVersion?, apiVersions?}
-> <derivation: a dir of Kustomization dirs>``

The main interface to Turboprop.

The first attrset instantiates the derivation builder:

- **charts** (attrs): A tree of Helm chart derivations.
- **userData** (attrs, default: ``{}``): Additional data to be used by the service modules.

The second attrset specifies the derivation to build:

- **pname** (str): The name of the derivation.
- **version** (str): The version of the derivation.
- **src** (path): The root of the source tree from which to build the derivation.
- **serviceRoot** (path): The root of the service tree.
- **nsMetadata** (attrs, default: ``{}``): Additional metadata to attach to the generated namespaces (see "Namespace metadata" below).
- **kubeVersion** (str, default: ``pkgs.kubernetes.version``): The version of the Kubernetes cluster to target.
- **apiVersions** ([str], default: ``[]``): API versions to declare in addition to those provided by generated services.

mkCharts
--------

``src -> attrs``

Searches a directory tree for Nix modules describing a chart and fetches each chart, returning the tree as an attrset of derivatives.

Each module must be an attrset with the signature ``{repo, chart, version, chartHash?}``; see the documentation of ``lib.fetchers.helmChart`` for more.

- **src** (path): Search root.

mkChartsWithNixhelm
-------------------

``src -> attrs``

Same as ``mkCharts``, but overlays the fetched charts onto the ones `provided by Nixhelm`_ through the flake input.

- **src** (path): Search root.

.. _provided by Nixhelm: https://github.com/farcaller/nixhelm/blob/f63710348e393d8640e9e9b896e74bb31a3ee871/flake.nix#L45

Fetchers
========

Fetcher functions download a resource into the Nix store. A fetcher may also serve as a builder for resources
intended to be used without modification or processing, such as a YAML file.

gitChart
--------
``{name, version, url, hash, chartPath, vPrefixInRef?, ...} -> <derivation: a dir containing a Helm chart>``

Fetch a Helm chart from a Git repository. Useful in the absence of a published Helm repo.

- **name** (str): Git repo name.
- **version** (str): The tag to check out, which should resemble ``1.0.0``.
- **url** (str): Git repo URL.
- **vPrefixInRef** (bool, default: ``false``): Whether the Git tag begins with an utterly redundant ``v``.
- **chartHash** (str): An `SRI-style hash`_.

.. _SRI-style hash: https://nixos.wiki/wiki/Nix_Hash

helmChart
---------
``{repo, chart, version, chartHash?} -> <derivation: a dir containing a Helm chart>``

Re-export of `kubelib.downloadHelmChart`_.

- **repo** (str): The repository from which to download the chart.
- **chart** (str): Chart name.
- **version** (str): Chart version, which will also be the derivation's version.
- **chartHash** (str, default: `fakeHash`_): An `SRI-style hash`_.

.. _kubelib.downloadHelmChart: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L49
.. _fakeHash: https://github.com/NixOS/nixpkgs/blob/5b528f99f73c4fad127118a8c1126b5e003b01a9/lib/deprecated.nix#L304

remoteYAMLFile
--------------
``{version, url, hash, ...} -> <derivation: a YAML file>``

Fetch a remote file. Useful for applications distributed as a YAML stream, e.g., `the Gateway API`_.

- **version** (str): Application version, which will also be the derivation's version.
- **url** (str): The URL from which to fetch the file.
- **hash** (str): An `SRI-style hash`_.

.. _the Gateway API: https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.0.0

Builders
========

Builder functions build a service derivation.

Builders receive ``name`` and ``namespace`` through Turboprop, so these two variables will be documented once:

- **name** (str): The name of the service. Usually reflected in the label ``app.kubernetes.io/instance``, as well as the derivation's name.
- **namespace** (str): The namespace into which to deploy the service.

derivation
----------
``{name, namespace, src, ...} -> <derivation>``

Copy a derivation verbatim.

- **src** (derivation): The derivation to copy.

helmChart
---------
``{name, namespace, chart, values?, includeCRDs?, kubeVersion?, apiVersions?, extraOpts?, ...} -> <derivation: a YAML file of Helm output>``

Re-export of `kubelib.buildHelmChart`_.

- **chart** (derivation): The chart from which to build.
- **values** (attrs, default: ``{}``): Values to pass into the chart.
- **includeCRDs** (bool, default: ``true``): Whether to include CustomResourceDefinitions in the template output.
- **kubeVersion** (str, default: ``pkgs.kubernetes.version``): Target Kubernetes version.
- **apiVersions** ([str], default: ``[]``): Sets `Capabilities.APIVersions`_.
- **extraOpts** ([str], default: ``[]``): Additional flags for ``helm template``.

.. _kubelib.buildHelmChart: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L82-L90
.. _Capabilities.APIVersions: https://helm.sh/docs/chart_template_guide/builtin_objects/#helm

yamlStream
----------

``{name, namespace, objs, ...} -> <derivation: a YAML file>``

Converts Kubernetes objects from Nix to YAML.

- **objs** ([attrs]): The objects to convert.

app-template.build
------------------

``{name, namespace, mainImage, values?, kubeVersion?, apiVersions?} -> {builder, args, extraObjects}``

Wrapper of ``helmChart`` that builds `app-template`_ images.

- **mainImage** (str): `OCI image address`_ for the main container.
- **values** (attrs, default: ``{}``): Values to pass into the chart.
- **kubeVersion** (str, default: ``pkgs.kubernetes.version``): Target Kubernetes version.
- **apiVersions** ([str], default: ``[]``): Sets `Capabilities.APIVersions`_.
- **extraOpts** ([str], default: ``[]``): Additional flags for ``helm template``.

.. _OCI image address: https://github.com/opencontainers/.github/blob/main/docs/docs/introduction/digests.md


Other signatures
================

Service (unbuilt)
-----------------

``{name, namespace, charts?, lib?, pkgs?, userData?} -> {builder, args, extraObjects?}``

A service module as defined in your flake.

Input attrset, of which any of its attributes may be omitted if unused:

- **charts** (attrs): A tree of Helm chart derivations.
- **lib** (attrs): Turboprop library.
- **pkgs** (attrs): Nixpkgs.
- **userData** (attrs): Additional data for the service module.

Output attrset:

- **builder** (function): A builder function.
- **args** (attrs): Arguments for the builder function.
- **extraObjects** ([attrs], default: ``null``): Kubernetes objects to deploy alongside the service.

Service (loaded)
-----------------

``{name, namespace, kubeVersion, apiVersions} -> {out, extra}``

A service module loaded by Turboprop and ready to produce derivations.

Input attrset:

- **name** (str): The name of the service.
- **namespace** (str): The namespace into which to deploy the service.
- **kubeVersion** (str): Target Kubernetes version.
- **apiVersions** ([str]): Sets `Capabilities.APIVersions`_.

Output attrset:

- **out** (derivation): The service as a YAML file.
- **extra** (derivation, default: ``null``): Extra objects as a YAML file.

Builder
-------

``{name, namespace, kubeVersion, apiVersions, ...} -> <derivation: a YAML file>``

The signature of a generic builder.

- **name** (str): The name of the service.
- **namespace** (str): The namespace into which to deploy the service.
- **kubeVersion** (str): Target Kubernetes version.
- **apiVersions** ([str]): Sets `Capabilities.APIVersions`_.

Namespace metadata
------------------

``{DEFAULT?, ...}``

The signature of the ``nsMetadata`` argument to ``mkDerivation``.

Each namespace is represented by an attrset; this attrset is copied to the resulting namespace's ``metadata`` key at build time. For example, this is equivalent to ``k label ns/default istio.io/rev=stable``::

  default = {
    labels = {
      "istio.io/rev" = "stable";
    }
  }

Metadata to be applied to all namespaces can be set in the special attrset ``DEFAULT``::

  DEFAULT = {
    labels = {
      "istio.io/rev" = "stable";
    };
  };

  # Opt a namespace out of the defaults.
  gateway-system = {};
  kube-system = {};
  longhorn-system = {};

  # To set data beyond the defaults,
  # opt the namespace back in.
  default =
    DEFAULT
    // {
      labels = {
        "words-words-words-words" = "punchline";
      };
    };

N.B.: namespaces set in extraMetadata but not present in ``namespaces`` aren't created.