.. vim: set et sw=2:

#########
Turboprop
#########

Problem: You have twenty or thirty Helm releases, all of which you template semi-manually to `retain WYSIWYG control`_. Deploying new applications involves tremendous amounts of copy-pasta.

Solution: Use Nix. With Nix, you can `ensure chart integrity`_, `generate repetitive data`_ in `subroutines`_, and `easily reuse variable data`_.

Turboprop templates your Helm charts for you, making an individual Nix derivation of each one; each of these derivations is then gathered into a mega-derivation complete with Kustomizations for every namespace and service. In short, you're two commands away from full cluster reconciliation::

  nix build && kubectl diff -k ./result

.. _retain WYSIWYG control: https://github.com/kubernetes-sigs/kustomize/blob/bfb00ecb2747dc711abfc27d9cf788ca1d7c637b/examples/chart.md#best-practice
.. _ensure chart integrity: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/charts/intel/device-plugins-gpu/default.nix#L5
.. _generate repetitive data: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L25-26
.. _subroutines: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L8-10
.. _easily reuse variable data: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/system/kube-system/csi-driver-nfs/default.nix#L16

***************
Acknowledgments
***************

- `Vladimir Pouzanov`_'s "`Nix and Kubernetes\: Deployments Done Right`_" (and `its notes`_) is the reason this project exists.
- Early on, I used `heywoodlh's Kubernetes flake`_ as a starting point.
- Once I discovered `Haumea`_, Turboprop *really* started coming together.

.. _Vladimir Pouzanov: https://github.com/farcaller
.. _Nix and Kubernetes\: Deployments Done Right: https://media.ccc.de/v/nixcon-2023-35290-nix-and-kubernetes-deployments-done-right
.. _its notes: https://gist.github.com/farcaller/c87c03fbb55eaeaeb840b938455f37ff
.. _heywoodlh's Kubernetes flake: https://github.com/heywoodlh/flakes/blob/aa5a52a/kube/flake.nix
.. _Haumea: https://github.com/nix-community/haumea

********
Tutorial
********

Installation
============

To start, add this flake to your flake's inputs, along with ``nixpkgs`` and ``flake-utils``:

.. code-block:: nix

  {
    inputs = {
      nixpkgs.url = "github:NixOS/nixpkgs";
      flake-utils.url = "github:numtide/flake-utils";
      turboprop = {
        url = "sourcehut:~goorzhel/turboprop";
        inputs.nixpkgs.follows = "nixpkgs";
      };
    };
    <...>
  }


Next, put it to use in your flake's output:

.. code-block:: nix

  {
    <...>
    outputs = {self, nixpkgs, flake-utils, turboprop}:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {inherit system;};

      turbo = turboprop.lib.${system};

      mkDerivation = turbo.mkDerivation {
        user = {
          # We'll get to this
          app-template = turbo.app-template;
        };
      };
    in {
        packages.default = let
          pname = "my-k8s-flake";
        in
          mkDerivation {
            inherit pname;
            version = "rolling";
            src = builtins.path {
              path = ./.;
              name = pname;
            };

            serviceRoot = ./services;
            nsMetadata = {};
        };
      }
    );
  }

Now set that aside for the time being.

Example service module
======================

This is a module that defines a *service derivation*:

.. code-block:: nix

    { charts, lib, user, ... }: {  # 1
      builder = lib.builders.helmChart; # 1.2; 2.1
      args = {  # < - - - - - - - - - - - 2.2
        chart = charts.jetstack.cert-manager; # 1.1
        values = {
          featureGates = "ExperimentalGatewayAPISupport=true";
          installCRDs = true;
          prometheus = {

            enabled = true;
            servicemonitor = {
              enabled = true;
              prometheusInstance = "monitoring";
            };
          };
          startupapicheck.podLabels."sidecar.istio.io/inject" = "false";
        };
      };
      extraObjects = [  # 2.3
        {
          apiVersion = "cert-manager.io/v1";
          kind = "ClusterIssuer";
          metadata.name = user.vars.k8sCert.name; # 1.3
          spec.ca.secretName = user.vars.k8sCert.name;
        }
      ];

    }

1. The module takes as input:

  #. A tree of *chart derivations*;
  #. the Turboprop library; and
  #. the Nixpkgs for the current system (``pkgs``); and
  #. user data specific to your flake. You may `omit any of these input variables`_.

2. The module has the output signature ``{builder, args, extraObjects}``.

  #. ``builder`` is the Turboprop builder that will create your derivation. Most often, you will use ``helmChart``; other builders exist for scenarios such as deploying a `collection of Kubernetes objects`_ or a `single remote YAML file`_. You may even `define your own builder`_.
  #. ``args`` are arguments passed to the builder. Refer to each builder's signature below.
  #. ``extraObjects`` are objects to deploy alongside the chart.

.. _omit any of these input variables: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/system/gateway-system/gateway-api/default.nix#L1
.. _collection of Kubernetes objects: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L12
.. _single remote YAML file: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/system/gateway-system/gateway-api/default.nix#L2
.. _define your own builder: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/breezewiki/default.nix#L6

Creating a service tree
=======================

Turboprop operates on *trees* of Nix modules, both in the filesystem sense (nested directories) and the Nix sense (nested attrsets), and uses `Haumea`_ to do so. A service tree consists of

#. an arbitrarily-named root, such as ``./services``, which contains
#. zero or more intermediate directories (we'll get to this), which each contain
#. directories representing Kubernetes namespaces, which each contain
#. Nix modules representing a templated deployment.

We'll start with building a flake containing two applications:

- the `Gateway API`_, and
- `Breezewiki`_ (through `app-template`_).

.. _Gateway API: https://gateway-api.sigs.k8s.io/
.. _Breezewiki: https://gitdab.com/cadence/breezewiki
.. _app-template: https://bjw-s.github.io/helm-charts/docs/app-template/

Normally, one would also deploy a Gateway controller, but this suffices for the example.

.. code-block:: nix

  # services/gateway-system/gateway-api/default.nix
  {lib, ...}: {
    builder = lib.builders.derivation;
    args = {
      src = lib.fetchers.remoteYAMLFile rec {
        version = "1.0.0";
        url = "https://github.com/kubernetes-sigs/gateway-api/releases/download/v${version}/experimental-install.yaml";
        hash = "sha256-bGAdzteHKpQNdvpmeuEmunGMtMbblw0Lq0kSjswRkqM=";
      };
    };
  }

.. code-block:: nix

  # services/default/breezewiki/default.nix
  {charts, lib, user, ...}: {
    builder = user.app-template.build;
    args = {
      mainImage = "quay.io/pussthecatorg/breezewiki:latest";
      values = let
        port = 10416;
      in {
        # app-template's schema can be found here:
        # https://github.com/bjw-s/helm-charts/blob/app-template-2.3.0/charts/library/common/values.yaml
        service.main.ports.http.port = port;
        route.main = {
          enabled = true;
          hostnames = ["breezewiki.example.com"];
          parentRefs = [
            {
              name = "gateway";
              namespace = "default";
              sectionName = "https";
            }
          ];
          rules = [
            {
              backendRefs = [
                {
                  name = "breezewiki";
                  namespace = "default";
                  inherit port;
                }
              ];
            }
          ];
        };
      };
    };
  }

Now build the flake::

  $ nix build
  $ ls -l result/*/*
  -r--r--r--   3 root root 88 Dec 31  1969 result/default/kustomization.yaml
  -r--r--r-- 130 root root 89 Dec 31  1969 result/gateway-system/kustomization.yaml

  result/default/breezewiki:
  total 12
  -r--r--r-- 1364 root root   90 Dec 31  1969 kustomization.yaml
  -r--r--r--    5 root root 2795 Dec 31  1969 SERVICE.yaml
  lrwxrwxrwx    4 root root   74 Dec 31  1969 SERVICE.yaml.drv -> /nix/store/sijp95rfkbijnrklmrb4smb9qvl7bd4v-yaml-stream-default-breezewiki

  result/gateway-system/gateway-api:
  total 768
  -r--r--r-- 1364 root root     90 Dec 31  1969 kustomization.yaml
  -r--r--r--   14 root root 775478 Dec 31  1969 SERVICE.yaml
  lrwxrwxrwx   11 root root     87 Dec 31  1969 SERVICE.yaml.drv -> /nix/store/0yi3y3b0lrgd71yrglgi7mjaxhk8khsm-copied-drv-gateway-system-gateway-api-1.0.0
  $ sha256sum result/gateway-system/gateway-api/SERVICE.yaml
  6c601dced7872a940d76fa667ae126ba718cb4c6db970d0bab49128ecc1192a3  result/gateway-system/gateway-api/SERVICE.yaml

Pretty cool, huh? Wait, hang on::

  $ grep -A1 'apiVersion: gateway' result/default/breezewiki/SERVICE.yaml
  apiVersion: gateway.networking.k8s.io/v1alpha2
  kind: HTTPRoute

A ``v1alpha2`` HTTP Route? That API isn't even *in* Gateway API v1. Applying this flake's output with ``kubectl`` will certainly fail. What gives?

Ordering services by provided APIs
==================================

Like most things in Nix, Helm derivations are *pure functions*: they have no room for external state. This means Helm cannot `poll the cluster`_ for data such as supported APIs, upon which charts such as ``app-template`` depend to calculate their output::

  {{- $routeKind := $routeObject.kind | default "HTTPRoute" -}}
  {{- $apiVersion := "gateway.networking.k8s.io/v1alpha2" -}}
  {{- if $rootContext.Capabilities.APIVersions.Has (printf "gateway.networking.k8s.io/v1beta1/%s" $routeKind) }}
    {{- $apiVersion = "gateway.networking.k8s.io/v1beta1" -}}
  {{- end -}}
  {{- if $rootContext.Capabilities.APIVersions.Has (printf "gateway.networking.k8s.io/v1/%s" $routeKind) }}
    {{- $apiVersion = "gateway.networking.k8s.io/v1" -}}
  {{- end -}}

This is a problem solved by Turboprop and all of its dependencies:

#. Helm `provides`_ the flags ``--api-versions`` and ``--kube-version`` with which to declare `capabilities`_.
#. `nix-kube-generators`_' Helm builder `offers`_ the variables ``kubeVersion`` and ``apiVersions`` with reasonable defaults.
#. Turboprop accumulates APIs as it evaluates service modules in order, providing each module with the APIs generated before it.

Which order? Well, Hamuea loads and Turboprop evaluates in alphabetical order.

And thus we arrive to the crux of the problem: ``gateway-api`` > ``default``. Luckily for us, it's trivial to solve::

  $ mkdir services/{1-gateway,2-main}
  $ mv services/gateway-system services/1-gateway
  $ mv services/default services/2-main
  $ nix build
  <...>
  $ grep -A1 'apiVersion: gateway' result/2-main/default/breezewiki/SERVICE.yaml
  apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute

While this example uses a naive numbering scheme, you may consider an old-school method, namely `run-parts numbering`_.

.. _poll the cluster: https://github.com/bjw-s/helm-charts/blob/c30dbd313011b63fa35e26f718d3161e476fe1fa/charts/library/common/templates/classes/_route.tpl#L9-L16
.. _provides: https://helm.sh/docs/helm/helm_template/
.. _capabilities: https://helm.sh/docs/chart_template_guide/builtin_objects/
.. _nix-kube-generators: https://github.com/farcaller/nix-kube-generators
.. _offers: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L88-L89
.. _run-parts numbering: https://askubuntu.com/q/988664


*********
Reference
*********

Library
=======

mkDerivation
------------

Turboprop overlays `its own charts`_ atop `Nixhelm's`_.

.. _its own charts: https://git.sr.ht/~goorzhel/turboprop/tree/main/item/charts
.. _Nixhelm's: https://github.com/farcaller/nixhelm/tree/master/charts


app-template
------------

mkCharts
--------

mkChartsWithNixhelm
-------------------

Fetchers
========

gitChart
--------
``{name, version, url, hash, chartPath, vPrefixInRef?} -> <derivation: a dir containing a Helm chart>``

Fetch a Helm chart from a Git repository. Useful in the absence of a published Helm repo.

- **name** (str): The name of the Git repo.
- **version** (str): The tag to check out, which should resemble ``1.0.0``.
- **url** (str): The URL of the Git repo.
- **vPrefixInRef** (bool, default: ``false``): Whether the Git tag begins with an utterly redundant ``v``.
- **chartHash** (str): An `SRI-style hash`_.

helmChart
---------
``{repo, chart, version, chartHash?} -> <derivation: a dir containing a Helm chart>``

Re-export of `kubelib.downloadHelmChart`_.

- **repo** (str): The repository from which to download the chart.
- **chart** (str): The name of the chart.
- **version** (str): The version of the chart, which will also be the derivation's version.
- **chartHash** (str, default: `fakeHash`_): An `SRI-style hash`_.

.. _kubelib.downloadHelmChart: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L49
.. _fakeHash: https://github.com/NixOS/nixpkgs/blob/5b528f99f73c4fad127118a8c1126b5e003b01a9/lib/deprecated.nix#L304

remoteYAMLFile
--------------
``{version, url, hash} -> <derivation: a YAML file>``

Fetch a remote file. Useful for applications distributed as a YAML stream, e.g., the `Gateway API`_.

- **version** (str): The version of the application, which will also be the derivation's version.
- **url** (str): The URL from which to fetch the file.
- **hash** (str): An `SRI-style hash`_.

.. _Gateway API: https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.0.0

Builders
========

Builder functions build a service derivation.

Builders receive ``name`` and ``namespace`` through Turboprop, so these two variables will be documented once:

- **name** (str): The name of the service. Usually reflected in the label ``app.kubernetes.io/instance``, as well as the derivation's name.
- **namespace** (str): The namespace into which to deploy the service.

derivation
----------
``{name, namespace, src, ...} -> <derivation>``

Copy a derivation verbatim. Useful in conjunction with a fetcher that produces a single file, like ``lib.fetchers.remoteYAMLFile``.

- **src** (derivation): The derivation to copy.

helmChart
---------
``{name, namespace, chart, values?, includeCRDs?, kubeVersion?, apiVersions?} -> <derivation: a YAML file of Helm output>``

Wrapped re-export of `kubelib.fromHelm`_ that sets ``metadata.namespace`` on all templated objects lacking it. As such, its signature is identical to `kubelib.buildHelmChart`_.

- **chart** (derivation): The chart from which to build.
- **values** (attrs, default: ``{}``): Values to pass into the chart.
- **includeCRDs** (bool, default: ``true``): Whether to include CustomResourceDefinitions in the template output.
- **kubeVersion** (str, default: ``pkgs.kubernetes.version``): The Kubernetes version to target.
- **apiVersions** ([str], default: ``[]``): Sets `Capabilities.APIVersions`_.

.. _kubelib.fromHelm: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L123
.. _kubelib.buildHelmChart: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L82-L90
.. _Capabilities.APIVersions: https://helm.sh/docs/chart_template_guide/builtin_objects/#helm

app-template.build
------------------


.. _SRI-style hash: https://nixos.wiki/wiki/Nix_Hash

### namespaces

Assign extra metadata in ``namespaces.nix``. For example,
``svc = {labels."istio.io/rev" = "1-18-1"}``
is the equivalent of
``k label ns/svc istio.io/rev=1-18-1``

Modules
=======

Service (unbuilt)
-----------------

``{charts, lib, pkg, user} -> {builder, args, extraObjects}``


Service (loaded)
-----------------

``{kubeVersion, apiVersion} -> {out, extra}``