.. vim: set et sw=2:

#########
Turboprop
#########

Problem: You have twenty or thirty Helm releases, all of which you template semi-manually to `retain WYSIWYG control`_. Deploying new applications involves tremendous amounts of copy-pasta.

Solution: Use Nix. With Nix, you can `ensure chart integrity`_, `generate repetitive data`_ in `subroutines`_, and `easily reuse variable data`_.

Turboprop templates your Helm charts for you, making an individual Nix derivation of each one; each of these derivations is then gathered into a mega-derivation complete with Kustomizations for every namespace and service. In short, you're two commands away from full cluster reconciliation::

  nix build && kubectl diff -k ./result

.. _retain WYSIWYG control: https://github.com/kubernetes-sigs/kustomize/blob/bfb00ecb2747dc711abfc27d9cf788ca1d7c637b/examples/chart.md#best-practice
.. _ensure chart integrity: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/charts/intel/device-plugins-gpu/default.nix#L5
.. _generate repetitive data: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L25-26
.. _subroutines: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L8-10
.. _easily reuse variable data: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/system/kube-system/csi-driver-nfs/default.nix#L16

***************
Acknowledgments
***************

- `Vladimir Pouzanov`_'s "`Nix and Kubernetes\: Deployments Done Right`_" (and `its notes`_) is the reason this project exists.
- Early on, I used `heywoodlh's Kubernetes flake`_ as a starting point.
- Once I discovered `Haumea`_, Turboprop *really* started coming together.

.. _Vladimir Pouzanov: https://github.com/farcaller
.. _Nix and Kubernetes\: Deployments Done Right: https://media.ccc.de/v/nixcon-2023-35290-nix-and-kubernetes-deployments-done-right
.. _its notes: https://gist.github.com/farcaller/c87c03fbb55eaeaeb840b938455f37ff
.. _heywoodlh's Kubernetes flake: https://github.com/heywoodlh/flakes/blob/aa5a52a/kube/flake.nix
.. _Haumea: https://github.com/nix-community/haumea

************************
Tutorial (short version)
************************

First, define services in ``./services``. Ensure that CRD-providing services
are evaluated first, usually with ordered directories like ``./services/01-service-mesh``.

Then, in your flake:

#. Add Turboprop to ``inputs``.
#. Call, at minimum, ``turboprop.lib.${system}.mkDerivation {} {pname, version, src, serviceRoot}``.

********
Tutorial
********

Installation
============

Add this flake to your flake's inputs, along with ``flake-utils``:

.. code-block:: nix

  {
    inputs = {
      flake-utils.url = "github:numtide/flake-utils";
      turboprop.url = "sourcehut:~goorzhel/turboprop";
    };
    <...>
  }


Next, put it to use in your flake's output:

.. code-block:: nix

  {
    <...>
    outputs = {self, flake-utils, turboprop}:
    flake-utils.lib.eachDefaultSystem (system: let
      turbo = turboprop.lib.${system};
    in {
      packages.default = let
        pname = "my-k8s-flake";
      in
        turbo.mkDerivation {} {
          inherit pname;
          version = "rolling";
          src = builtins.path {
            path = ./.;
            name = pname;
          };

          serviceRoot = ./services;
          nsMetadata = {};
        };
    }
    );
  }

Now set that aside for the time being.

Example service module
======================

This is a module that defines a *service derivation*:

.. code-block:: nix

    { charts, lib, user, ... }: {  # 1
      builder = lib.builders.helmChart; # 1.2; 2.1
      args = {  # < - - - - - - - - - - - 2.2
        chart = charts.jetstack.cert-manager; # 1.1
        values = {
          featureGates = "ExperimentalGatewayAPISupport=true";
          installCRDs = true;
          prometheus = {

            enabled = true;
            servicemonitor = {
              enabled = true;
              prometheusInstance = "monitoring";
            };
          };
          startupapicheck.podLabels."sidecar.istio.io/inject" = "false";
        };
      };
      extraObjects = [  # 2.3
        {
          apiVersion = "cert-manager.io/v1";
          kind = "ClusterIssuer";
          metadata.name = user.vars.k8sCert.name; # 1.3
          spec.ca.secretName = user.vars.k8sCert.name;
        }
      ];

    }

1. The module takes as input, any of which you may omit:

  #. A tree of *chart derivations*;
  #. the Turboprop library;
  #. the Nixpkgs for the current system (``pkgs``);
  #. the name and namespace of the service (``name``, ``namespace``); and
  #. user data specific to your flake.

2. The module has the output signature ``{builder, args, extraObjects}``.

  #. ``builder`` is the Turboprop builder that will create your derivation. Most often, you will use ``helmChart``; other builders exist for scenarios such as deploying a `collection of Kubernetes objects`_ or a `single remote YAML file`_. You may even `define your own builder`_.
  #. ``args`` are arguments passed to the builder. Refer to each builder's signature below.
  #. ``extraObjects`` are objects to deploy alongside the chart.

.. _collection of Kubernetes objects: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L12
.. _single remote YAML file: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/system/gateway-system/gateway-api/default.nix#L2
.. _define your own builder: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/breezewiki/default.nix#L6

Creating a service tree
=======================

Turboprop operates on *trees* of Nix modules, both in the filesystem sense (nested directories) and the Nix sense (nested attrsets), and uses `Haumea`_ to do so. A service tree consists of

#. an arbitrarily-named root, such as ``./services``, which contains
#. zero or more intermediate directories (we'll get to this), which each contain
#. directories representing Kubernetes namespaces, which each contain
#. Nix modules representing a templated deployment.

We'll start with building a flake containing two applications:

- the `Gateway API`_, and
- `Breezewiki`_ (through `app-template`_).

.. _Gateway API: https://gateway-api.sigs.k8s.io/
.. _Breezewiki: https://gitdab.com/cadence/breezewiki
.. _app-template: https://bjw-s.github.io/helm-charts/docs/app-template/

Normally, one would also deploy a Gateway controller, but this suffices for the example.

.. code-block:: nix

  # services/gateway-system/gateway-api/default.nix
  {lib, ...}: {
    builder = lib.builders.derivation;
    args = {
      src = lib.fetchers.remoteYAMLFile rec {
        version = "1.0.0";
        url = "https://github.com/kubernetes-sigs/gateway-api/releases/download/v${version}/experimental-install.yaml";
        hash = "sha256-bGAdzteHKpQNdvpmeuEmunGMtMbblw0Lq0kSjswRkqM=";
      };
    };
  }

.. code-block:: nix

  # services/default/breezewiki/default.nix
  {charts, lib, name, namespace ...}: {
    builder = lib.app-template.build;
    args = {
      mainImage = "quay.io/pussthecatorg/breezewiki:latest";
      values = let
        port = 10416;
      in {
        # app-template's schema can be found here:
        # https://github.com/bjw-s/helm-charts/blob/app-template-2.3.0/charts/library/common/values.yaml
        service.main.ports.http.port = port;
        route.main = {
          enabled = true;
          hostnames = ["${name}.example.com"];
          parentRefs = [
            {
              name = "gateway";
              inherit namespace;
              sectionName = "https";
            }
          ];
          rules = [
            {backendRefs = [{inherit name namespace port;}];}
          ];
        };
      };
    };
  }

Now build the flake::

  $ nix build
  $ ls -l result/*/*
  -r--r--r--   3 root root 88 Dec 31  1969 result/default/kustomization.yaml
  -r--r--r-- 130 root root 89 Dec 31  1969 result/gateway-system/kustomization.yaml

  result/default/breezewiki:
  total 12
  -r--r--r-- 1364 root root   90 Dec 31  1969 kustomization.yaml
  -r--r--r--    5 root root 2795 Dec 31  1969 SERVICE.yaml
  lrwxrwxrwx    4 root root   74 Dec 31  1969 SERVICE.yaml.drv -> /nix/store/sijp95rfkbijnrklmrb4smb9qvl7bd4v-yaml-stream-default-breezewiki

  result/gateway-system/gateway-api:
  total 768
  -r--r--r-- 1364 root root     90 Dec 31  1969 kustomization.yaml
  -r--r--r--   14 root root 775478 Dec 31  1969 SERVICE.yaml
  lrwxrwxrwx   11 root root     87 Dec 31  1969 SERVICE.yaml.drv -> /nix/store/0yi3y3b0lrgd71yrglgi7mjaxhk8khsm-copied-drv-gateway-system-gateway-api-1.0.0
  $ sha256sum result/gateway-system/gateway-api/SERVICE.yaml
  6c601dced7872a940d76fa667ae126ba718cb4c6db970d0bab49128ecc1192a3  result/gateway-system/gateway-api/SERVICE.yaml

Pretty cool, huh? Now to install the services...

.. code-block::

  $ kubectl apply -f result/namespaces.yaml
  namespace/default configured
  namespace/gateway-system created
  $ kubectl apply -k result/gateway-system/
  <...>
  $ kubectl apply -k result/default/breezewiki
  service/breezewiki created
  deployment.apps/breezewiki created
  error: resource mapping not found for name: "breezewiki" namespace: "default" from "result/default": no matches for kind "HTTPRoute" in version "gateway.networking.k8s.io/v1alpha2"
  ensure CRDs are installed first

Wait, what? A ``v1alpha2`` HTTP Route? That API isn't even *in* Gateway API v1. What gives?

Ordering services by provided APIs
==================================

Like most things in Nix, Helm derivations are *pure functions*: they have no room for external state. This means Helm cannot `poll a Kubernetes cluster`_ for data such as supported APIs, upon which charts such as ``app-template`` `depend`_ to calculate their output::

  {{- $routeKind := $routeObject.kind | default "HTTPRoute" -}}
  {{- $apiVersion := "gateway.networking.k8s.io/v1alpha2" -}}
  {{- if $rootContext.Capabilities.APIVersions.Has (printf "gateway.networking.k8s.io/v1beta1/%s" $routeKind) }}
    {{- $apiVersion = "gateway.networking.k8s.io/v1beta1" -}}
  {{- end -}}
  {{- if $rootContext.Capabilities.APIVersions.Has (printf "gateway.networking.k8s.io/v1/%s" $routeKind) }}
    {{- $apiVersion = "gateway.networking.k8s.io/v1" -}}
  {{- end -}}

This is a problem solved by Turboprop and all of its dependencies:

#. Helm `provides`_ the flags ``--api-versions`` and ``--kube-version`` with which to declare capabilities.
#. `nix-kube-generators`_' Helm builder `offers`_ the variables ``kubeVersion`` and ``apiVersions`` with reasonable defaults.
#. Turboprop accumulates APIs as it evaluates service modules in order, providing each module with the APIs generated before it.

Which order? Well, Hamuea loads and Turboprop evaluates in alphabetical order. And thus we arrive to the crux of the problem: ``gateway-api`` > ``default``. Luckily, it's trivial to solve::

  $ mkdir services/{1-gateway,2-main}
  $ mv services/gateway-system services/1-gateway
  $ mv services/default services/2-main
  $ nix build
  <...>
  $ grep -A1 'apiVersion: gateway' result/2-main/default/breezewiki/SERVICE.yaml
  apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute

.. _poll a Kubernetes cluster: https://helm.sh/docs/chart_template_guide/builtin_objects/
.. _depend: https://github.com/bjw-s/helm-charts/blob/c30dbd313011b63fa35e26f718d3161e476fe1fa/charts/library/common/templates/classes/_route.tpl#L9-L16
.. _provides: https://helm.sh/docs/helm/helm_template/
.. _nix-kube-generators: https://github.com/farcaller/nix-kube-generators
.. _offers: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L88-L89
.. _run-parts numbering: https://askubuntu.com/q/988664


*********
Reference
*********

Library
=======

mkDerivation
------------

``{charts?, user?}
-> {pname, version, src, serviceRoot, nsMetadata?, kubeVersion?, apiVersions?}
-> <derivation: a dir of Kustomization dirs>``

The main interface to Turboprop.

- **charts** (attrs, default: ``nixhelm.chartsDerivations.${system}``): A tree of fetched Helm charts.
- **user** (attrs, default: ``{}``): Additional data to be used by the service modules.

- **pname** (str): The name of the derivation.
- **version** (str): The version of the derivation.
- **src** (path): The root of the source tree from which to build the derivation.
- **serviceRoot** (path): The root of the service tree.
- **nsMetadata** (attrs, default: ``{}``): Additional metadata to attach to the generated namespaces.
- **kubeVersion** (str, default: ``pkgs.kubernetes.version``): The version of the Kubernetes cluster to target.
- **apiVersions** ([str], default: ``[]``): API versions to declare in addition to those provided by generated services.

.. _Nixhelm: https://github.com/farcaller/nixhelm/tree/master/charts


mkCharts
--------

mkChartsWithNixhelm
-------------------

Fetchers
========

gitChart
--------
``{name, version, url, hash, chartPath, vPrefixInRef?} -> <derivation: a dir containing a Helm chart>``

Fetch a Helm chart from a Git repository. Useful in the absence of a published Helm repo.

- **name** (str): The name of the Git repo.
- **version** (str): The tag to check out, which should resemble ``1.0.0``.
- **url** (str): The URL of the Git repo.
- **vPrefixInRef** (bool, default: ``false``): Whether the Git tag begins with an utterly redundant ``v``.
- **chartHash** (str): An `SRI-style hash`_.

helmChart
---------
``{repo, chart, version, chartHash?} -> <derivation: a dir containing a Helm chart>``

Re-export of `kubelib.downloadHelmChart`_.

- **repo** (str): The repository from which to download the chart.
- **chart** (str): The name of the chart.
- **version** (str): The version of the chart, which will also be the derivation's version.
- **chartHash** (str, default: `fakeHash`_): An `SRI-style hash`_.

.. _kubelib.downloadHelmChart: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L49
.. _fakeHash: https://github.com/NixOS/nixpkgs/blob/5b528f99f73c4fad127118a8c1126b5e003b01a9/lib/deprecated.nix#L304

remoteYAMLFile
--------------
``{version, url, hash} -> <derivation: a YAML file>``

Fetch a remote file. Useful for applications distributed as a YAML stream, e.g., the `Gateway API`_.

- **version** (str): The version of the application, which will also be the derivation's version.
- **url** (str): The URL from which to fetch the file.
- **hash** (str): An `SRI-style hash`_.

.. _Gateway API: https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.0.0

Builders
========

Builder functions build a service derivation.

Builders receive ``name`` and ``namespace`` through Turboprop, so these two variables will be documented once:

- **name** (str): The name of the service. Usually reflected in the label ``app.kubernetes.io/instance``, as well as the derivation's name.
- **namespace** (str): The namespace into which to deploy the service.

derivation
----------
``{name, namespace, src, ...} -> <derivation>``

Copy a derivation verbatim. Useful in conjunction with a fetcher that produces a single file, like ``lib.fetchers.remoteYAMLFile``.

- **src** (derivation): The derivation to copy.

helmChart
---------
``{name, namespace, chart, values?, includeCRDs?, kubeVersion?, apiVersions?} -> <derivation: a YAML file of Helm output>``

Wrapped re-export of `kubelib.fromHelm`_ that sets ``metadata.namespace`` on all templated objects lacking it. As such, its signature is identical to `kubelib.buildHelmChart`_.

- **chart** (derivation): The chart from which to build.
- **values** (attrs, default: ``{}``): Values to pass into the chart.
- **includeCRDs** (bool, default: ``true``): Whether to include CustomResourceDefinitions in the template output.
- **kubeVersion** (str, default: ``pkgs.kubernetes.version``): The Kubernetes version to target.
- **apiVersions** ([str], default: ``[]``): Sets `Capabilities.APIVersions`_.

.. _kubelib.fromHelm: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L123
.. _kubelib.buildHelmChart: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L82-L90
.. _Capabilities.APIVersions: https://helm.sh/docs/chart_template_guide/builtin_objects/#helm

app-template.build
------------------


.. _SRI-style hash: https://nixos.wiki/wiki/Nix_Hash

### namespaces

Assign extra metadata in ``namespaces.nix``. For example,
``svc = {labels."istio.io/rev" = "1-18-1"}``
is the equivalent of
``k label ns/svc istio.io/rev=1-18-1``

Modules
=======

Service (unbuilt)
-----------------

``{charts, lib, name, namespace, pkgs, user} -> {builder, args, extraObjects}``


Service (loaded)
-----------------

``{kubeVersion, apiVersion} -> {out, extra}``