mirror of
https://git.sr.ht/~goorzhel/turboprop
synced 2024-12-14 11:37:37 +00:00
No description
adr | ||
charts | ||
lib | ||
services | ||
src | ||
user | ||
.gitignore | ||
flake.lock | ||
flake.nix | ||
LICENSE | ||
Makefile | ||
README.rst |
.. vim: set et sw=2: ######### Turboprop ######### Problem: You have twenty or thirty Helm releases, all of which you template semi-manually to `retain WYSIWYG control`_. Deploying new applications involves tremendous amounts of copy-pasta. Solution: Use Nix. With Nix, you can `ensure chart integrity`_, `generate repetitive data`_ in `subroutines`_, and `easily reuse variable data`_. Turboprop templates your Helm charts for you, making an individual Nix derivation of each one; each of these derivations is then gathered into a mega-derivation complete with Kustomizations for every namespace and service. In short, you're two commands away from full cluster reconciliation:: nix build && kubectl diff -k ./result .. _retain WYSIWYG control: https://github.com/kubernetes-sigs/kustomize/blob/bfb00ecb2747dc711abfc27d9cf788ca1d7c637b/examples/chart.md#best-practice .. _ensure chart integrity: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/charts/intel/device-plugins-gpu/default.nix#L5 .. _generate repetitive data: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L25-26 .. _subroutines: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L8-10 .. _easily reuse variable data: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/system/kube-system/csi-driver-nfs/default.nix#L16 *************** Acknowledgments *************** Without `Vladimir Pouzanov`_'s "`Nix and Kubernetes\: Deployments Done Right`_" (and `its notes`_), this project would not exist. I also used `heywoodlh's Kubernetes flake`_ as a starting point early on. .. _Vladimir Pouzanov: https://github.com/farcaller .. _Nix and Kubernetes\: Deployments Done Right: https://media.ccc.de/v/nixcon-2023-35290-nix-and-kubernetes-deployments-done-right .. _its notes: https://gist.github.com/farcaller/c87c03fbb55eaeaeb840b938455f37ff .. _heywoodlh's Kubernetes flake: https://github.com/heywoodlh/flakes/blob/aa5a52a/kube/flake.nix ********************** Usage and architecture ********************** Getting started =============== To start, add this flake to your flake's inputs, along with ``nixpkgs`` and ``flake-utils``: .. code-block:: nix { inputs = { nixpkgs.url = "github:NixOS/nixpkgs"; flake-utils.url = "github:numtide/flake-utils"; turboprop = { url = "sourcehut:~goorzhel/turboprop"; inputs.nixpkgs.follows = "nixpkgs"; }; }; <...> } Next, split your flake's output into two sections: #. One running in pure Nix that "rakes" your module data into a *tree* (more on that later); and #. one that builds derivations from your module data, using your current system's ``nixpkgs``. .. code-block:: nix { <...> outputs = {self, nixpkgs, flake-utils, turboprop}: let rake = turboprop.rake in { # I'll explain the distinction in a later chapter systemServiceData = rake.leaves ./system; serviceData = rake.leaves ./services; repos = rake.leaves ./charts; namespaces = rake.namespaces { roots = [./system ./services]; extraMetadata = import ./namespaces.nix; }; } // flake-utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; overlays = [devshell.overlays.default]; }; turbo = turboprop.packages.${system}; # We'll get back to this too flakeBuilders = turbo.flakeBuilders {}; namespaces = <...>; paths = <...>; in { packages.default = turbo.mkDerivation { # This too }; } ); } Now set that aside for the time being. Example service module ====================== This is a module that defines a *service derivation*: .. code-block:: nix { charts, lib, user, ... }: { # 1 builder = lib.builders.helmChart; # 1.2; 2.1 args = { # < - - - - - - - - - - - 2.2 chart = charts.jetstack.cert-manager; # 1.1 values = { featureGates = "ExperimentalGatewayAPISupport=true"; installCRDs = true; prometheus = { enabled = true; servicemonitor = { enabled = true; prometheusInstance = "monitoring"; }; }; startupapicheck.podLabels."sidecar.istio.io/inject" = "false"; }; }; extraObjects = [ # 2.3 { apiVersion = "cert-manager.io/v1"; kind = "ClusterIssuer"; metadata.name = user.vars.k8sCert.name; # 1.3 spec.ca.secretName = user.vars.k8sCert.name; } ]; } 1. The module takes as input: #. A tree of *chart derivations*; #. the Turboprop library; and #. user data specific to your flake. You may `omit any of these input variables`_ if they're not used. 2. The module has the output signature ``{builder, args, extraObjects}``. #. ``builder`` is the Turboprop builder that will create your derivation. Most often, you will use ``helmChart``; other builders exist for scenarios such as deploying a `collection of Kubernetes objects`_ or a `single remote YAML file`_. You may even `define your own builder`_. #. ``args`` are arguments passed to the builder. Refer to each builder's signature below. #. ``extraObjects`` are objects to deploy alongside the chart. .. _omit any of these input variables: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/system/gateway-system/gateway-api/default.nix#L1 .. _collection of Kubernetes objects: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/gateway/default.nix#L12 .. _single remote YAML file: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/system/gateway-system/gateway-api/default.nix#L2 .. _define your own builder: https://git.sr.ht/~goorzhel/kubernetes/tree/f3cba6831621288228581b7ad7b6762d6d58a966/item/services/svc/breezewiki/default.nix#L6 Trees of Nix modules ==================== Turboprop's operates on *trees* of Nix modules, both in the filesystem sense (nested directories) and the Nix sense (nested attrsets). A service tree, then, consists of #. an arbitrarily-named root, such as ``./services``, which contains #. directories representing Kubernetes namespaces, which each contain #. Nix modules representing a templated deployment. This metaphor extends to charts. Both Turboprop and nixhelm, from which Turboprop borrows heavily, contain a chart tree: #. an arbitrarily-named root, ``./charts``, which contains #. directories representing Helm repositories, which each contain #. Nix modules representing a(n untemplated) chart. In practice:: $ nix run nixpkgs#tree -- ~/src/kubernetes/{chart,service}s --noreport /home/ag/src/kubernetes/charts ├── kubernetes-dashboard │ └── kubernetes-dashboard │ └── default.nix <...> /home/ag/src/kubernetes/services <...> ├── istio-system │ ├── 1-18-1 │ │ └── default.nix │ └── kiali │ └── default.nix └── svc ├── breezewiki │ └── default.nix <...> └── vaultwarden └── default.nix You may have noticed that, if neither Nixhelm nor Turboprop provide a chart you need, you may define it within your flake. (PRs welcome, though.) Builders and flake builders =========================== *Builders* are Nix functions that build a derivation, be it of a chart, a service, or a service's extra objects. *Flake builders* are Nix functions that contribute toward the flake's output, using the modules' defined builders and your flake's directory structure to render a tree of Kustomizations. Recall the definition of ``flakeBuilders`` in the system-specific let-in midway through your flake's outputs: .. code-block:: nix { <...> outputs = {<...>, turboprop}: <...> // flake-utils.lib.eachDefaultSystem (system: let <...> turbo = turboprop.packages.${system}; flakeBuilders = turbo.flakeBuilders {}; namespaces = <...>; paths = <...>; in { <...> }); } .. TODO: Defining multiple flake builders sounds like boilerplate. Can't all of that be wrapped into turbo.mkDerivation? Services and system services ============================ .. TODO - Helm cannot see your cluster from inside the sandbox. - Custom APIs must be gathered and passed explicitly. - This can't be done in one step due to infinite recursion. Finally: your flake's output ============================ ********* Reference ********* Utilities ========= Charts ====== Turboprop overlays `its own charts`_ atop `Nixhelm's`_. .. _its own charts: https://git.sr.ht/~goorzhel/turboprop/tree/main/item/charts .. _Nixhelm's: https://github.com/farcaller/nixhelm/tree/master/charts Fetchers ======== gitChart -------- ``{name, version, url, hash, chartPath, vPrefixInRef?} -> <derivation: a dir containing a Helm chart>`` Fetch a Helm chart from a Git repository. Useful in the absence of a published Helm repo. - **name** (str): The name of the Git repo. - **version** (str): The tag to check out, which should resemble ``1.0.0``. - **url** (str): The URL of the Git repo. - **vPrefixInRef** (bool, default: ``false``): Whether the Git tag begins with an utterly redundant ``v``. - **chartHash** (str): An `SRI-style hash`_. helmChart --------- ``{repo, chart, version, chartHash?} -> <derivation: a dir containing a Helm chart>`` Re-export of `kubelib.downloadHelmChart`_. - **repo** (str): The repository from which to download the chart. - **chart** (str): The name of the chart. - **version** (str): The version of the chart, which will also be the derivation's version. - **chartHash** (str, default: `fakeHash`_): An `SRI-style hash`_. .. _kubelib.downloadHelmChart: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L49 .. _fakeHash: https://github.com/NixOS/nixpkgs/blob/5b528f99f73c4fad127118a8c1126b5e003b01a9/lib/deprecated.nix#L304 remoteYAMLFile -------------- ``{version, url, hash} -> <derivation: a YAML file>`` Fetch a remote file. Useful for applications distributed as a YAML stream, e.g., the `Gateway API`_. - **version** (str): The version of the application, which will also be the derivation's version. - **url** (str): The URL from which to fetch the file. - **hash** (str): An `SRI-style hash`_. .. _Gateway API: https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.0.0 Builders ======== Builder functions build a service derivation. Builders receive ``name`` and ``namespace`` through the flake builders, so these two variables will be documented once: - **name** (str): The name of the service. Usually reflected in the label ``app.kubernetes.io/instance``, as well as the derivation's name. - **namespace** (str): The namespace into which to deploy the service. derivation ---------- ``{name, namespace, src, ...} -> <derivation>`` Copy a derivation verbatim. Useful in conjunction with a fetcher that produces a single file, like ``lib.fetchers.remoteYAMLFile``. - **src** (derivation): The derivation to copy. helmChart --------- ``{name, namespace, chart, values?, includeCRDs?, kubeVersion?, apiVersions?} -> <derivation: a YAML file of Helm output>`` Wrapped re-export of `kubelib.fromHelm`_ that sets ``metadata.namespace`` on all templated objects lacking it. As such, its signature is identical to `kubelib.buildHelmChart`_. - **chart** (derivation): The chart from which to build. - **values** (attrset, default: ``{}``): Values to pass into the chart. - **includeCRDs** (bool, default: ``true``): Whether to include CustomResourceDefinitions in the template output. - **kubeVersion** (str, default: ``pkgs.kubernetes.version``): The Kubernetes version to target. - **apiVersions** ([str], default: ``[]``): Sets `Capabilities.APIVersions`_. .. _kubelib.fromHelm: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L123 .. _kubelib.buildHelmChart: https://github.com/farcaller/nix-kube-generators/blob/cdb5810a8d5d553cdd0d04fa53378d5105b529b2/lib/default.nix#L82-L90 .. _Capabilities.APIVersions: https://helm.sh/docs/chart_template_guide/builtin_objects/#helm Flake builders ============== .. _SRI-style hash: https://nixos.wiki/wiki/Nix_Hash ## Architecture Services expected to provide custom APIs (e.g.: Gateway API, Istio, Longhorn) go in ``./system``. All others in ``./services``, including system-service charts dependent on other APIs. This prevents infinite recursion when gathering APIs. Each of the leaves of the ``services`` attrsets is a derivation (explained better in ``lib/flake-builders.nix``). Here, they are gathered into one mega-derivation, with Kustomizations at each level for usage with ``k apply -k $path``. ### namespaces Assign extra metadata in ``namespaces.nix``. For example, ``svc = {labels."istio.io/rev" = "1-18-1"}`` is the equivalent of ``k label ns/svc istio.io/rev=1-18-1``