Merge pull request #539 from achetronic/add-gitops-deployment-documentation

Add FluxCD GitOps documentation

docs/guides-gitops-using-fluxcd.md

@@ -0,0 +1,133 @@
+# GitOps using FluxCD (v2)
+
+FluxCD is a GitOps operator for Kubernetes. It synchronizes the status of the cluster from manifests allocated in
+different repositories (Git or Helm). This approach fits perfectly with External Secrets on clusters which are dynamically
+created, to get credentials with no manual intervention from the beginning.
+
+## Advantages
+
+This approach has several advantages as follows:
+
+* **Homogenize environments** allowing developers to use the same toolset in Kind in the same way they do in the cloud
+  provider distributions such as EKS or GKE. This accelerates the development
+* **Reduce security risks**, because credentials can be easily obtained, so temptation to store them locally is reduced.
+* **Application compatibility increase**: Applications are deployed in different ways, and sometimes they need to share
+  credentials. This can be done using External Secrets as a wire for them at real time.
+* **Automation by default** oh, come on!
+
+## The approach
+
+FluxCD is composed by several controllers dedicated to manage different custom resources. The most important
+ones are **Kustomization** (to clarify, Flux one, not Kubernetes' one) and **HelmRelease** to deploy using the approaches
+of the same names.
+
+External Secrets can be deployed using Helm [as explained here](guides-getting-started.md). The deployment includes the
+CRDs if enabled on the `values.yaml`, but after this, you need to deploy some `SecretStore` to start
+getting credentials from your secrets manager with External Secrets.
+
+> The idea of this guide is to deploy the whole stack, using flux, needed by developers not to worry about the credentials,
+> but only about the application and its code.
+
+## The problem
+
+This can sound easy, but External Secrets is deployed using Helm, which is managed by the HelmController,
+and your custom resources, for example a `ClusterSecretStore` and the related `Secret`, are often deployed using a
+`kustomization.yaml`, which is deployed by the KustomizeController.
+
+Both controllers manage the resources independently, at different moments, with no possibility to wait each other.
+This means that we have a wonderful race condition where sometimes the CRs (`SecretStore`,`ClusterSecretStore`...) tries
+to be deployed before than the CRDs needed to recognize them.
+
+## The solution
+
+Let's see the conditions to start working on a solution:
+
+* The External Secrets operator is deployed with Helm, and admits disabling the CRDs deployment
+* The race condition only affects the deployment of `CustomResourceDefinition` and the CRs needed later
+* CRDs can be deployed directly from the Git repository of the project using a Flux `Kustomization`
+* Required CRs can be deployed using a Flux `Kustomization` too, allowing dependency between CRDs and CRs
+* All previous manifests can be applied with a Kubernetes `kustomization`
+
+## Create the main kustomization
+
+To have a better view of things needed later, the first manifest to be created is the `kustomization.yaml`
+
+```yaml
+{% include 'gitops/kustomization.yaml' %}
+```
+
+## Create the secret
+
+To access your secret manager, External Secrets needs some credentials. They are stored inside a Secret, which is intended
+to be deployed by automation as a good practise. This time, a placeholder called `secret-token.yaml` is show as an example:
+
+```yaml
+# The namespace.yaml first
+{% include 'gitops/namespace.yaml' %}
+```
+
+```yaml
+{% include 'gitops/secret-token.yaml' %}
+```
+
+## Creating the references to repositories
+
+Create a manifest called `repositories.yaml` to store the references to external repositories for Flux
+
+```yaml
+{% include 'gitops/repositories.yaml' %}
+```
+
+## Deploy the CRDs
+
+As mentioned, CRDs can be deployed using the official Helm package, but to solve the race condition, they will be deployed
+from our git repository using a Kustomization manifest called `deployment-crds.yaml` as follows:
+
+```yaml
+{% include 'gitops/deployment-crds.yaml' %}
+```
+
+## Deploy the operator
+
+The operator is deployed using a HelmRelease manifest to deploy the Helm package, but due to the special race condition,
+the deployment must be disabled in the `values` of the manifest called `deployment.yaml`, as follows:
+
+```yaml
+{% include 'gitops/deployment.yaml' %}
+```
+
+## Deploy the CRs
+
+Now, be ready for the arcane magic. Create a Kustomization manifest called `deployment-crs.yaml` with the following content:
+
+```yaml
+{% include 'gitops/deployment-crs.yaml' %}
+```
+
+There are several interesting details to see here, that finally solves the race condition:
+
+1. First one is the field `dependsOn`, which points to a previous Kustomization called `external-secrets-crds`. This
+   dependency forces this deployment to wait for the other to be ready, before start being deployed.
+2. The reference to the place where to find the CRs
+   ```yaml
+   path: ./infrastructure/external-secrets/crs
+   sourceRef:
+    kind: GitRepository
+    name: flux-system
+   ```
+   Custom Resources will be searched in the relative path `./infrastructure/external-secrets/crs` of the GitRepository
+   called `flux-system`, which is a reference to the same repository that FluxCD watches to synchronize the cluster.
+   With fewer words, a reference to itself, but going to another directory called `crs`
+
+Of course, allocate inside the mentioned path `./infrastructure/external-secrets/crs`, all the desired CRs to be deployed,
+for example, a manifest `clusterSecretStore.yaml` to reach your Hashicorp Vault as follows:
+
+```yaml
+{% include 'gitops/crs/clusterSecretStore.yaml' %}
+```
+
+## Results
+
+At the end, the required files tree is shown in the following picture:
+
+![FluxCD files tree](./pictures/screenshot_gitops_final_directory_tree.png)

docs/snippets/gitops/crs/clusterSecretStore.yaml

@@ -0,0 +1,17 @@
+apiVersion: external-secrets.io/v1alpha1
+kind: ClusterSecretStore
+metadata:
+  name: vault-backend-global
+spec:
+  provider:
+    vault:
+      server: "https://vault.your-domain.com"
+      path: secret
+      version: v2
+      auth:
+        # points to a secret that contains a vault token
+        # https://www.vaultproject.io/docs/auth/token
+        tokenSecretRef:
+          name: "vault-token-global"
+          key: "token"
+          namespace: external-secrets

docs/snippets/gitops/crs/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- clusterSecretStore.yaml

docs/snippets/gitops/deployment-crds.yaml

@@ -0,0 +1,13 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
+kind: Kustomization
+metadata:
+  name: external-secrets-crds
+  namespace: flux-system
+spec:
+  interval: 10m
+  path: ./deploy/crds
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: external-secrets

docs/snippets/gitops/deployment-crs.yaml

@@ -0,0 +1,15 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
+kind: Kustomization
+metadata:
+  name: external-secrets-crs
+  namespace: flux-system
+spec:
+  dependsOn:
+    - name: external-secrets-crds
+  interval: 10m
+  path: ./infrastructure/external-secrets/crs
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system

docs/snippets/gitops/deployment.yaml

@@ -0,0 +1,28 @@
+# How to manage values files. Ref: https://fluxcd.io/docs/guides/helmreleases/#refer-to-values-inside-the-chart
+# How to inject values: https://fluxcd.io/docs/guides/helmreleases/#cloud-storage
+---
+apiVersion: helm.toolkit.fluxcd.io/v2beta1
+kind: HelmRelease
+metadata:
+  name: external-secrets
+  namespace: flux-system
+spec:
+  # Override Release name to avoid the pattern Namespace-Release
+  # Ref: https://fluxcd.io/docs/components/helm/api/#helm.toolkit.fluxcd.io/v2beta1.HelmRelease
+  releaseName: external-secrets
+  targetNamespace: external-secrets
+  interval: 10m
+  chart:
+    spec:
+      chart: external-secrets
+      version: 0.3.9
+      sourceRef:
+        kind: HelmRepository
+        name: external-secrets
+        namespace: flux-system
+  values:
+    installCRDs: false
+
+  # Ref: https://fluxcd.io/docs/components/helm/api/#helm.toolkit.fluxcd.io/v2beta1.Install
+  install:
+    createNamespace: true

docs/snippets/gitops/kustomization.yaml

@@ -0,0 +1,20 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+# Deploy the Vault access secret
+- namespace.yaml
+- secret-token.yaml
+
+# Deploy the repositories
+- repositories.yaml
+
+# Deploy the CRDs
+- deployment-crds.yaml
+
+# Deploy the operator
+- deployment.yaml
+
+# Deploy default Custom Resources from 'crs' directory
+# INFO: This depends on the CRDs deployment. Will happen after it
+- deployment-crs.yaml

docs/snippets/gitops/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: external-secrets

docs/snippets/gitops/repositories.yaml

@@ -0,0 +1,20 @@
+# Reference to Helm repository
+apiVersion: source.toolkit.fluxcd.io/v1beta1
+kind: HelmRepository
+metadata:
+  name: external-secrets
+  namespace: flux-system
+spec:
+  interval: 10m
+  url: https://charts.external-secrets.io
+---
+apiVersion: source.toolkit.fluxcd.io/v1beta1
+kind: GitRepository
+metadata:
+  name: external-secrets
+  namespace: flux-system
+spec:
+  interval: 10m
+  ref:
+    branch: main
+  url: http://github.com/external-secrets/external-secrets

docs/snippets/gitops/secret-token.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: vault-token-global
+  namespace: external-secrets
+stringData:
+  # This token must be patched by overlays. Not here for security reasons
+  token: change-me-placeholder

hack/api-docs/mkdocs.yml

@@ -36,6 +36,7 @@ nav:
     - Multi Tenancy: guides-multi-tenancy.md
     - Metrics: guides-metrics.md
     - Using Latest Image: guides-using-latest-image.md
+    - GitOps using FluxCD: guides-gitops-using-fluxcd.md
   - Provider:
     - AWS:
       - Secrets Manager: provider-aws-secrets-manager.md