external-secrets/design/005-secret-generator-group.md

---
title: Secret Generators
version: v1alpha1
authors: Christian Hünig, Jan Steffen, Moritz Johner
creation-date: 2022-07-08
status: draft
---

Secret Generators

Table of Contents

// autogen please

Summary

There are use-cases where a secret could be generated by the operator instead of using pre-existing ones from the secret store. For example a MongoDB could be spun up using a password generated by the operator. This password is then used by an application in the cluster to access the database. No human ever needs to read that secret or have to specify it beforehand. A similar feature is available in a project called VaultOperator which tries to solve similar problems like this operator just for Hashicorp Vault as a secret store.

Motivation

The motivation is to boost adoption of zero-trust secrets management and to make it simple to create secure secrets for machine users of services running inside clusters (i.e. databases etc).

Goals

• enable zero-trust secret management for purely technical and internal users
• enable secure passwords by default
• reduce error rates when creating RSA or ECDSA private keys

Non-Goals

• auto-rotation of secrets

Proposal

Implement several generators to support not only a number of data types but also generators that interface with external APIs.

API

We add a new resource group generators.external-secrets.io. For each generator we add a new CRD, e.g. for password generation, VaultPKI, ECR, GCR etc. This custom resource contains all the configuration details to create a secure value. It can be referenced from a Kind=ExternalSecret or Kind=PushSecret.

The controller generates the value and passes it onto the Kind=Secret with all the values that have been fetched from the provider. When used with a Kind=PushSecret it will generate the value and push it to the configured provider.

A generator generates a known set of outputs, e.g. username and password, private_key and certificate or others.

Example CRD

The following custom resource Kind=VaultPKIBackend contains all the parameters for issuing a certificate via HashiCorp Vault PKI engine.

apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultPKIBackend
metadata:
  name: my-vault-pki-creds
spec:
  # specific params for generating
  commonName: example.com
  altNames:
    - example.org
    - localhost
  # ... more opts
  auth: {} # service account etc. etc.

This resource can be referenced from an ExternalSecret or PushSecret. The ExternalSecret controller generates the secret value and it can be post-processed with existing features like template or key-rewrite.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  # ...
  dataFrom:

    # the generator is referenced like here
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: VaultPKIBackend
          name: foobar

SecretStoreRef vs. SourceRef

In order to accomodate the generator implementation we need to enhance the spec.secretStoreRef functionality and add data[].sourceRef and dataFrom[].sourceRef that allows a user to reference generator resources. This allows us to:

• improve composability and use multiple SecretStores within a single ExternalSecret
• use generators from both data[] and dataFrom[] nodes

If sourceRef is not defined we fall back to spec.secretStoreRef.

spec:
  data:
    - secretKey: foo
      remoteRef: {}
      # can point to a SecretStore
      # or a generator/generatorRef
      sourceRef:
        storeRef:
          name: foo
  dataFrom:
    # allow using multiple stores from a single ExternalSecret
    - find: { ... }
      sourceRef:
        storeRef:
          name: "foo"
    - find: { ... }
      sourceRef:
        storeRef:
          name: "bar"

Specifying more than one attribute within a sourceRef is not supported and must be rejected.

sourceRef:
  storeRef: {}
  generatorRef: {} # ILLEGAL!

PushSecret example

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: "hello-world"
spec:
  secretStoreRefs: [] # ...
  selector:
    # we can consider supporting multiple generators per PushSecret
    # but need to take care of key collision
    generatorRef:
      apiVersion: generators.external-secrets.io/v1alpha1
      kind: VaultPKIBackend
      name: foobar
  data: {} # ...

Password Generator Example

apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: foo
spec:
  characterSet: "..." # TBD
  length: 42
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  # ...
  dataFrom:
    - generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: Password
        name: foo

Example: multiple generators with key rewrite

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  # ...
  dataFrom:
    # using multiple generators
    # this will produce the following values:
    # foo-username
    # foo-password
    # bar-username
    # bar-password
    - generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECR
        name: foo
      rewrite:
        - regexp:
            source: .+
            target: foo-$1
    - generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECR
        name: bar
      rewrite:
        - regexp:
            source: .+
            target: bar-$1

Composability and extensibility

We can maintain a set of in-tree generators (e.g. for password, binary, hash, alphanumeric, VaultPKI, VaultAWS, ECR, GCR, ACR, Harbor...) but that doesn't provide an extensible interface for users. We can consider the following two approaches (possibly more) that provide extensibility:

Note: This section is just an outlook and nothing that we aim to implement right away. We will re-iterate on this feature and see if we need to make the generator implementation more extensible.

(1) HTTP/gRPC interface

We can provide an in-tree generator that interfaces through a well-known protocol (e.g. HTTP webhook or gRPC).

apiVersion: generators.external-secrets.io/v1alpha1
kind: HTTPWebhook
metadata:
  name: example
spec:
  # similar to webhook provider
  # see: https://external-secrets.io/v0.5.8/provider-webhook/
  url: "http://httpbin.org/get?parameter={{ .remoteRef.key }}"
  result:
    jsonPath: "$.args.parameter"
  headers:
    Content-Type: application/json
    Authorization: Basic {{ print .auth.username ":" .auth.password | b64enc }}
  secrets:
    - name: auth
      secretRef:
        name: webhook-credentials

Pros:

• simple to extend from a user perspective
• simple simple to add from external-secrets perspective

Cons:

• prone to network errors
• high coupling of secret sync and generator

(2) with an intermediate GenerateRequest

This is very similar to cert-manager's external issuer API that allows users to implement their own controller to issue certifiates. We maintain some core generator controllers and leave the implementation of others up to the user.

     ┌───────────────────┐         ┌──────────────────────┐
     │                   │   1     │                      │    2       ┌───────────────┐
     │  ExternalSecret/  │ create  │ Kind:GenerateRequest │ generate   │               │
     │  PushSecret       ├────────►│ generatorRef:        │◄───────────┤  FooGenerator │
     │  Controller       │◄────────┤   kind: Foo          │            │  Controller   │
     │                   │  result │   name: bar          │            │               │
     └─────────┬─────────┘   3     │ result:              │            └───────────────┘
               │                   │   username: foo      │
          4    │                   │   password: bar      │
        create │                   │                      │
               │                   └──────────────────────┘
               │
               ▼
     ┌──────────────────┐
     │                  │
     │ Kind: Secret     │
     │ data:            │
     │   username: foo  │
     │   password: bar  │
     │                  │
     └──────────────────┘

Pros:

• loose coupling of generator and secret sync workflows

Cons:

• very high complexity
• hard to extend from a user perspective

Behavior

with ExternalSecret

Every time an ExternalSecret reconciles the generator will produce a new set of output values. Hence, the secrets are rotated. The frequency can be controlled with refreshInterval. With refreshInterval=0 the Secret will be created exactly once and will not rotate.

with PushSecret

The generator produces an output once which is pushed to the provider. If a secret already exists in that place it can not be overridden.

---