external-secrets/design/005-secret-generator-group.md

```yaml
---
title: Secret Generators
version: v1alpha1
authors: Christian Hünig, Jan Steffen, Moritz Johner
creation-date: 2022-07-08
status: draft
---
```

# Secret Generators

## Table of Contents

<!-- toc -->
// autogen please
<!-- /toc -->

## Summary

There are use-cases where a secret could be generated by the operator instead of using pre-existing ones from the secret store. For example a MongoDB could be spun up using a password generated by the operator. This password is then used by an application in the cluster to access the database. No human ever needs to read that secret or have to specify it beforehand. A similar feature is available in a project called [VaultOperator](https://github.com/finleap-connect/vaultoperator/) which tries to solve similar problems like this operator just for Hashicorp Vault as a secret store.

## Motivation

The motivation is to boost adoption of zero-trust secrets management
and to make it simple to create secure secrets for machine users
of services running inside clusters (i.e. databases etc).

### Goals

- enable zero-trust secret management for purely technical and internal users
- enable secure passwords by default
- reduce error rates when creating RSA or ECDSA private keys

### Non-Goals

- auto-rotation of secrets

## Proposal

Implement several generators to support not only a number of data types but also generators that interface with external APIs.

### API

We add a new resource group `generators.external-secrets.io`. For each generator we add a new CRD, e.g. for password generation, VaultPKI, ECR, GCR etc.
This custom resource contains all the configuration details to create a secure value.
It can be referenced from a `Kind=ExternalSecret` or `Kind=PushSecret`.

The controller generates the value and passes it onto the `Kind=Secret` with all the values
that have been fetched from the provider.
When used with a `Kind=PushSecret` it will generate the value and push it to the configured provider.

A generator generates a known set of outputs, e.g. `username` and `password`, `private_key` and `certificate` or others.

#### Example CRD

The following custom resource `Kind=VaultPKIBackend` contains all the parameters for issuing a certificate via HashiCorp Vault PKI engine.

```yaml
apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultPKIBackend
metadata:
  name: my-vault-pki-creds
spec:
  # specific params for generating
  commonName: example.com
  altNames:
    - example.org
    - localhost
  # ... more opts
  auth: {} # service account etc. etc.
```

This resource can be **referenced** from an ExternalSecret or PushSecret.
The ExternalSecret controller generates the secret value and it can be post-processed with
existing features like template or key-rewrite.

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  # ...
  dataFrom:

    # the generator is referenced like here
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: VaultPKIBackend
          name: foobar
```

#### `SecretStoreRef` vs. `SourceRef`

In order to accomodate the generator implementation we need to enhance the `spec.secretStoreRef` functionality
and add `data[].sourceRef` and `dataFrom[].sourceRef` that allows a user to reference generator resources.
This allows us to:

- improve composability and **use multiple SecretStores** within a single ExternalSecret
- use generators from both `data[]` and `dataFrom[]` nodes

If `sourceRef` is not defined we fall back to `spec.secretStoreRef`.

```yaml
spec:
  data:
    - secretKey: foo
      remoteRef: {}
      # can point to a SecretStore
      # or a generator/generatorRef
      sourceRef:
        storeRef:
          name: foo
  dataFrom:
    # allow using multiple stores from a single ExternalSecret
    - find: { ... }
      sourceRef:
        storeRef:
          name: "foo"
    - find: { ... }
      sourceRef:
        storeRef:
          name: "bar"
```

Specifying more than one attribute within a sourceRef is not supported and must be rejected.

```
sourceRef:
  storeRef: {}
  generatorRef: {} # ILLEGAL!
```

<details>
<summary>PushSecret example</summary>

```yaml
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: "hello-world"
spec:
  secretStoreRefs: [] # ...
  selector:
    # we can consider supporting multiple generators per PushSecret
    # but need to take care of key collision
    generatorRef:
      apiVersion: generators.external-secrets.io/v1alpha1
      kind: VaultPKIBackend
      name: foobar
  data: {} # ...
```

</details>

<details>
<summary>Password Generator Example</summary>

```yaml
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: foo
spec:
  characterSet: "..." # TBD
  length: 42
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  # ...
  dataFrom:
    - generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: Password
        name: foo
```

</details>

<details>
<summary>Example: multiple generators with key rewrite</summary>

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  # ...
  dataFrom:
    # using multiple generators
    # this will produce the following values:
    # foo-username
    # foo-password
    # bar-username
    # bar-password
    - generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECR
        name: foo
      rewrite:
        - regexp:
            source: .+
            target: foo-$1
    - generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECR
        name: bar
      rewrite:
        - regexp:
            source: .+
            target: bar-$1
```

</details>

### Composability and extensibility

We can maintain a set of in-tree generators (e.g. for password, binary, hash, alphanumeric, VaultPKI, VaultAWS, ECR, GCR, ACR, Harbor...) but that doesn't provide an extensible interface for users.
We can consider the following two approaches (possibly more) that provide extensibility:

> Note: This section is just an outlook and nothing that we aim to implement right away.
>       We will re-iterate on this feature and see if we need to make the
>       generator implementation more extensible.

#### (1) HTTP/gRPC interface

We can provide an in-tree generator that interfaces through a well-known protocol (e.g. HTTP webhook or gRPC).

```yaml
apiVersion: generators.external-secrets.io/v1alpha1
kind: HTTPWebhook
metadata:
  name: example
spec:
  # similar to webhook provider
  # see: https://external-secrets.io/v0.5.8/provider-webhook/
  url: "http://httpbin.org/get?parameter={{ .remoteRef.key }}"
  result:
    jsonPath: "$.args.parameter"
  headers:
    Content-Type: application/json
    Authorization: Basic {{ print .auth.username ":" .auth.password | b64enc }}
  secrets:
    - name: auth
      secretRef:
        name: webhook-credentials
```

Pros:

- simple to extend from a user perspective
- simple simple to add from external-secrets perspective

Cons:

- prone to network errors
- high coupling of secret sync and generator

#### (2) with an intermediate GenerateRequest

This is very similar to [cert-manager's external issuer API](https://cert-manager.io/docs/contributing/external-issuers/) that allows users to implement their own controller to issue certifiates.
We maintain some core generator controllers and leave the implementation of others up to the user.

```
     ┌───────────────────┐         ┌──────────────────────┐
     │                   │   1     │                      │    2       ┌───────────────┐
     │  ExternalSecret/  │ create  │ Kind:GenerateRequest │ generate   │               │
     │  PushSecret       ├────────►│ generatorRef:        │◄───────────┤  FooGenerator │
     │  Controller       │◄────────┤   kind: Foo          │            │  Controller   │
     │                   │  result │   name: bar          │            │               │
     └─────────┬─────────┘   3     │ result:              │            └───────────────┘
               │                   │   username: foo      │
          4    │                   │   password: bar      │
        create │                   │                      │
               │                   └──────────────────────┘
               │
               ▼
     ┌──────────────────┐
     │                  │
     │ Kind: Secret     │
     │ data:            │
     │   username: foo  │
     │   password: bar  │
     │                  │
     └──────────────────┘
```

Pros:

- loose coupling of generator and secret sync workflows

Cons:

- very high complexity
- hard to extend from a user perspective

### Behavior

#### with ExternalSecret

Every time an ExternalSecret reconciles the generator will produce a new set of output values.
Hence, the secrets are rotated. The frequency can be controlled with `refreshInterval`.
With `refreshInterval=0` the Secret will be created exactly once and will not rotate.

#### with PushSecret

The generator produces an output once which is pushed to the provider. If a secret already exists in that place it can not be overridden.

---