Admission reports are created at resource admission time.
Every time a resource is being admitted, the webhook looks up the policies that need to be evaluated against the resource and a report is created containing the results for this particular admission request.
```mermaid
sequenceDiagram
Webhook->>Policy cache: Lookup policies for admitted resource
loop Evaluation
Webhook->>Engine: Evaluate policies against admitted resource
Engine-->>Webhook: Collect and aggregate results
end
Webhook->>Report: Create admission report
```
It's important to note that a resource can be admitted multiple times, CREATE/UPDATE/DELETE, etc... the exact same resource can be admitted multiple times and it's perfectly fine.
One consequence of this is that Kyverno can produce a lot of admission reports for the same resource.
Keeping all those admission reports would be highly inefficient and they would accumulate in the cluster as new admission reports come in.
To prevent admission reports from piling up in the cluster, Kyverno continuously aggregates admission reports into a single one per underlying resource.
B(Background scan controller) --> E(Evaluates policies against existing resource)
E(Evaluates policies against existing resource) -->|create/update| REPORT(Background scan report)
```
### Policy reports
Policy reports (and cluster policy reports) are the result of aggregating admission reports and background scan reports per policy, at the namespace level (for namespaced resources) and at the cluster level (for cluster wide resources).
Please note that only aggregated admission reports are considered, it would not make sense to consider short lived admission reports here.
To reduce the computation power needed, especially when resources change frequently, we introduce a queue and a delay (`30s`):
- Watch when an admission report and/or background scan report is CREATED, UPDATED or DELETED
- Put the namespace of the report in the queue (`""` if the report is clustered) with a delay of `30s`
- The queue guarantees that only a single worker will process an item at the same time
- When the worker gets the namespace from the queue it lists all admission reports and background scan reports for that particular namespace, aggregates them together and reconciles the existing policy reports
This process can have a significant impact on the number of API calls and the memory footprint, especially when we have large reports.
To mitigate this we use LIST api calls with pagination and reports are aggregated before loading the next page.
- background scan controller watches resources/policies and maintains an up to date background scan report
## Storage considerations
The system stores everything in etcd, admission reports (aggregated and short lived ones), background scan reports, and policy reports/cluster policy reports.
Short lived admission reports are ephemeral in nature, as long as they cleaned up correctly they shouldn't impact storage too much.
If for some reason Kyverno fails to cleanup those reports fast enough it can become a severe issue though.
Of course not all resources will have background scan reports, some policies can have `background` disabled but you get the picture.
One thing to note here is that it doesn't vary with the number of policies in the cluster. The number of policies will have an impact on the report size though, but not on the number of reports.
Still, admission and background scan reports are usually small sized because they are for a single resource.
### Policy reports and cluster policy reports
For policy reports it's different, the number of policy reports will depend on the number of policies and the number of namespaces:
- one report per policy and per namespace for namespaced resources (10 policies and 20 namespaces -> 200 reports)
- one report per policy for clustered resources (10 policies -> 10 reports)
Policy reports and cluster policy reports are usually much larger, depending on the number of resources in the cluster/namespace.
Sometimes they are too big to be stored in etcd and we split them into multiple reports containing a configurable number entries (`1000` by default).
Quick note about reports deletion, we use the builtin Kubernetes garbage collection mechanism for that. Reports are owned by the resource they apply to and when the resource goes away, reports are garbage collected and deleted automatically.
Sizing is not an easy task and will depend on the number of policy/rules and resources in a cluster.
The size of individual reports will also vary depending on the rule message, the longer the message, the more disk space will be consumed.
The best approach is to measure the average size of an individual report and multiply this size by three times the number of resources for a given type.
```
If a report for a Pod is 1KB and there's 100 pods.
Disk budget = 100 * 1KB * 3
```
See below for an example based on PSS policies.
## PSS policies example
When deploying PSS policies (with `restricted` profile), it creates:
With autogen you have to account for `Deployment`, `StatefulSet`, `Job`, `CronJob`, `ReplicaSet` too (reports for them will be approximately the same size).
And of course ephemeral admission reports consume space too, even if they should be cleaned up quickly.
Finally you need to take policy reports into account too.
Basically they should consume approximately the same disk space as background scan reports (approximately because if some policies don't run in the background, results will come from aggregated admission reports).
The number of policy reports is usually inferior to the number of other reports but they are bigger in size.
In the end, to use PSS policies (with `restricted` profile) on a 1000 pods cluster, you should budget around 25MB of etcd storage, spread across ~2000 reports.
In comparison, an average `Pod` is around 5KB (at least for a pod with a single container). The storage for 1000 pods is around 5MB. We can see that reports has a significant storage impact. Fortunately ETCD stores data on disk and associated cost should be negligible.
