6 KiB
Reports
This document contains scripts to help troubleshooting reports issues.
Getting reports detailed information
When querying reports, you can add -o wide to get a more detailed output.
This will show information about the resource associated with the report.
It can be useful to determine if a particular resource kind is responsible for creating too many reports.
If APIVERSION, KIND and SUBJECT is empty it means the report is orphan and it is an issue if the report is more than a couple of minutes old.
# list cluster admission reports
kubectl get cadmr -o wide
# list cluster background scan reports
kubectl get cbgscanr -o wide
# list admission reports
kubectl get admr -A -o wide
# list background scan reports
kubectl get bgscanr -A -o wide
Below is an example of the output:
$ kubectl get cadmr -o wide
NAME APIVERSION KIND SUBJECT PASS FAIL WARN ERROR SKIP AGE HASH
06aa537a-e81d-4253-8eb2-cd72f366a000 rbac.authorization.k8s.io/v1 ClusterRole view 0 0 0 0 1 8h 5c71e236747d42e7b2a0cedd3f36434d
076d70b7-64c4-41b0-957e-07122680f930 apiextensions.k8s.io/v1 CustomResourceDefinition generaterequests.kyverno.io 0 0 0 0 1 7h48m b82b99dd89e7ed7ec064d2f96d4b690a
Getting the number of reports in a cluster
This will help checking if reports are incorrectly accumulating in the cluster.
COUNT=$(kubectl get cadmr --no-headers 2> /dev/null | wc -l)
echo "number of cluster admission reports: $COUNT"
COUNT=$(kubectl get cbgscanr --no-headers 2> /dev/null | wc -l)
echo "number of cluster background scan reports: $COUNT"
COUNT=$(kubectl get admr -A --no-headers 2> /dev/null | wc -l)
echo "number of admission reports: $COUNT"
COUNT=$(kubectl get bgscanr -A --no-headers 2> /dev/null | wc -l)
echo "number of background scan reports: $COUNT"
NS_LIST=$(kubectl get ns -o jsonpath='{.items[*].metadata.name}')
for ns in $NS_LIST
do
COUNT=$(kubectl get -n $ns admr --no-headers 2> /dev/null | wc -l)
echo "number of admission reports in $ns: $COUNT"
COUNT=$(kubectl get -n $ns bgscanr --no-headers 2> /dev/null | wc -l)
echo "number of background scan reports in $ns: $COUNT"
done
Getting the number of reports per kind
Use the script below to get number of reports per resource kind in a cluster.
This will help determining if a particular resource kind is responsible for creating too many reports.
API_LIST=$(kubectl api-resources --namespaced=false --no-headers | awk '{print $NF}')
for api in $API_LIST
do
COUNT=$(kubectl get cadmr --no-headers -o jsonpath="{range .items[?(@.metadata.ownerReferences[0].kind=='$api')]}{.metadata.name}{'\n'}{end}" 2> /dev/null | wc -l)
echo "number of cluster admission reports for $api: $COUNT"
COUNT=$(kubectl get cbgscanr --no-headers -o jsonpath="{range .items[?(@.metadata.ownerReferences[0].kind=='$api')]}{.metadata.name}{'\n'}{end}" 2> /dev/null | wc -l)
echo "number of cluster background scan reports for $api: $COUNT"
done
API_LIST=$(kubectl api-resources --namespaced=true --no-headers | awk '{print $NF}')
for api in $API_LIST
do
COUNT=$(kubectl get admr -A --no-headers -o jsonpath="{range .items[?(@.metadata.ownerReferences[0].kind=='$api')]}{.metadata.name}{'\n'}{end}" 2> /dev/null | wc -l)
echo "number of admission reports for $api: $COUNT"
COUNT=$(kubectl get bgscanr --no-headers -o jsonpath="{range .items[?(@.metadata.ownerReferences[0].kind=='$api')]}{.metadata.name}{'\n'}{end}" 2> /dev/null | wc -l)
echo "number of background scan reports for $api: $COUNT"
done
Watching report changes
By using --watch-only with kubectl you can view report changes only without first listing existing reports.
Listing existing reports can take a long time when there is a high number of reports.
With --watch-only you only get an output for reports that are created, updated or deleted.
This is useful to determine if a particular resource kind is responsible for creating too many reports.
# watch changing cluster admission reports
kubectl get cadmr -o wide -w --watch-only
# watch changing cluster background scan reports
kubectl get cbgscanr -o wide -w --watch-only
# watch changing admission reports
kubectl get admr -A -o wide -w --watch-only
# watch changing background scan reports
kubectl get bgscanr -A -o wide -w --watch-only
Getting orphan reports count
Orphan reports can exist in a cluster but should stay pretty low.
Orphan reports will be either adopted or deleted.
A high number of orphan reports indicates that something is not working correctly.
ALL=$(kubectl get cadmr --no-headers | wc -l)
NOT_ORPHANS=$(kubectl get cadmr --no-headers -o jsonpath="{range .items[?(@.metadata.ownerReferences[0].uid)]}{.metadata.name}{'\n'}{end}" 2> /dev/null | wc -l)
echo "number of orphan cluster admission reports: $((ALL-NOT_ORPHANS)) ($ALL - $NOT_ORPHANS)"
ALL=$(kubectl get cadmr --no-headers 2> /dev/null | wc -l)
NOT_ORPHANS=$(kubectl get cadmr --no-headers -o jsonpath="{range .items[?(@.metadata.ownerReferences[0].uid)]}{.metadata.name}{'\n'}{end}" 2> /dev/null | wc -l)
echo "number of orphan cluster background scan reports: $((ALL-NOT_ORPHANS)) ($ALL - $NOT_ORPHANS)"
ALL=$(kubectl get admr -A --no-headers 2> /dev/null | wc -l)
NOT_ORPHANS=$(kubectl get admr -A --no-headers -o jsonpath="{range .items[?(@.metadata.ownerReferences[0].uid)]}{.metadata.name}{'\n'}{end}" 2> /dev/null | wc -l)
echo "number of orphan admission reports: $((ALL-NOT_ORPHANS)) ($ALL - $NOT_ORPHANS)"
ALL=$(kubectl get bgscanr -A --no-headers 2> /dev/null | wc -l)
NOT_ORPHANS=$(kubectl get bgscanr -A --no-headers -o jsonpath="{range .items[?(@.metadata.ownerReferences[0].uid)]}{.metadata.name}{'\n'}{end}" 2> /dev/null | wc -l)
echo "number of orphan background scan reports: $((ALL-NOT_ORPHANS)) ($ALL - $NOT_ORPHANS)"