Add testing guidelines (#5903)

* Add testing guidelines

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Add separate Makefile targets for e2e-tests

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Apply suggestions from code review

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

Co-authored-by: Simon Pasquier <spasquie@redhat.com>

---------

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
Co-authored-by: Simon Pasquier <spasquie@redhat.com>

.github/PULL_REQUEST_TEMPLATE.md

@@ -15,6 +15,10 @@ _What type of changes does your code introduce to the Prometheus operator? Put a
 - [ ] `ENHANCEMENT` (non-breaking change which improves existing functionality)
 - [ ] `NONE` (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)
 
+## Verification
+<!-- How you tested it? How do you know it works? -->
+Please check the [Prometheus-Operator testing guidelines](../TESTING.md) for recommendations about automated tests.
+
 ## Changelog entry
 
 _Please put a one-line changelog entry below. This will be copied to the changelog file during the release process._

CONTRIBUTING.md

@@ -60,8 +60,7 @@ This is a rough outline of what a contributor's workflow looks like:
 - Make commits of logical units.
 - Make sure your commit messages are in the proper format (see below).
 - Push your changes to a topic branch in your fork of the repository.
-- Make sure the tests pass, and add any new tests as appropriate.
-- If the tests are checking long strings such as YAML, JSON or any other complex content, ensure you're using [golden files](https://pkg.go.dev/gotest.tools/v3/golden).
+- Make sure the tests pass, and add any new tests as appropriate. ([Testing guidelines](TESTING.md))
 - Submit a pull request to the original repository.
 
 Many files (documentation, manifests, ...) in this repository are auto-generated. For instance, `bundle.yaml` is generated from the *Jsonnet* files in `/jsonnet/prometheus-operator`. Before submitting a pull request, make sure that you've executed `make generate` and committed the generated changes.

Makefile

@@ -362,6 +362,30 @@ test-e2e: KUBECONFIG?=$(HOME)/.kube/config
 test-e2e: test/instrumented-sample-app/certs/cert.pem test/instrumented-sample-app/certs/key.pem
 	go test -timeout 120m -v ./test/e2e/ $(TEST_RUN_ARGS) --kubeconfig=$(KUBECONFIG) --operator-image=$(IMAGE_OPERATOR):$(TAG) -count=1
 
+.PHONY: test-e2e-alertmanager
+test-e2e-alertmanager:
+	EXCLUDE_PROMETHEUS_TESTS=exclude EXCLUDE_PROMETHEUS_ALL_NS_TESTS=exclude EXCLUDE_THANOSRULER_TESTS=exclude EXCLUDE_OPERATOR_UPGRADE_TESTS=exclude FEATURE_GATED_TESTS=exclude EXCLUDE_PROMETHEUS_UPGRADE_TESTS=exclude $(MAKE) test-e2e
+
+.PHONY: test-e2e-prometheus
+test-e2e-alertmanager:
+	EXCLUDE_ALERTMANAGER_TESTS=exclude EXCLUDE_PROMETHEUS_ALL_NS_TESTS=exclude EXCLUDE_THANOSRULER_TESTS=exclude EXCLUDE_OPERATOR_UPGRADE_TESTS=exclude FEATURE_GATED_TESTS=exclude EXCLUDE_PROMETHEUS_UPGRADE_TESTS=exclude $(MAKE) test-e2e
+
+.PHONY: test-e2e-prometheus-all-namespaces
+test-e2e-alertmanager:
+	EXCLUDE_ALERTMANAGER_TESTS=exclude EXCLUDE_PROMETHEUS_TESTS=exclude EXCLUDE_THANOSRULER_TESTS=exclude EXCLUDE_OPERATOR_UPGRADE_TESTS=exclude FEATURE_GATED_TESTS=exclude EXCLUDE_PROMETHEUS_UPGRADE_TESTS=exclude $(MAKE) test-e2e
+
+.PHONY: test-e2e-thanos-ruler
+test-e2e-alertmanager:
+	EXCLUDE_ALERTMANAGER_TESTS=exclude EXCLUDE_PROMETHEUS_TESTS=exclude EXCLUDE_PROMETHEUS_ALL_NS_TESTS=exclude EXCLUDE_OPERATOR_UPGRADE_TESTS=exclude FEATURE_GATED_TESTS=exclude EXCLUDE_PROMETHEUS_UPGRADE_TESTS=exclude $(MAKE) test-e2e
+
+.PHONY: test-e2e-operator-upgrade
+test-e2e-alertmanager:
+	EXCLUDE_ALERTMANAGER_TESTS=exclude EXCLUDE_PROMETHEUS_TESTS=exclude EXCLUDE_PROMETHEUS_ALL_NS_TESTS=exclude EXCLUDE_THANOSRULER_TESTS=exclude FEATURE_GATED_TESTS=exclude EXCLUDE_PROMETHEUS_UPGRADE_TESTS=exclude $(MAKE) test-e2e
+
+.PHONY: test-e2e-prometheus-upgrade
+test-e2e-alertmanager:
+	EXCLUDE_ALERTMANAGER_TESTS=exclude EXCLUDE_PROMETHEUS_TESTS=exclude EXCLUDE_PROMETHEUS_ALL_NS_TESTS=exclude EXCLUDE_THANOSRULER_TESTS=exclude FEATURE_GATED_TESTS=exclude EXCLUDE_OPERATOR_UPGRADE_TESTS=exclude $(MAKE) test-e2e
+
 ############
 # Binaries #
 ############

README.md

@@ -161,59 +161,9 @@ kubectl delete --ignore-not-found customresourcedefinitions \
   prometheusrules.monitoring.coreos.com
 ```
 
-## Development
+## Testing
 
-### Prerequisites
-
-- golang environment
-- docker (used for creating container images, etc.)
-- kind (optional)
-
-### Testing
-
-> Ensure that you're running tests in the following path:
-> `$GOPATH/src/github.com/prometheus-operator/prometheus-operator` as tests expect paths to
-> match. If you're working from a fork, just add the forked repo as a remote and
-> pull against your local prometheus-operator checkout before running tests.
-
-#### Running *unit tests*:
-
-`make test-unit`
-
-#### Running *end-to-end* tests on local kind cluster:
-
-1. `kind create cluster --image=kindest/node:<latest>`. e.g `v1.23.0` version.
-
-> Note: In case you are running kind on MacOS using podman, it is recommended to create podman machine 4 CPUs and 8GiB memory.
-> Less resources might cause end to end tests fail because of lack of resources for cluster.
->
-> `podman machine init --cpus=4 --memory=8192 --rootful --now`
-
-2. `kubectl cluster-info --context kind-kind`. kind version >= 0.6.x
-3. `make image` - build Prometheus Operator docker image locally.
-
-> Note: In case you are running kind using podman, the step 3 won't work for you. You will need to switch command in Makefile:
->
-> `CONTAINER_CLI=podman make image`
-
-4. publish locally built images to be accessible inside kind
-
-   ```bash
-   for n in "prometheus-operator" "prometheus-config-reloader" "admission-webhook"; do kind load docker-image "quay.io/prometheus-operator/$n:$(git rev-parse --short HEAD)"; done;
-   ```
-
-> Note: In case you are running kind using podman, docker-image command won't work. You need to use image archives instead:
->
-> `for n in "prometheus-operator" "prometheus-config-reloader" "admission-webhook"; do podman save --quiet -o images/$n.tar "quay.io/prometheus-operator/$n:$(git rev-parse --short HEAD)"; kind load image-archive images/$n.tar; done`
-
-5. `make test-e2e`
-
-#### Running *end-to-end* tests on local minikube cluster:
-
-1. `minikube start --kubernetes-version=stable --memory=4096 --extra-config=apiserver.authorization-mode=NODE,RBAC`
-2. `eval $(minikube docker-env) && make image` - build Prometheus Operator
-   docker image on minikube's docker
-3. `make test-e2e`
+See [TESTING](TESTING.md)
 
 ## Contributing
 

TESTING.md

@@ -0,0 +1,116 @@
+When contributing code to Prometheus-Operator, you'll notice that every Pull Request will run against an extensive test suite. Among an extensive list of benefits that tests brings to the Project's overall health and reliability, it can be the reviewer's and contributors's best friend during development:
+
+* Test cases serve as documentation, providing insights into the expected behavior of the software.
+* Testing can prevent regressions by verifying that new changes don't break existing functionality.
+* Running tests locally accelerate the feedback loop, removing the dependency that contributors might have on CI when working on a Pull Request.
+
+This document will focus on teaching you about the different test suites that we currently have and how to run different scenarios to help your development experience!
+
+# Test categories
+
+## Unit tests
+
+Unit tests are used to test particular code snippets in isolation. They are your best ally when looking for quick feedback loops in a particular function.
+
+Imagine you're working on a PR that adds a new field to the ScrapeConfig CRD and you want to test if your change is reflected to the configmap. Instead of creating a full Kubernetes cluster, installing all the CRDs, running the Prometheus-Operator, deploying a Prometheus resource with ScrapeConfigSelectors and finally check if your change made it to the live object, you could simply write or extend a unit test for the configmap generation.
+
+Here is an example test that checks if the string generated from ScrapeConfigs are equal to an expected file.
+
+https://github.com/prometheus-operator/prometheus-operator/blob/cb534214415beb3c39353988ae85f2bb07f245e7/pkg/prometheus/promcfg_test.go#L4945-L4956
+
+Unit tests can be run with:
+
+```bash
+make test-unit
+```
+
+They can also be run for particular packages:
+
+```
+go test ./pkg/prometheus/server
+```
+
+Or even particular functions:
+
+```
+go test -run ^TestPodLabelsAnnotations$ ./pkg/prometheus/server
+```
+
+### Testing multiline string comparison - Golden files
+
+Golden files are plaintext documents designed to facilitate the validation of lengthy strings. They come in handy when, for instance, you need to test a Prometheus configuration that's generated using Go structures. You can marshal this configuration into YAML and then compare it against a static reference to ensure a match. Golden files offer an elegant solution to this challenge, sparing you the need to hard-code the static configuration directly into your test code.
+
+In the example below, we're generating the Prometheus configuration (which can easily have 100+ lines for each individual test) and comparing it against a golden file:
+
+https://github.com/prometheus-operator/prometheus-operator/blob/aeceb0b4fadc8307a44dc55afdceca0bea50bbb0/pkg/prometheus/promcfg_test.go#L102-L277
+
+If not for golden files, the test above, instead of ~150 lines, would easily require around ~1000 lines. The usage of golden files help us maintain test suites with several multiline strings comparison without sacrifing test readability.
+
+### Updating Golden Files
+
+There are contributions, e.g. adding a new required field to an existing configuration, that require to update several golden files at once. This can easily be done with the command below:
+
+```
+make test-unit-update-golden
+```
+
+## End-to-end tests
+
+Sometimes, running tests in isolation is not enough and we really want test the behavior of Prometheus-Operator when running in a working Kubernetes cluster. For those occasions, end-to-end tests are our choice.
+
+To run e2e-tests locally, first start a Kubernetes cluster. We recommend [KinD](https://kind.sigs.k8s.io/) because it is lightweight (it can run on small notebooks) and this is what the project's CI uses. [MiniKube](https://minikube.sigs.k8s.io/docs/start/) is also another option.
+
+For manual testing, you can use the utility script [scripts/run-external.sh](scripts/run-external.sh), it will check all the requirements and run your local version of the Prometheus Operator on your Kind cluster:
+
+```
+./scripts/run-external.sh -c
+```
+
+For automated end-to-end tests, we have the command:
+
+```
+make test-e2e
+```
+
+`make test-e2e` will run the complete end-to-end test suite. Those are the same tests we run in Pull Requests pipelines and it will make sure all features requirements amongst ***all*** controllers are working.
+
+When working on a contribution though, it's rare that you'll need to make a change that impacts all controllers at once. Running the complete test suite takes a ***long time***, so you might want to run only the tests that are relevant to your change while developing it.
+
+### Skipping test suites
+
+https://github.com/prometheus-operator/prometheus-operator/blob/272df8a2411bcf877107b3251e79ae8aa8c24761/test/e2e/main_test.go#L46-L50
+
+As shown above, particular test suites can be skipped with Environment Variables. You can also look at our [CI pipeline as example](https://github.com/prometheus-operator/prometheus-operator/blob/272df8a2411bcf877107b3251e79ae8aa8c24761/.github/workflows/e2e.yaml#L85-L94). Altough we always run all tests in CI, skipping irrelevant tests are great during development as they shorten the feedback loop.
+
+The following Makefile targets can run specific end-to-end tests:
+
+* `make test-e2e-alertmanager` - Will run Alertmanager tests.
+* `make test-e2e-thanos-ruler` - Will run Thanos-Ruler tests.
+* `make test-e2e-prometheus` - Will run Prometheus tests with limited namespace permissions.
+* `make test-e2e-prometheus-all-namespaces` - Will run regular Prometheus tests.
+* `make test-e2e-operator-upgrade` - Will validate that a monitoring stack managed by the previous version of Prometheus-Operator will continue to work after an upgrade to the current version.
+* `make test-e2e-prometheus-upgrade` - Will validate that a series of Prometheus versions can be sequentially upgraded.
+
+### Running just a particular end-to-end test
+
+A few test suites can easily take more than an hour even when running in powerful notebooks. If you're debugging a particular test, it might be advantageous for you to comment code just to accelerate your tests.
+
+```patch
+// TestDenylist tests the Prometheus Operator configured not to watch specific namespaces.
+func TestDenylist(t *testing.T) {
+	skipPrometheusTests(t)
+	testFuncs := map[string]func(t *testing.T){
++		// "Prometheus":     testDenyPrometheus,
++		// "ServiceMonitor": testDenyServiceMonitor,
+-		"Prometheus":     testDenyPrometheus,
+-		"ServiceMonitor": testDenyServiceMonitor,
+		"ThanosRuler":    testDenyThanosRuler,
+	}
+
+	for name, f := range testFuncs {
+		t.Run(name, f)
+	}
+}
+```
+
+In the example above we're commenting 2 tests, in combination with Environment Variables to skip other test suites, to make sure we focus on what really matters to us at the moment. Just don't forget to remove the comments once you're done!!