kube-arangodb/docs/design/acceptance_test.md

Acceptance test for kube-arangodb operator on specific Kubernetes platform

This acceptance test plan describes all test scenario's that must be executed successfully in order to consider the kube-arangodb operator production ready on a specific Kubernetes setup (from now on we'll call a Kubernetes setup a platform).

Platform parameters

Before the test, record the following parameters for the platform the test is executed on.

• Name of the platform
• Version of the platform
• Upstream Kubernetes version used by the platform (run kubectl version)
• Number of nodes used by the Kubernetes cluster (run kubectl get node)
• StorageClasses provided by the platform (run kubectl get storageclass)
• Does the platform use RBAC? (run kubectl describe clusterrolebinding)
• Does the platform support services of type LoadBalancer?

If one of the above questions can have multiple answers (e.g. different Kubernetes versions) then make the platform more specific. E.g. consider "GKE with Kubernetes 1.10.2" a platform instead of "GKE" which can have version "1.8", "1.9" & "1.10.2".

Platform preparations

Before the tests can be run, the platform has to be prepared.

Deploy the ArangoDB operators

Deploy the following ArangoDB operators:

• ArangoDeployment operator
• ArangoDeploymentReplication operator
• ArangoLocalStorage operator

To do so, follow the instructions in the documentation.

PersistentVolume provider

If the platform does not provide a PersistentVolume provider, create one by running:

kubectl apply -f examples/arango-local-storage.yaml

Basis tests

The basis tests are executed on every platform with various images:

Run the following tests with the following images:

• Community
• Enterprise

For every tests, one of these images can be chosen, as long as each image is used in a test at least once.

Test 1a: Create single server deployment

Create an ArangoDeployment of mode Single.

Hint: Use tests/acceptance/single.yaml.

• The deployment must start
• The deployment must yield 1 Pod
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 1b: Create active failover deployment

Create an ArangoDeployment of mode ActiveFailover.

Hint: Use tests/acceptance/activefailover.yaml.

• The deployment must start
• The deployment must yield 5 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 1c: Create cluster deployment

Create an ArangoDeployment of mode Cluster.

Hint: Use tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 1d: Create cluster deployment with dc2dc

This test requires the use of the enterprise image.

Create an ArangoDeployment of mode Cluster and dc2dc enabled.

Hint: Derive from tests/acceptance/cluster-sync.yaml.

• The deployment must start
• The deployment must yield 15 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The deployment must yield a Service named <deployment-name>-sync
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 2a: Scale an active failover deployment

Create an ArangoDeployment of mode ActiveFailover.

• The deployment must start
• The deployment must yield 5 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Change the value of spec.single.count from 2 to 3.

• A single server is added
• The deployment must yield 6 Pods

Change the value of spec.single.count from 3 to 2.

• A single server is removed
• The deployment must yield 5 Pods

Test 2b: Scale a cluster deployment

Create an ArangoDeployment of mode Cluster.

Hint: Use tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Change the value of spec.dbservers.count from 3 to 5.

• Two dbservers are added
• The deployment must yield 11 Pods

Change the value of spec.coordinators.count from 3 to 4.

• A coordinator is added
• The deployment must yield 12 Pods

Change the value of spec.dbservers.count from 5 to 2.

• Three dbservers are removed (one by one)
• The deployment must yield 9 Pods

Change the value of spec.coordinators.count from 4 to 1.

• Three coordinators are removed (one by one)
• The deployment must yield 6 Pods

Test 3: Production environment

Production environment tests are only relevant if there are enough nodes available that Pods can be scheduled on.

The number of available nodes must be >= the maximum server count in any group.

Test 3a: Create single server deployment in production environment

Create an ArangoDeployment of mode Single with an environment of Production.

Hint: Derive from tests/acceptance/single.yaml.

• The deployment must start
• The deployment must yield 1 Pod
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 3b: Create active failover deployment in production environment

Create an ArangoDeployment of mode ActiveFailover with an environment of Production.

Hint: Derive from tests/acceptance/activefailover.yaml.

• The deployment must start
• The deployment must yield 5 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 3c: Create cluster deployment in production environment

Create an ArangoDeployment of mode Cluster with an environment of Production.

Hint: Derive from tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 3d: Create cluster deployment in production environment and scale it

Create an ArangoDeployment of mode Cluster with an environment of Production.

Hint: Derive from tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Change the value of spec.dbservers.count from 3 to 4.

• Two dbservers are added
• The deployment must yield 10 Pods

Change the value of spec.coordinators.count from 3 to 4.

• A coordinator is added
• The deployment must yield 11 Pods

Change the value of spec.dbservers.count from 4 to 2.

• Three dbservers are removed (one by one)
• The deployment must yield 9 Pods

Change the value of spec.coordinators.count from 4 to 2.

• Three coordinators are removed (one by one)
• The deployment must yield 7 Pods

Test 4a: Create cluster deployment with ArangoLocalStorage provided volumes

Ensure an ArangoLocalStorage is deployed.

Hint: Use from tests/acceptance/local-storage.yaml.

Create an ArangoDeployment of mode Cluster with a StorageClass that is mapped to an ArangoLocalStorage provider.

Hint: Derive from tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 4b: Create cluster deployment with a platform provided StorageClass

This test only applies to platforms that provide their own StorageClasses.

Create an ArangoDeployment of mode Cluster with a StorageClass that is provided by the platform.

Hint: Derive from tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Test 5a: Test Pod resilience on single servers

Create an ArangoDeployment of mode Single.

Hint: Use from tests/acceptance/single.yaml.

• The deployment must start
• The deployment must yield 1 Pod
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Delete the Pod of the deployment that contains the single server.

• The Pod must be restarted
• After the Pod has restarted, the server must have the same data and be responsive again

Test 5b: Test Pod resilience on active failover

Create an ArangoDeployment of mode ActiveFailover.

Hint: Use from tests/acceptance/activefailover.yaml.

• The deployment must start
• The deployment must yield 5 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Delete a Pod of the deployment that contains an agent.

• While the Pod is gone & restarted, the cluster must still respond to requests (R/W)
• The Pod must be restarted

Delete a Pod of the deployment that contains a single server.

• While the Pod is gone & restarted, the cluster must still respond to requests (R/W)
• The Pod must be restarted

Test 5c: Test Pod resilience on clusters

Create an ArangoDeployment of mode Cluster.

Hint: Use from tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Delete a Pod of the deployment that contains an agent.

• While the Pod is gone & restarted, the cluster must still respond to requests (R/W)
• The Pod must be restarted

Delete a Pod of the deployment that contains a dbserver.

• While the Pod is gone & restarted, the cluster must still respond to requests (R/W), except for requests to collections with a replication factor of 1.
• The Pod must be restarted

Delete a Pod of the deployment that contains an coordinator.

• While the Pod is gone & restarted, the cluster must still respond to requests (R/W), except requests targeting the restarting coordinator.
• The Pod must be restarted

Test 6a: Test Node reboot on single servers

Create an ArangoDeployment of mode Single.

Hint: Use from tests/acceptance/single.yaml.

• The deployment must start
• The deployment must yield 1 Pod
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Reboot the Node of the deployment that contains the single server.

• The Pod running on the Node must be restarted
• After the Pod has restarted, the server must have the same data and be responsive again

Test 6b: Test Node reboot on active failover

Create an ArangoDeployment of mode ActiveFailover with an environment of Production.

Hint: Use from tests/acceptance/activefailover.yaml.

• The deployment must start
• The deployment must yield 5 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Reboot a Node.

• While the Node is restarting, the cluster must still respond to requests (R/W)
• All Pods on the Node must be restarted

Test 6c: Test Node reboot on clusters

Create an ArangoDeployment of mode Cluster with an environment of Production.

Hint: Use from tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Reboot a Node.

• While the Node is restarting, the cluster must still respond to requests (R/W)
• All Pods on the Node must be restarted

Test 6d: Test Node removal on single servers

This test is only valid when StorageClass is used that provides network attached PersistentVolumes.

Create an ArangoDeployment of mode Single.

Hint: Use from tests/acceptance/single.yaml.

• The deployment must start
• The deployment must yield 1 Pod
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Remove the Node containing the deployment from the Kubernetes cluster.

• The Pod running on the Node must be restarted on another Node
• After the Pod has restarted, the server must have the same data and be responsive again

Test 6e: Test Node removal on active failover

Create an ArangoDeployment of mode ActiveFailover with an environment of Production.

Hint: Use from tests/acceptance/activefailover.yaml.

• The deployment must start
• The deployment must yield 5 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Remove a Node containing the Pods of the deployment from the Kubernetes cluster.

• While the Pods are being restarted on new Nodes, the cluster must still respond to requests (R/W)
• The Pods running on the Node must be restarted on another Node
• After the Pods have restarted, the server must have the same data and be responsive again

Test 6f: Test Node removal on clusters

This test is only valid when:

• A StorageClass is used that provides network attached PersistentVolumes
• or all collections have a replication factor of 2 or higher

Create an ArangoDeployment of mode Cluster with an environment of Production.

Hint: Use from tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Remove a Node containing the Pods of the deployment from the Kubernetes cluster.

• While the Pods are being restarted on new Nodes, the cluster must still respond to requests (R/W)
• The Pods running on the Node must be restarted on another Node
• After the Pods have restarted, the server must have the same data and be responsive again

Test 6g: Test Node removal on clusters with replication factor 1

This test is only valid when:

• A StorageClass is used that provides Node local PersistentVolumes
• and at least some collections have a replication factor of 1

Create an ArangoDeployment of mode Cluster with an environment of Production.

Hint: Use from tests/acceptance/cluster.yaml.

• The deployment must start
• The deployment must yield 9 Pods
• The deployment must yield a Service named <deployment-name>
• The deployment must yield a Service named <deployment-name>-ea
• The Service named <deployment-name>-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Remove a Node, containing the dbserver Pod that holds a collection with replication factor 1, from the Kubernetes cluster.

• While the Pods are being restarted on new Nodes, the cluster must still respond to requests (R/W), except requests involving collections with a replication factor of 1
• The Pod running the dbserver with a collection that has a replication factor of 1 must NOT be restarted on another Node

Remove the collections with the replication factor of 1

• The remaining Pods running on the Node must be restarted on another Node
• After the Pods have restarted, the server must have the same data, except for the removed collections, and be responsive again

Test 7a: Test DC2DC on 2 clusters, running in the same Kubernetes cluster

This test requires the use of the enterprise image.

Create 2 ArangoDeployment of mode Cluster and dc2dc enabled.

Hint: Derive from tests/acceptance/cluster-sync.yaml, name the deployments cluster1 and cluster2.

Make sure to include a name ('cluster1-to-2`) for an external access package.

apiVersion: "database.arangodb.com/v1alpha"
kind: "ArangoDeployment"
metadata:
  name: "cluster1"
spec:
  mode: Cluster
  image: ewoutp/arangodb:3.3.14
  sync:
    enabled: true
    externalAccess:
      accessPackageSecretNames: ["cluster1-to-2"]

• The deployments must start
• The deployments must yield 15 Pods
• The deployments must yield a Service named cluster[1|2]
• The deployments must yield a Service named cluster[1|2]-ea
• The deployments must yield a Service named cluster[1|2]-sync
• The Services named cluster[1|2]-ea must be accessible from outside (LoadBalancer or NodePort) and show WebUI

Create an ArangoDeploymentReplication from tests/acceptance/cluster12-replication.yaml.

It will take some time until the synchronization (from cluster1 to cluster2) is configured.

• The status of the cluster12-replication resource shows ....
• The webUI of cluster1 shows that you can create a new collection there.
• The webUI of cluster2 shows that you cannot create a new collection there.

Create a collection named testcol with a replication factor 2 and 3 shards (using the webUI of cluster1).

• The webUI of cluster2 shows collection testcol with the given replication factor and number of shards.

Create multiple documents in the collection named testcol (using the webUI of cluster1).

• The documents are visible in webUI of cluster2.

Modify multiple documents in the collection named testcol (using the webUI of cluster1).

• The modified documents are visible in webUI of cluster2.

Remove one or more documents from the collection named testcol (using the webUI of cluster1).

• The documents are no longer visible in webUI of cluster2.

Create a new database called db2 (using the webUI of cluster1).

• The webUI of cluster2 shows database db2.