kube-arangodb/docs/design/recovery.md

Manual Recovery

Overview

Let's consider a situation where we had a ArangoDeployment in Cluster mode (3 DbServers, 3 Coordinators, 3 Agents) with Local storage attached (only one K8s Node in the K8s cluster).

Due to some reason the ArangoDeployment was deleted (e.g. ETCD storage has been wiped out) and we want to recover it. Fortunately, we have a backup of the data on the disk.

To recover the ArangoDeployment we need to:

1. Create PV and PVC for each member with persistent storage (agent, dbservers, single)
2. Create a new ArangoDeployment with the same members IDs

Local storage data

We have a members (Agents & DbServers) data in the following directories:

> ls -1 /var/data/
f9rs2htwc9e0bzme
fepwdnnbf0keylgx
gqnkahucthoaityt
vka6ic19qcl1y3ec
rhlf8vixbsbewefo
rlzl467vfgsdpofu

To find out the name of the members to which data should be attached, we need to check the UUID file content in each directory:

> cat /var/data/*/UUID
AGNT-pntg5yc8
AGNT-kfyuj8ow
AGNT-bv5rofcz
PRMR-9xztmg4t
PRMR-l1pp19yl
PRMR-31akmzrp

Initial ArangoDeployment

Here is an example of the initial ArangoDeployment before deletion:

cat <<EOF | kubectl apply -f -
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  externalAccess:
    type: NodePort
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        storageClassName: my-local-ssd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  dbservers:
    volumeClaimTemplate:
      spec:
        storageClassName: my-local-ssd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
EOF

Create PV and PVC

1. We need to create ArangoLocalStorage first:

   cat <<EOF | kubectl apply -f -
   apiVersion: "storage.arangodb.com/v1alpha"
   kind: "ArangoLocalStorage"
   metadata:
     name: "local-storage"
   spec:
     storageClass:
       name: my-local-ssd
       isDefault: true
     localPath:
     - /mnt/data
   EOF
   

2. Now create PV and PVC for every directory listed above
   • Agents - here is an example for AGNT-pntg5yc8(f9rs2htwc9e0bzme directory)
     • PV

       cat <<EOF | kubectl apply -f -
       apiVersion: "v1"
       kind: PersistentVolume
       metadata:
         labels:
           arango_deployment: cluster
           role: agent
         name: agent-pntg5yc8-f9rs2htwc9e0bzme
       spec:
         accessModes:
         - ReadWriteOnce
         capacity:
           storage: 1Gi
         local:
           path: /mnt/data/f9rs2htwc9e0bzme
         persistentVolumeReclaimPolicy: Retain
         storageClassName: my-local-ssd
         volumeMode: Filesystem
         nodeAffinity:
           required:
             nodeSelectorTerms:
             - matchExpressions:
               - key: kubernetes.io/hostname
                 operator: In
                 values:
                 - minikube
       EOF
       

     • PVC

       cat <<EOF | kubectl apply -f -
       apiVersion: v1
       kind: PersistentVolumeClaim
       metadata:
         labels:
           app: arangodb
           arango_deployment: cluster
           role: agent
         name: agent-pntg5yc8
       spec:
         accessModes:
         - ReadWriteOnce
         resources:
           requests:
             storage: 1Gi
         storageClassName: my-local-ssd
         volumeMode: Filesystem
         volumeName: agent-pntg5yc8-f9rs2htwc9e0bzme
       EOF
       

   • DbServers - here is an example for PRMR-9xztmg4t (vka6ic19qcl1y3ec directory)
     • PV

       cat <<EOF | kubectl apply -f -
       apiVersion: "v1"
       kind: PersistentVolume
       metadata:
         labels:
           arango_deployment: cluster
           role: dbserver
         name: dbserver-9xztmg4t-vka6ic19qcl1y3ec
       spec:
         accessModes:
         - ReadWriteOnce
         capacity:
           storage: 1Gi
         local:
           path: /mnt/data/vka6ic19qcl1y3ec
         persistentVolumeReclaimPolicy: Retain
         storageClassName: my-local-ssd
         volumeMode: Filesystem
         nodeAffinity:
           required:
             nodeSelectorTerms:
             - matchExpressions:
               - key: kubernetes.io/hostname
                 operator: In
                 values:
                 - minikube
       EOF
       

     • PVC

       cat <<EOF | kubectl apply -f -
       apiVersion: v1
       kind: PersistentVolumeClaim
       metadata:
         labels:
           app: arangodb
           arango_deployment: cluster
           role: dbserver
         name: dbserver-9xztmg4t
       spec:
         accessModes:
         - ReadWriteOnce
         resources:
           requests:
             storage: 1Gi
         storageClassName: my-local-ssd
         volumeMode: Filesystem
         volumeName: dbserver-9xztmg4t-vka6ic19qcl1y3ec
       EOF
       

Create ArangoDeployment with previously created PVC

Now we can create ArangoDeployment with previously created PVCs:

cat <<EOF | kubectl apply -f -
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  externalAccess:
    type: NodePort
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        storageClassName: my-local-ssd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  dbservers:
    volumeClaimTemplate:
      spec:
        storageClassName: my-local-ssd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
status:
  agency:
    ids:
      - AGNT-pntg5yc8
      - AGNT-kfyuj8ow
      - AGNT-bv5rofcz
    size: 3
  members:
    agents:
      - id: AGNT-pntg5yc8
        persistentVolumeClaim:
          name: agent-pntg5yc8
        persistentVolumeClaimName: agent-pntg5yc8
      - id: AGNT-kfyuj8ow
        persistentVolumeClaim:
          name: agent-kfyuj8ow
        persistentVolumeClaimName: agent-kfyuj8ow
      - id: AGNT-bv5rofcz
        persistentVolumeClaim:
          name: agent-bv5rofcz
        persistentVolumeClaimName: agent-bv5rofcz
    dbservers:
      - id: PRMR-9xztmg4t
        persistentVolumeClaim:
          name: cluster-dbserver-9xztmg4t
        persistentVolumeClaimName: cluster-dbserver-9xztmg4t
      - id: PRMR-l1pp19yl
        persistentVolumeClaim:
          name: cluster-dbserver-l1pp19yl
        persistentVolumeClaimName: cluster-dbserver-l1pp19yl
      - id: PRMR-31akmzrp
        persistentVolumeClaim:
          name: cluster-dbserver-31akmzrp
        persistentVolumeClaimName: cluster-dbserver-31akmzrp
EOF

That's it! Now you can use ArangoDB with restored data.