Another day in hell, MicroK8s is not running, and almost all pods are in Unknown or Terminating state. Trying to stop/restart microk8s and also the whole server but still the end, microk8s is still faling down. Any rescure is no luck.

This runbook is about step by step to performing the rescure the whole Microk8s.

• Backing up and restoring manifests
• PVC data recovery
• Cluster node recovery steps
• Restarting core services (etcd, containerd, etc.)
• Troubleshooting tips (e.g., nodes not ready, pods stuck)

Let’s started!

Clean Restart Everything

sudo microk8s stop
sleep 3
sudo microk8s start

Wait 10–20 seconds, then:

microk8s status --wait-ready

It may hang so long. Try to Hard Reset Snap Services:

sudo systemctl restart snap.microk8s.*

Then:

microk8s status --wait-ready

If luckily, the microk8s is started. Another world, It may hand so long.

Inspect the problem

Inspecting the problem:

microk8s inspect

showing services are up, but on one line:

cp: cannot stat ‘/var/snap/microk8s/7971/var/kubernetes/backend/localnode.yaml’: No such file or directory

Try to check if embedded datastore (Dqlite) is damaged or missing:

ls -la /var/snap/microk8s/current/var/kubernetes/backend/

If it’s empty or missing localnode.yaml, the cluster DB is corrupted.

We have to backup full Microk8s and the Statefull Set pod like Database, PVC in sections below.

Reinitialize Cluster

Using the command to reset Microk8s, then reinstall everythings.

sudo microk8s reset

I prefer reinstall the whole VPS and install Microk8s again. But before performing the full install VPS, let\’s backups all things.

Backing up

Backup full deployments/services:

microk8s kubectl get all --all-namespaces -o yaml > /microk8s-backup-config.yaml

or all the pvc,pv,configmap,secret,storageclass:

microk8s kubectl get pvc,pv,configmap,secret,storageclass --all-namespaces -o yaml > /microk8s-backup-config.yaml

Locate the PV path on disk:

sudo ls -lah /var/snap/microk8s/common/default-storage/

Identify which PV belongs to your DB pod:

microk8s kubectl get pvc -A

=> Ready to Backup Databases from StatefulSets when MicroK8s is down:

sudo tar -czvf /mariadb-data-backup.tar.gz /var/snap/microk8s/common/default-storage/

Once fixing MicroK8s or rebuild it, we can:

• Redeploy the StatefulSet
• Replace the new PVC contents with your backup files
• Restart the pod — and your DB should come back up

Restore From yaml in MicroK8s

Make sure MicroK8s is running and ready before restoring.

sudo microk8s start
microk8s status --wait-ready

Apply the backup YAML

microk8s kubectl apply -f /microk8s-backup-config.yaml

Restore any Persistent Volumes (PV/PVC) manually

Restore the data by extracting the archive back to the exact same path, assuming the original PVCs are recreated and their names (UUIDs) match.

If the name of PVCs are difference from the backup, then, we have to manually rename the folder in /var/snap/microk8s/common/default-storage/ to match new PVC names.

Stop MicroK8s (to avoid file lock/corruption):

sudo microk8s stop

Extract the backup:

sudo tar -xzvf ~/mariadb-data-backup.tar.gz -C /

This will restore everything to /var/snap/microk8s/common/default-storage/

If needed, rename the backed PVC to the new name which have just created:

sudo mv /var/snap/microk8s/common/default-storage/pvc-OLD-ID /var/snap/microk8s/common/default-storage/pvc-NEW-ID

Starting back the Microk8s and watching the DB pods:

sudo microk8s start
microk8s status --wait-ready
microk8s kubectl get pods -A -w

All the service now should be started up.

kubectl get nodes
watch -n 1 microk8s kubectl get all --all-namespaces

Hope this help.
Nam Le,

devops microk8s