Sometimes, particularly after a node in the cluster has crashed or been restarted, you might end up with containers that cannot start because their PVC(s) cannot be attached to the node where they are running. The actual problem is that the cannot detach from the node they were previously attached to. If you are running Longhorn, the volume is usually listed as "Detaching" but never transitions from that state (probably something similar with other storage classes).
The culprit is something called VolumeAttachment, which is the entity that keeps track of what PVC is attached to what node. A VolumeAttachment is supposed to be deleted when the last container on a node terminates and there are no more attachments to the PVC, but sometimes that just fails.
Fix 1, lightly stuck
To list the VolumeAttachments in your namespace:
kubectl -n namespace get volumeattachment
This should list your VolumeAttachment:
NAME ATTACHER PV NODE ATTACHED AGE
csi-3fe... driver.longhorn.io pvc-5fd... cluster-node-2 true 308dIf this is a standard stuck VolumeAttachment, just delete it. You are not deleting the volume, no data will be lost (but always make backups!) - you are just deleting the metadata that tells kubernetes what node the PVC is attached to.
kubectl delete -n namespace delete volumeattachment csi-3fe...
Fix 2, more stuck
If deleting the VolumeAttachment doesn't work, there's something preventing it from being deleted. That something is usually a finalizer. Delete all finalizers:
kubectl edit volumeattachment csi-3fe...
Look for this section under metadata:
finalizers:
- external-attacher/driver-longhorn-ioRemove the second row (keep finalizers:). Now the VolumeAttachment should delete itself, if not use Fix 1 above.