k8s node can't mount pvc until node restarted

[morgoved]

2024-05-17T04:03:30.818498066Z stderr F E0517 04:03:30.818193 1 utils.go:81] GRPC error: rpc error: code = Internal desc = failed to stat file /var/lib/kubelet/pods/8dff9f0d-7f82-415c-9a5b-c32e81fb3ea4/volumes/kubernetes.iocsi/pvc-a06f67fa-953d-4136-bec3-363c4ea5f687/mount: lstat /var/lib/kubelet/pods/8dff9f0d-7f82-415c-9a5b-c32e81fb3ea4/volumes/kubernetes.iocsi/pvc-a06f67fa-953d-4136-bec3-363c4ea5f687/mount: host is down

What happened:
In one day service said that we can get files from pvc.
after restart i see this into csi logs.
Only atfer that i restarted kubernetes nodes and all worked.

• CSI Driver version: v1.14.0
• Kubernetes version (use kubectl version): 1.29.4
• OS (e.g. from /etc/os-release): VERSION="22.04.4 LTS (Jammy Jellyfish)"
• Kernel (e.g. uname -a): 5.15.0-107-generic
• Install tools: helm
• Others: k8s installed on kvm vms by kubeadm, cni - calico

[andyzhangx]

CSI driver is not responsible for remount, you need to reschedule the pod to other node to make remount happen

[morgoved]

CSI driver is not responsible for remount, you need to reschedule the pod to other node to make remount happen

It's woked if i reboot the node... I did cronjob for this for reboot node, but it's very big problem for use this on production environment..

[andyzhangx]

have you tried this?

there is a reconnect mount option in CIFS mount options that you can use to remount the CIFS share when the server is unavailable and then recovers.

Here's an example command to mount a CIFS share with the reconnect option:

mount -t cifs -o reconnect //<cifs-server>/<cifs-share> <mount-point>

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kxs-jnadeau]

How about adding the "hard" mount option on the PV ? That's what we do to survive NFS backend hiccups.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten