Volume failed to attach : "Input/output error (os error 5)"

[veenadong]

Mayastor 2.7.1, after 1 node is disconnected, the volume failed to attach with error:

Events:
  Type     Reason       Age                 From     Message
  ----     ------       ----                ----     -------
  Warning  FailedMount  90s (x18 over 29m)  kubelet  MountVolume.MountDevice failed for volume "pvc-209c15bd-a46d-41d6-b9f5-6b63f5f90f39" : rpc error: code = Internal desc = Failed to stage volume 209c15bd-a46d-41d6-b9f5-6b63f5f90f39: attach failed: IO error: Input/output error (os error 5), args: hostnqn=nqn.2019-05.io.openebs:node-name:sc-os-160-node1.glcpdev.cloud.hpe.com,hostid=42168bd1-20a2-2591-5779-353d1054684f,nqn=nqn.2019-05.io.openebs:209c15bd-a46d-41d6-b9f5-6b63f5f90f39,transport=tcp,traddr=10.245.244.161,trsvcid=8420,reconnect_delay=10,ctrl_loss_tmo=1980,nr_io_queues=2

Logs:
mayastor-2024-10-15--20-08-44-UTC_failedto_attach.tar.gz