The cluster physical disk resource could not be brought online

Last week I found a very challenging issue on a Microsoft Windows Server 2003 R2 database cluster.
We were planning on doing some maintenance on a cluster. After we were done, we found out that not all cluster resource groups were working on one of the nodes.
After investigating, it seemed that those resource groups had not worked on that node since somewhere in October last year!

Problem:
Cluster resources could not be brought online on a specific cluster node. The resource works fine on other nodes.
Only the pfysical disk cluster resource could not be brought online, the IP Address and Network Name have no problems with being brought online.

Cause:
It seems someone has been playing around on the cluster node with a tool named “TrueCrypt”.
This tool claims drive letters. When you do that on a passive cluster node where the drive letter is assigned but not in use, it will corrupt the drive assignment.

Solution:
Following these steps has solved my issue, since it basically re-creates the disk association on the cluster node:

  1. Take the cluster resource group offline (Cluster Administrator).
  2. Remove all dependencies to the physical disk resource (Cluster Administrator).
  3. Remove the physical disk resource from the cluster (Cluster Administrator).
  4. Rescan the disk on the cluster node where it has the problems (Windows Disk Management).
  5. Remove the disk on the cluster node where it has the problems (Windows Disk Management).
  6. Reboot the cluster node.
  7. Attach the drive letter to the disk again on the cluster node where it had the problems (Windows Disk Management).
  8. Add the physical disk resource to the cluster resource group (Cluster Administrator).
  9. Configure the dependencies to the physical disk resource which you have previously removed (Cluster Administrator).
  10. Bring the cluster resource online (Cluster Administrator).


Lesson learned:
Test all cluster resources on all cluster nodes before you begin maintenance on a cluster.

Leave a Reply

Your email address will not be published. Required fields are marked *