This was one of an interesting issue reported by my client. They informed that after some hardening by the security team, they are not able to bring SQL resource online in the cluster. In this blog, we would learn about how to fix the issue where SQL cluster is not coming online.
In my client’s environment, there were two nodes (Node1 and Node2) They noticed that in “Failover Cluster Manager” and found that SQL resource was in “Online Pending” state.
After 3 minutes, it went to “failed” state. This happens when the cluster is not able to connect to SQL and showing in online pending even if service is running. Pending timeout in the cluster was set to 3 minutes. (default value is 3 minutes). Which means that we would see “online pending” for 3 minutes before goes to the failed state. After this SQL Service goes to stopped automatically.
I checked ERRORLOG and found below
2018-04-28 03:05:47.43 spid5s Recovery is complete. This is an informational message only. No user action is required.
2018-04-28 03:05:50.07 Logon Error: 18470, Severity: 14, State: 1.
2018-04-28 03:05:50.07 Logon Login failed for user ‘Domain\Node1$’. Reason: The account is disabled. [CLIENT: 128.xxx.xxx.xxx]
We also checked Cluster log and found the same error there as well.
Now, we know that there is no issue with SQL startup, it is the SYSTEM account which is not able to connect to SQL Server to show the cluster resource online.
To fix the issue, we followed steps outlined below.
- Started SQL via command line.
NET START MSSQLSEVER
- Connected to SQL via SSMS
- Found that “NT AUTHORITY\SYSTEM” account was disabled. We enabled the account by using below T-SQL.
ALTER LOGIN [NT AUTHORITY\SYSTEM] ENABLE
Note: you can also do it via SSMS by going to Login > Properties > Status tab (as shown below).
Enable the login and hit OK.
- We stopped SQL via command line
NET STOP MSSQLSEVER
After following above steps, we attempted to bring SQL online and it came online. We also tested failover to another node and it worked like a charm.
Reference: Pinal Dave (https://blog.sqlauthority.com)