When they contacted me, the situation was that the primary replica was showing secondary in a disconnected state. When I asked to check SQL Server ERRORLOG, we found below on secondary replica.
Database Mirroring login attempt failed with error: ‘Connection handshake failed. An OS call failed: (80090350) 0x80090350(The system cannot contact a domain controller to service the authentication request. Please try again later.). State 66.’. [SERVER: 10.xx.xx.xx]
The IP address which is at the end of the error message was the IP address of the primary replica. One of the interesting things about error message is that even though they were not using database mirroring the error message was saying -“Database Mirroring Login”. I think this is because of the fact that the endpoint is same for availability group and mirroring.
I asked more about how they landed up in this situation. They informed me that there were some network issues. Their network team fixed the issue but right now, they have an issue, where the mirror server is not able to get in sync with the principal and LDF file, is growing.
I have an existing blog on the same error message. SQL SERVER – Database Mirroring login attempt failed – Connection handshake failed
In our case hexadecimal code was different. It is 0x80090350. I used COM Error Codes (Security and Setup) and it says SEC_E_DOWNGRADE_DETECTED – The system cannot contact a domain controller to service the authentication request. Please try again later. That is the message which we are seeing ERRORLOG as well. So, we are in the right direction.
We were able to log in to SQL using Windows accounts on the mirror server. We were also able to ping domain controller. I asked them to restart the machine so that it re-establishes the connection to the domain controller. Since it was a secondary replica, there were no problems in rebooting it. As expected, it worked for them and after the restart, the replicas were able to talk to each other and AlwaysOn started working.
Have you ever encountered such cryptic errors?
Reference: Pinal Dave (https://blog.SQLAuthority.com)