SQL SERVER – Availability Group Missing or Deleted Automatically?

An availability group missing from SSMS does not by itself prove that someone deleted it. In the original 2017 case, my friend John reported a network incident. Afterward, two availability groups were absent from one SQL Server instance, while their cluster roles still appeared in a failed state. The databases on that node showed RESTORING.

Gouache painting of two server groups with one database card missing from the second group
Compare the SQL Server view with the cluster view before deciding what is missing.

John had already tried SQL Server restarts and server reboots without resolving the issue. I asked for the SQL Server error log. If you need its location, see ways to find ERRORLOG.

What the Log Established

Error: 19435, Severity: 16, State: 1.
AlwaysOn: WSFC AG integrity check failed for AG 'PROD_AG'
with error 41041, severity 16, state 0.
The local replica of availability group 'PROD_AG' is being removed.

The longer message said that SQL Server could not validate the availability-group configuration in the Windows Server Failover Clustering store. It also noted that local removal is expected when the group has been removed from another instance. A later line recorded the local replica changing from SECONDARY_NORMAL to NOT_AVAILABLE.

This establishes a local configuration-validation failure and replica-state change. It does not, on its own, identify the original cause or prove that the latest committed data exists on another copy. The earlier conclusion that there was automatically no data loss was too broad.

Check the Instance and Local Metadata

Connect directly to each expected SQL Server instance and confirm that SSMS is using the intended instance and login. Refresh Object Explorer, then compare the results below with the AG dashboard and cluster resources. Run these diagnostic queries with appropriate server visibility. The state DMVs require VIEW SERVER STATE on SQL Server 2019 and earlier, or VIEW SERVER PERFORMANCE STATE on SQL Server 2022 and later. Catalog metadata is also limited by the permissions of the login.

SELECT @@SERVERNAME AS registered_server_name,
       CONVERT(sysname, SERVERPROPERTY('ServerName')) AS current_server_name,
       SERVERPROPERTY('IsHadrEnabled') AS is_hadr_enabled;

SELECT ag.name AS availability_group, ar.replica_server_name,
       rs.is_local, rs.role_desc, rs.operational_state_desc,
       rs.connected_state_desc
FROM sys.availability_groups AS ag
JOIN sys.availability_replicas AS ar ON ar.group_id = ag.group_id
LEFT JOIN sys.dm_hadr_availability_replica_states AS rs
    ON rs.group_id = ar.group_id AND rs.replica_id = ar.replica_id
ORDER BY ag.name, ar.replica_server_name;

SELECT name, state_desc, group_database_id, replica_id
FROM sys.databases
WHERE group_database_id IS NOT NULL
   OR state_desc = N'RESTORING'
ORDER BY name;

On a secondary, the replica-state DMV reports local information; missing state columns for remote replicas do not establish a remote failure. Catalog visibility also depends on permissions and cluster communication. An empty result on one connection is evidence to investigate, not permission to rebuild the whole configuration.

A database in RESTORING is another observation, not a repair instruction. Do not run RESTORE WITH RECOVERY merely to make that label disappear. First establish the database’s intended role, synchronization history and the recovery plan.

Collect the Cluster and SQL Evidence Together

  • Preserve SQL Server error logs from every replica around the incident time.
  • Collect the Windows System events and WSFC logs, including quorum, resource and network changes.
  • Record which instance still has the AG metadata, the current primary if one exists, and the state of every database copy.
  • Check recent configuration work, host renames, patches and administrative actions alongside the network incident.
  • Confirm backup availability and the business recovery-point requirement before selecting a repair.

Errors 19435 and 41041 are not unique to one cause. For example, Microsoft documents a hostname mismatch scenario on particular SQL Server 2017 and 2019 update levels. That is a separate documented scenario, not evidence that John’s earlier case involved a rename.

Choose Recovery Only After Comparing the Copies

Do not use Failover Cluster Manager to force an AG move or delete cluster resources as an exploratory step. Likewise, dropping and recreating the AG is not the default response to this message. A repair may involve configuration correction or rejoining a replica, but the surviving topology and data state determine the supported procedure.

If the primary is unavailable, planned failover, forced failover and restoring from backups have different prerequisites and data-loss implications. Use the documented recovery procedure for the verified situation and involve the cluster or support team when configuration integrity remains uncertain.

The saved case shows the diagnostic message, but it does not contain a verified end-to-end recovery record. The useful lesson is to correlate the local SQL view with the cluster state and preserve evidence before changing either.

A missing availability group is not proof of a safe rebuild, it is a reason to establish the state of every copy.

Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.


Discover more from SQL Authority with Pinal Dave

Subscribe to get the latest posts sent to your email.

AlwaysOn, SQL Error Messages, SQL High Availability, SQL Server
Previous Post
SQL SERVER – Maintenance Plan – Could not load type ‘Microsoft.SqlServer.Dts. Runtime.Wrapper. IDTSPackageSigning100’ from Assembly
Next Post
SQL SERVER – Microsoft.SqlServer.Management.Sdk. Sfc.EnumeratorException: Failed to Retrieve Data for This Request

Related Posts

5 Comments. Leave new

  • Stephane St-Louis
    March 5, 2019 10:04 pm

    HI Pinel,
    Thanks for the post. I ended up here because I’m running into a similar problem. Except I feel you post only acknowledges the problem, without providing any insight as to what may have caused the issue.

    In my case, I’m trying to setup an Availability Group on Azure VMs. The failure occurs upon deployment, when attempting to run the CREATE AVAILABILITY GROUP command, although my automated scripts have no issue whatsoever when ran in a local DEV environment.

    I am currently working of workarounds in order to get things working in Azure. The main issue is with the way Azure handles IP addresses and routing. All IP addresses used in Azure VMs need to be defined in Azure, otherwise they are ignored. Therefore, the cluster IP address which moved around from one node to the other is simply not reachable. I’ve found posts about how to set up a Internal Load Balancer hack, which consists of setting up the cluster IP address as the FrontEnd address of the ILB, then setting up probes and rules for the ILB to be able to detect which of the nodes is currently the primary node, and route traffic accordingly.

    I got this hack to work.. so far… I am now able to have node 2 join the cluster, set up CAU, etc… Now I run into the problem of running into an error when trying to create the availability group. I believe it is probably related to the Cluster IP address routing issue, or something similar. Here are the actual error messages I get in my “Proof Of Concept” environment:

    – The state of the local availability replica in availability group ‘ag_POC_PROD’ has changed from ‘NOT_AVAILABLE’ to ‘RESOLVING_NORMAL’. The state changed because the local availability replica is joining the availability group. For more information, see the SQL Server error log or cluster log. If this is a Windows Server Failover Clustering (WSFC) availability group, you can also see the WSFC management console.
    – The state of the local availability replica in availability group ‘ag_POC_PROD’ has changed from ‘RESOLVING_NORMAL’ to ‘NOT_AVAILABLE’. The state changed because either the associated availability group has been deleted, or the local availability replica has been removed from another SQL Server instance. For more information, see the SQL Server error log or cluster log. If this is a Windows Server Failover Clustering (WSFC) availability group, you can also see the WSFC management console.
    – Error: 19435, Severity: 16, State: 1.
    – Always On: AG integrity check failed for AG ‘ag_POC_PROD’ with error 41044, severity 16, state 1.

    I’m working on it… Googling the error, I ended up here.

    I know I’ll have to apply a similar Load Balancer hack for the Listener. But I haven’t reached the point yet where I can create the listener, having not been able to even create the availability group.

    Back to your friend John: If my gut feeling is right, he probably ran into his problem after some soft of network issue. Maybe a switch or router fault or misconfiguration has caused a similar error pattern to the one I’m running into, and has caused the node to “disconnect”.

    Reply
  • Stephane St-Louis
    March 6, 2019 12:38 am

    Follow up: If anyone else is running into the same issue as I was, here’s how I finally managed to solve the problem for my specific case:

    After reviewing the Cluster logs, I found the following errors:

    ODBC Error: [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]The user does not have permission to perform this action. (297)
    Failed to run diagnostics command. See previous log for error message
    Disconnect from SQL Server

    It turned out that after solving the network issues, the problem now was improper permissions. After a little research, I ended up running the following SQL commands:

    GRANT ALTER ANY AVAILABILITY GROUP TO [NT AUTHORITYSYSTEM];
    GRANT CONNECT SQL TO [NT AUTHORITYSYSTEM];
    GRANT VIEW SERVER STATE TO [NT AUTHORITYSYSTEM];

    Afterward, I was able to create the Availability Group.

    What troubles me is that I never had to grant those permissions under my local DEV environment, but here under Azure, I had to.

    Reply
    • This is amazing – I wish you can write a blog post on this subject and I will publish it with due credit to you.

      Reply
  • shailja Mishra
    October 18, 2019 8:27 am

    I have configured SQL server always on, it working properly, had configured listener too listener is able to communicate from any of the cluster node, its IP is online…but I am unable to do the failover of SQL server availablity group, its failing with timeout error 41131.

    1)I have granted the access to endpoints to NT authority system
    2)port 5022 is listening from both the servers
    3)have also granted access to endpoints
    4)error code 1257 ,1258 is getting recorded in cluster logs for CNO permissions, but I am getting same in other servers while doing failover but its getting failed over (so not sure if its exactly a cause of issue)

    Is anyone out here who can comment a help here to me

    Reply
  • am also facing the same issue any solution?..no option to join the db’s in secondary as ag group missing in secondary.do we need to reconfugure the ag again?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.