SQL SERVER – How to get historical deadlock Information from System Health Extended Events?

Let me start off by asking a simple question. How many of you have seen this error about historical deadlock information earlier?

Msg 1205, Level 13, State 45, Line 4
Transaction (Process ID 52) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

I am sure most of us who work with SQL Server would say “Yes!”

Now, next question is – what would you do if you see such error? Few DBAs would say; we need to enable trace flag 1222 and wait for the next occurrence so that information is recorded in the SQL Server ERRORLOG file.

Let me share a piece of advice. There is a session called as System_Health which is created in SQL Server and captured a lot of extended events by default.

SQL SERVER - How to get historical deadlock Information from System Health Extended Events? sys-health-deadlock-01

Here is the script of the session.

CREATE EVENT SESSION [system_health] ON SERVER 
ADD EVENT sqlclr.clr_allocation_failure(
    ACTION(package0.callstack,sqlserver.session_id)),
ADD EVENT sqlclr.clr_virtual_alloc_failure(
    ACTION(package0.callstack,sqlserver.session_id)),
ADD EVENT sqlos.memory_broker_ring_buffer_recorded,
ADD EVENT sqlos.memory_node_oom_ring_buffer_recorded(
    ACTION(package0.callstack,sqlserver.session_id,sqlserver.sql_text,sqlserver.tsql_stack)),
ADD EVENT sqlos.process_killed(
    ACTION(package0.callstack,sqlserver.client_app_name,sqlserver.client_hostname,sqlserver.client_pid,sqlserver.query_hash,sqlserver.session_id,sqlserver.session_nt_username)),
ADD EVENT sqlos.scheduler_monitor_deadlock_ring_buffer_recorded,
ADD EVENT sqlos.scheduler_monitor_non_yielding_iocp_ring_buffer_recorded,
ADD EVENT sqlos.scheduler_monitor_non_yielding_ring_buffer_recorded,
ADD EVENT sqlos.scheduler_monitor_non_yielding_rm_ring_buffer_recorded,
ADD EVENT sqlos.scheduler_monitor_stalled_dispatcher_ring_buffer_recorded,
ADD EVENT sqlos.scheduler_monitor_system_health_ring_buffer_recorded,
ADD EVENT sqlos.wait_info(
    ACTION(package0.callstack,sqlserver.session_id,sqlserver.sql_text)
    WHERE ([duration]>(15000) AND ([wait_type]>=N'LATCH_NL' AND ([wait_type]>=N'PAGELATCH_NL' AND [wait_type]<=N'PAGELATCH_DT' OR [wait_type]<=N'LATCH_DT' OR [wait_type]>=N'PAGEIOLATCH_NL' AND [wait_type]<=N'PAGEIOLATCH_DT' OR [wait_type]>=N'IO_COMPLETION' AND [wait_type]<=N'NETWORK_IO' OR [wait_type]=N'RESOURCE_SEMAPHORE' OR [wait_type]=N'SOS_WORKER' OR [wait_type]>=N'FCB_REPLICA_WRITE' AND [wait_type]<=N'WRITELOG' OR [wait_type]=N'CMEMTHREAD' OR [wait_type]=N'TRACEWRITE' OR [wait_type]=N'RESOURCE_SEMAPHORE_MUTEX') OR [duration]>(30000) AND [wait_type]<=N'LCK_M_RX_X'))), ADD EVENT sqlos.wait_info_external( ACTION(package0.callstack,sqlserver.session_id,sqlserver.sql_text) WHERE ([duration]>(5000) AND ([wait_type]>=N'PREEMPTIVE_OS_GENERICOPS' AND [wait_type]<=N'PREEMPTIVE_OS_ENCRYPTMESSAGE' OR [wait_type]>=N'PREEMPTIVE_OS_INITIALIZESECURITYCONTEXT' AND [wait_type]<=N'PREEMPTIVE_OS_QUERYSECURITYCONTEXTTOKEN' OR [wait_type]>=N'PREEMPTIVE_OS_AUTHZGETINFORMATIONFROMCONTEXT' AND [wait_type]<=N'PREEMPTIVE_OS_REVERTTOSELF' OR [wait_type]>=N'PREEMPTIVE_OS_CRYPTACQUIRECONTEXT' AND [wait_type]<=N'PREEMPTIVE_OS_DEVICEOPS' OR [wait_type]>=N'PREEMPTIVE_OS_NETGROUPGETUSERS' AND [wait_type]<=N'PREEMPTIVE_OS_NETUSERMODALSGET' OR [wait_type]>=N'PREEMPTIVE_OS_NETVALIDATEPASSWORDPOLICYFREE' AND [wait_type]<=N'PREEMPTIVE_OS_DOMAINSERVICESOPS' OR [wait_type]=N'PREEMPTIVE_OS_VERIFYSIGNATURE' OR [duration]>(45000) AND ([wait_type]>=N'PREEMPTIVE_OS_SETNAMEDSECURITYINFO' AND [wait_type]<=N'PREEMPTIVE_CLUSAPI_CLUSTERRESOURCECONTROL' OR [wait_type]>=N'PREEMPTIVE_OS_RSFXDEVICEOPS' AND [wait_type]<=N'PREEMPTIVE_OS_DSGETDCNAME' OR [wait_type]>=N'PREEMPTIVE_OS_DTCOPS' AND [wait_type]<=N'PREEMPTIVE_DTC_ABORT' OR [wait_type]>=N'PREEMPTIVE_OS_CLOSEHANDLE' AND [wait_type]<=N'PREEMPTIVE_OS_FINDFILE' OR [wait_type]>=N'PREEMPTIVE_OS_GETCOMPRESSEDFILESIZE' AND [wait_type]<=N'PREEMPTIVE_ODBCOPS' OR [wait_type]>=N'PREEMPTIVE_OS_DISCONNECTNAMEDPIPE' AND [wait_type]<=N'PREEMPTIVE_CLOSEBACKUPMEDIA' OR [wait_type]=N'PREEMPTIVE_OS_AUTHENTICATIONOPS' OR [wait_type]=N'PREEMPTIVE_OS_FREECREDENTIALSHANDLE' OR [wait_type]=N'PREEMPTIVE_OS_AUTHORIZATIONOPS' OR [wait_type]=N'PREEMPTIVE_COM_COCREATEINSTANCE' OR [wait_type]=N'PREEMPTIVE_OS_NETVALIDATEPASSWORDPOLICY' OR [wait_type]=N'PREEMPTIVE_VSS_CREATESNAPSHOT')))), ADD EVENT sqlserver.connectivity_ring_buffer_recorded(SET collect_call_stack=(1)), ADD EVENT sqlserver.error_reported( ACTION(package0.callstack,sqlserver.database_id,sqlserver.session_id,sqlserver.sql_text,sqlserver.tsql_stack) WHERE ([severity]>=(20) OR ([error_number]=(17803) OR [error_number]=(701) OR [error_number]=(802) OR [error_number]=(8645) OR [error_number]=(8651) OR [error_number]=(8657) OR [error_number]=(8902) OR [error_number]=(41354) OR [error_number]=(41355) OR [error_number]=(41367) OR [error_number]=(41384) OR [error_number]=(41336) OR [error_number]=(41309) OR [error_number]=(41312) OR [error_number]=(41313)))),
ADD EVENT sqlserver.security_error_ring_buffer_recorded(SET collect_call_stack=(1)),
ADD EVENT sqlserver.sp_server_diagnostics_component_result(SET collect_data=(1)
    WHERE ([sqlserver].[is_system]=(1) AND [component]<>(4))),
ADD EVENT sqlserver.sql_exit_invoked(
    ACTION(package0.callstack,sqlserver.client_app_name,sqlserver.client_hostname,sqlserver.client_pid,sqlserver.query_hash,sqlserver.session_id,sqlserver.session_nt_username)),
ADD EVENT sqlserver.xml_deadlock_report
ADD TARGET package0.event_file(SET filename=N'system_health.xel',max_file_size=(5),max_rollover_files=(4)),
ADD TARGET package0.ring_buffer(SET max_events_limit=(5000),max_memory=(4096))
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=120 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=ON)
GO

There is no need to worry about a lot of information there. The information which I want to highlight is below.

ADD EVENT sqlserver.xml_deadlock_report

Above means that by default SQL Server should be capturing XML deadlock graph. To check this I have done little experiment.

Create Database and Objects

First create a database and two tables.

CREATE DATABASE DeadlockDemo
GO
USE DeadlockDemo
GO
CREATE TABLE MyT1 (i INT)
GO
CREATE TABLE MyT2 (i INT)
GO
INSERT INTO MyT1
VALUES (1)
GO
INSERT INTO MyT2
VALUES (1)
GO

Reproduce Deadlock Error

We need to connections to reproduce deadlock.

Connection IDQuery
Connection # 1begin tran
Connection # 2begin tran
Connection # 1update MyT1 set i= 3
Connection # 2update MyT2 set i= 3
Connection # 1Select * from MyT2
Connection # 2Select * from MyT1

Once above steps are complete, once of the connection would get deadlock error “Transaction (Process ID 52) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.”

Look at the deadlock graph from System health session:

Here is the query which we can use to read deadlock graph which is captured in system health session.

SELECT XEvent.query('(event/data/value/deadlock)[1]') AS DeadlockGraph
FROM (
	SELECT XEvent.query('.') AS XEvent
	FROM (
		SELECT CAST(target_data AS XML) AS TargetData
		FROM sys.dm_xe_session_targets st
		INNER JOIN sys.dm_xe_sessions s ON s.address = st.event_session_address
		WHERE s.NAME = 'system_health'
			AND st.target_name = 'ring_buffer'
		) AS Data
CROSS APPLY TargetData.nodes('RingBufferTarget/event[@name="xml_deadlock_report"]') AS XEventData(XEvent)
) AS source;

And we will get same graph which we get via trace flag 1222 in the error log.

Do above experiment and share your experience via the comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

Deadlock, SQL Error Messages, SQL Extended Events, SQL Scripts, SQL Server
Previous Post
SQL SERVER – Unable to Start SQL Service – Server TCP provider failed to listen on [‘any’ 1433]. Tcp port is already in use.
Next Post
SQL SERVER – Query to Get the List of Logins Having System Admin (sysadmin) Permission

Related Posts

26 Comments. Leave new

  • Great and Thanks ..with out trace flag no need to consider about performance. this is i am looking . so many clients not interest to enable trace flags as it may cause of performance degradation ..!!

    Reply
  • This looks awesome, it appears though that it gives only the single last deadlock. Is there a way to look back further?

    Reply
  • Hello Pinal, I tried this and its working as expected in SQL 2012. I am really thankful to you

    Reply
  • Post is awesome. I have question, I was checking for a deadlock happened 2 days ago and when I get the reports from this query, it is only showing deadlocks of today. (I am referring to “lasttranstarted” in process header to check date.). Is there a way I can get details for old deadlocks.

    Thanks

    Reply
    • This doesn’t keep records for all deadlocks. These traces are designed to roll over to have latest information. you have to deploy your own xevent for long term monitoring.

      Reply
  • On my system, the deadlock event refused to show up in the ring buffer. I think some other event was using all the ring buffer space. However the deadlock would show up in the system health file.

    select event_data = CONVERT(XML, event_data)
    from sys.fn_xe_file_target_read_file(N’system_health*.xel’, NULL, NULL, NULL)
    where event_data like ‘%xml_deadlock%’

    Also:

    https://docs.microsoft.com/en-us/collaborate/connect-redirect#tabs

    https://www.sqlskills.com/blogs/jonathan/why-i-hate-the-ring_buffer-target-in-extended-events/

    Reply
  • Hi Pinal, I was able to produce the deadlock error but when I run the query to see the Graph I do not get any output. I’m using SQL 2014. Am I missing anything? Thanks!

    Reply
  • I was just notified to find causes of deadlock that happened 20 days ago. Which is the best way to achieve that? Its urgent if you have the solution. Thank you.

    Reply
  • I am receiving below error for this querry:

    Msg 9420, Level 16, State 1, Line 1
    XML parsing: line 27870, character 60, illegal xml character

    Any idea what is wrong here?

    Reply
  • Hi Pinal,

    In sql server 2014 enterprise, deadlock event is not being captured sometimes in system_health extended events.

    Is there any fix for this?

    Reply
    • Might be an issue with extended events capturing deadlock. Do you have a repro? You should report to Microsoft.

      Reply
  • Fabulous….

    Thank you Pinal, as always

    Reply
  • Sreejith Rajeevan
    May 10, 2018 8:37 pm

    Can i use the same script to to capture deadlock in SQL Server 2008 R2.?

    Reply
  • hi can you pls help me. how to resolve lock in sql server 2012?. i am new for the sql server!

    Reply
  • Hello Pinal,

    I have tried the above the scenario of creating DB and other stuffs but couldn’t see the deadlock graph there. Can you please elaborate more on this?

    Thanks!

    Reply
  • Hello.

    Will this trace affect anything on the system regarding CPU or memory?

    Hilde

    Reply
  • hi pinal dave how to remove dedlocks pls help

    i’m getting this issue…

    Msg 1205, Level 13, State 45, Line 4
    Transaction (Process ID 71) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

    Reply
  • HI Pinal , we found Deadlock information xml query how to know its occurs which Database and table?

    Reply
  • We had a locking issues a few days ago and so ran your query against our database, however it returns no records.

    Reply

Leave a Reply