Script of Backup, Integrity Check and Index Optimization are the most important scripts for any developer. SQL Expert and true SQL enthusiast Ola Hallengren is known for his excellent scripts.

Please try it out and let me know what you think. The documentation is available on http://ola.hallengren.com/Documentation.html and the script can be downloaded from http://ola.hallengren.com.

Here is brief documentation sent by Ola himself for his script in his own words.

Backup Maintenance

I think that most of you have experienced the error messages “BACKUP LOG cannot be performed because there is no current database backup.” and “Cannot perform a differential backup for database “”, because a current database backup does not exist.”.

This usually happens when you have created a new database or when you have changed recovery model of a database from Simple to Full.

The consequence is that the database is not getting backed up, until a full backup (or a differential backup for the BACKUP LOG error message if that can be done) has been performed. It will also create some noise in your monitoring system.

The solution is to check if a differential or transaction log backup can be performed before doing the backup. This can be done by checking sys.master_files.differential_base_lsn and sys.database_recovery_status.last_log_backup_lsn.

The backup solution that I have developed has a parameter called @ChangeBackupType.

EXECUTE dbo.DatabaseBackup @Databases = 'USER_DATABASES', @Directory = 'C:\Backup', @BackupType = 'LOG', @ChangeBackupType = 'Y'

If it’s a new database in Full recovery model, the backup type for that database and job run will be changed to full. If it’s a database that was newly changed to Full recovery model, then the backup type for that database and job run will be changed to differential. The next time the job runs a transaction log backup will be performed for that database.

New databases start getting backed up quickly and no more “BACKUP LOG cannot be performed because there is no current database backup.” and “Cannot perform a differential backup for database “”, because a current database backup does not exist.”.

Index Optimization

The design idea is categorize all indexes based on their fragmentation level (High, Medium or Low) and whether there are columns with LOB (Large Object) data types. For each category you can define an action. The possible actions are to rebuild indexes online or offline, reorganize indexes, update statistics, reorganize indexes and update statistics or to do nothing.

Here’s an example.

EXECUTE dbo.IndexOptimize
@Databases = 'USER_DATABASES',
@FragmentationHigh_LOB = 'INDEX_REBUILD_OFFLINE',
@FragmentationHigh_NonLOB = 'INDEX_REBUILD_ONLINE',
@FragmentationMedium_LOB = 'INDEX_REORGANIZE',
@FragmentationMedium_NonLOB = 'INDEX_REORGANIZE',
@FragmentationLow_LOB = 'NOTHING',
@FragmentationLow_NonLOB = 'NOTHING',
@FragmentationLevel1 = 5,
@FragmentationLevel2 = 30,
@PageCountLevel = 1000


Indexes with a fragmentation above 30% are to be rebuilt, online if possible (no LOB columns), otherwise offline (LOB columns). Indexes with a fragmentation between 5% and 30% are to be reorganized. Indexes with a fragmentation below 5% or a size below 1000 pages are not to be touched.

If you are using partitioning IndexOptimize has a parameter, @PartitionLevel to do index rebuilds and reorganizations on the partition level. If you prefer to do sort operations in tempdb you can do that with the parameter @SortInTempdb and if you would like to set a fillfactor you can do that with the parameter @FillFactor.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

This is very short note announcing details about my course details for ‘MS SQL Server 2005/2008 Query Optimization And Performance Tuning‘.

The course is scheduled to be delivered from 19th Nov, 2009 and 21st Nov, 2009 in Pune, India.

This three day course is an intensive course designed to give attendees an in-depth look at the query optimization and performance tuning concepts and methods found in SQL Server 2005/2008. This course is designed to prepare the SQL Server developers and administrators for a transition to SQL Server 2005/2008 while discussing best practices for a variety of topics.

Day 1 : Strengthen the basics along with SQL Server 2005/2008 New Features

Module 01: Subqueries, Ranking Functions, Joins and Set Operations
Module 02: Table Expressions
Module 03: TOP and APPLY
Module 04: SQL Server 2008 Enhancements

Day 2: Query Optimization & Performance Tuning 1

Module 05: Logical Query Processing
Module 06: Query Tuning
Module 07:  Introduction to the Query Processor
Module 08:  Review of common query coding which causes poor performance

Day 3: Query Optimization & Performance Tuning 2

Module 09:  SQL Server Indexing and index maintenance
Module 10:  Plan Guides, query hints, UDFs, and Computed Columns
Module 11:  Understanding SQL Server Execution Plans
Module 12: Real World Index and Optimization Tips

The space are limited and I encourage you to register as soon as possible.

Call to register – +91 98817 95739, +91 98508 43664, +91 99229 33319

Email to register – workshop “at” seedinfotech.com

If due to any reason, you can not attend this course, please call or email on details above for future schedule for the course and location.

Click to Enlarge

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Recently, I have been working on Query Optimization project. While working on it, I found the following interesting observation. This entire concept may appear very simple, but if you are working in the area of query optimization and server tuning, you will find such useful hints.

Before we start, let us understand the difference between Seek Predicate and Predicate. Seek Predicate is the operation that describes the b-tree portion of the Seek. Predicate is the operation that describes the additional filter using non-key columns. Based on the description, it is very clear that Seek Predicate is better than Predicate as it searches indexes whereas in Predicate, the search is on non-key columns – which implies that the search is on the data in page files itself.

Earlier, I had posted regarding how to remove key lookup and bookmark lookup in the following:

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3

Let us see an example where we will remove the bookmark lookup first using the covering index. On removing the bookmark lookup, it resulted in Index Scan, which is not good for performance. When Index Scan is converted to Index Seek, it provides  a significant improvement in performance.

Run following SELECT, which is based on the database AdventureWorks.

USE AdventureWorks
GO
-- CTRL + M Enforces Key Lookup
-- Try 1
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = 14417807
GO

Let us check the execution plan for the same. The execution plan consists of the key lookup because there are columns that we are trying to retrieve in SELECT as well in WHERE clause.

Let us create a covering index on this table HumanResources.Employee.

-- Create Non clustered Index
CREATE NONCLUSTERED INDEX [IX_HumanResources_Employee_Example] ON HumanResources.Employee
(
NationalIDNumber ASC, HireDate, MaritalStatus
) ON [PRIMARY]
GO

After creating above index, let us run the same SELECT statement again.

-- CTRL + M Removes Key Lookup, but it still enforces Index Scan
-- Try 2
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = 14417807
GO

Due to non-clustered index, Key Lookup is removed along with Nested Loops but Index Scan still remains, and this is not good for performance.

I tried to remove the scan, but my attempts were unsuccessful. I finally looked at the datatype of the NationalIDNumber. All this time, I was assuming that this datatype is INT, but on a careful check, I found that the datatype of NationalIDNumber is nvarchar(15).

In the SELECT statement, we were comparing the datatype of NVARCHAR to INT, and this was forcing the predicate operation while executing the query.

As discussed earlier, due to predicate operation, there has to be explicit conversion on the side of NationalIDNumber, which forces the query optimizer to not use the index and instead it has to scan the complete data in table to get the necessary data. This is not the desired solution. Index Scan reduces performance. The reason for this conversion is because I am using INT value in WHERE clause instead of NVARCHAR.

I changed my WHERE clause and passed STRING as the parameter instead of INT.

-- CTRL + M Removes Key Lookup and it enforces Index Seek
-- Try 3
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = '14417807'
GO

After running the query with the changed WHERE clause, the Index Scan is now converted into Index Seek.

Index Seek is definitely the most optimal solution in this particular scenario.

When the detail execution plan was checked, I found the following two notable points. First, the predicate is converted to seek predicate, which is the reason for performance improvement, as described earlier. Instead of scanning data in the table, Index Seek is performed. Second, as the datatype of the NationalIDNumber is NVARCHAR and the parameters are passed as VARCHAR, the conversion happens on the parameters instead of NationalIDNumber column, and this forces Index Scan to Index Seek.

If we pass the parameter as NVARCHAR instead of VARCHAR, the execution plan will remain the same, but CONVERT_IMPLICIT will not be required any more. Let us run the following query, which has NVARCHAR as the parameter.

-- CTRL + M Removes Key Lookup and it enforces Index Seek and no CONVERT_IMPLICIT
-- Try 4
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = N'14417807'
GO

The execution plan of the above query is very similar to that in which we had passed the parameter as VARCHAR.

Now let us check the execution plan for the same.

In the WHERE condition, the operators we have on both the sides of “=” are of NVARCHAR. NationalIDNumber and parameter passed – both are NVARCHAR, which has removed CONVERT_IMPLICIT. However, there are no changes in the execution plan.

In summary, when Key Lookup is removed and index seek replaces index scan, the performance is tuned up. Let us quickly compare the execution plan of the above four options. I have included the complete code here for easy reference.

USE AdventureWorks
GO
-- CTRL + M Enforces Key Lookup
-- Try 1
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = 14417807
GO
-- Create Non clustered Index
CREATE NONCLUSTERED INDEX [IX_HumanResources_Employee_Example] ON HumanResources.Employee
(
NationalIDNumber ASC, HireDate, MaritalStatus
) ON [PRIMARY]
GO
--WAITFOR DELAY '00:00:30'
-- CTRL + M Removes Key Lookup, but it still enforces Index Scan
-- Try 2
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = 14417807
GO
-- CTRL + M Removes Key Lookup and it enforces Index Seek
-- Try 3
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = '14417807'
GO
-- CTRL + M Removes Key Lookup and it enforces Index Seek and no CONVERT_IMPLICIT
-- Try 4
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = N'14417807'
GO
/* What is the reason for difference between Try 2 and Try 3?
Check the exeuction plan
*/
-- Drop Index
DROP INDEX [IX_HumanResources_Employee_Example] ON HumanResources.Employee WITH ( ONLINE = OFF )
GO

Let us look at the execution plan.

Summary points:

• In general, Index Seek is better than Index Scan (I am not talking about it depends conditions)
• Understand Predicates and Seek Predicates and see if you have only Seek Predicates.
• In case of key lookup or bookmark lookup, see if you can create a covering index or included column index.
• Use the datatype wisely even though there is no change in the resultset.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Change data capture (CDC) is a new feature in SQL Server 2008 designed to capture insert, update, merge, and delete activities applied to SQL Server tables and to avail those changes in an easy-to-understand format.

Conventionally, detecting changes in a source database to transfer these changes to a data warehouse required any of the following:

• Special columns in the source tables (time stamps, row versions).
• Triggers that capture changes.
• Comparison of the source and the destination systems.

The above methods can have significant disadvantages: special columns require a change in the source database schema, and in many cases, a change in the application logic. Triggers must be implemented manually, and this can lead to significant additional overhead on DML commands to the source system. Comparing the source and target databases can also place a heavy load on both systems.

CDC offers a win-win alternative that allows the transfer of changes without forcing the DBA to either place a heavy load on the server or add the above change-detection steps. CDC enables the capture of changes by asynchronously reading the transaction log of the source database and then recording the changes in those logs in special change tables for uploading later into the target database. Therefore, no comparison between the source and target systems is needed for changes.

Read white paper of Tuning the Performance of Change Data Capture in SQL Server 2008

Download Script of Change Data Capture (CDC)

Introduction to Change Data Capture (CDC)

Reference: Pinal Dave (http://blog.sqlauthority.com)

PASS Summit 2009 is now only 10 days away and I am very excited for the same. I can not wait to attend the summit as this is the most awaited conference of SQL Server in world. Everybody will be there and there will be something for everybody. My core expertise is in Query Optimization and Performance Tuning area, and when I see the list of PASS session on the subject, I am totally speechless. There are so many great speaker at PASS who are there to talk on the subject.

It is absolutely not possible to attend all of them many of them are running parallel to each other. In any case, I strongly suggest if you are the expert who try till last bit to optimize the query and performance, I expect you to attend many of the following session. There is always option to get recorded DVD for the sessions afterwords.

You can read my article here regarding Why I am Going to Attend PASS Summit Unite 2009- Seattle.

Here is my list of the sessions related to performance tuning and query optimization.

Making Effective Use of the Plan Cache
Greg Low (Solid Q Australia)

Capturing and Analyzing File & Wait Stats
Andrew Kelly (Solid Quality Mentors)

T-SQL Tips & Tricks
Itzik Ben-Gan (Solid Quality Mentors)

Best Practices for Working With Execution Plans
Grant Fritchey (FM Global)

DMV’s as a Shortcut to Procedure Tuning
Grant Fritchey (FM Global)

Lies, damned lies and statistics
Gail Shaw (XpertEase)

Insight into Indexes
Gail Shaw (XpertEase)

Enhance your T-SQL to perform better
Joe Webb (WebbTech Solutions, LLC)

Exploring Index Internals
Kalen Delaney (SQLearning)

Super Bowl, Super Load – A Look at Performance Tuning for VLDB’’s
Michelle Ufford (GoDaddy)

Optimizing SQL Server 2008 Applications using Table Valued Parameters, XML and MERGE
Tobias Ternstrom (Microsoft Corp.)

SQL Server Execution Plans From Compilation To Caching To Reuse
Maciej Pilecki (Project Botticelli Ltd.)

Using SQL Server 2008 for Performance Tuning
Buck Woody (Microsoft Corp.)

SQLCAT: Designing High Performance I/O for SQL Server (90 Mins)
Thomas Kejser (Microsoft Corp.)
Mike Ruthruff (Microsoft SQL CAT)

SQL Server 2005 / 2008 Performance Tuning and Optimization Techniques (90 Mins)
David Pless (Microsoft CSS)

Dr. DMV: How to Use Dynamic Management Views to Monitor and Diagnose Performance Issues With High Volume OLTP Workloads
Glenn Berry (NewsGator Technologies)

How the Query Optimizer Works
Ben Nevarez (AIG)

I will personally try to attend many of the above listed sessions. Here are two sessions not from above list, which I will be sure attending. You are encouraged to meet me and attend this two rock star sessions.

Overcoming SSIS Deployment and Configuration Challenges
Rushabh Mehta (Solid Quality Learning)

SQL Server 2008 – Create powerful XML Schema collections to validate your XML documents
Jacob Sebastian (beyondrelational.com)

Let me know what sessions you will be sure attending by leaving your comment here. It would be good idea to create the list of the session which one is confident about attending.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

Earlier I have written two different articles on the subject Remove Bookmark Lookup. This article is as part 3 of original article. Please read first two articles here before continuing reading this article.

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

We read in above articles that we can remove bookmark lookup using covering index. Covering Index is the index, which contains all the columns used in SELECT as well in JOINs and WHERE conditions. In our example we have created clustered index first.

-- Create Clustered Index
CREATE CLUSTERED INDEX [IX_OneIndex_ID] ON [dbo].[OneIndex]
(
[ID] ASC
) ON [PRIMARY]
GO

Based on clustered index we have created following non clustered index. Please note that we do not have to create both the index together. We can create either covering index or included column index along with it. Please note, I am suggesting to create either of it, not both of it.

In earlier article I have suggested to include all columns but in reality in any non clustered index there is no need to include columns included in clustered index. All non clusteredindex automatically contains pointers to clustered index any way.

We should create index described in earlier article as following.

Method 1: Creating covering non-clustered index.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Cover] ON [dbo].[OneIndex]
(
City, FirstName
) ON [PRIMARY]
GO

Method 2: Creating included column non-lustered index.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Include] ON [dbo].[OneIndex]
(
City
) INCLUDE (FirstName) ON [PRIMARY]
GO

Let us examine the execution plan and compare the query costs and also verify that if both the index usages are forcing index seek instead of index scan.

As discussed in the example, any non clustered index does not need to include columns which are included in the clustered index.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Related Post:

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3

Download Logical Query Processing Poster

Earlier this year, I had written article on SQL SERVER – Logical Query Processing Phases – Order of Statement Execution and I had asked one question to readers. I got very good response for this question. Today, I am going to discuss about one of the errata I have made there. I had displayed the Logical Query Processing order, where I had incorrectly listed the last two operations. I have listed the operations as ORDER BY first and TOP afterwards. The fact is that TOP is always executed first and ORDER BY after that.

The correct order of the Logical Query Processing Phases is as follows:

1. FROM
2. ON
3.OUTER
4.WHERE
5.GROUP BY
6.CUBE | ROLLUP
7.HAVING
8. SELECT
9. DISTINCT
10. TOP
11. ORDER BY

This logical order is properly explained by Solid Quality Mentor and SQL Expert Itzik Ben-Gan in Logical Query Processing Phases in his book Inside Microsoft SQL Server 2008: T-SQL Querying. This book contains the poster that explains the Logical Query Processing Phases.

Download Logical Query Processing Poster

I had mentioned in the original post that I will give the best explanation of the subject in my blog post with due credit. Following are the excellent comments and explanation received as comments on the blog. I am listing the selected portion from the original comments.

Brian Tkatch

If the query includes an ORDER BY clause, the first expression rows, or expression percent of rows, ordered by the ORDER BY clause are returned. If the query has no ORDER BY clause, the order of the rows is arbitrary.

Mike Walsh

The ORDER BY is ordering your results, it is forcing them to come back in a certain way (contrary to relational theory :) ). The TOP operator is telling SQL Server to restrict the rows returned to a certain number of rows (either explicitly or by percentage). If TOP happened first that would mean you would just be ordering by the number of rows specified in the top.

It would appear as though to do the top it has to know the Order By clause and it needs to understand there is an Order By, and then filter only the top number/percent of rows according to that.

The actual ordering for display or passing on to the next query happens after the TOP filter is applied (even though it looked at the order by).

Bob

In the case of ORDER BY and TOP, that in the absence of an ORDER BY clause, TOP will return arbitrary rows. Therefore, you can consider that ORDER BY precedes TOP when ORDER BY is specified, however, that may not be what actually happens procedurally, just what apparently happens when the results come back.

I am sure that after downloading poster and reading the comments on the original article, it will clarify why TOP operators come earlier than ORDER BY in the query execution plan.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Posted in Pinal Dave, SQL, SQL Authority, SQL Download, SQL Optimization, SQL Query, SQL Server, SQL Tips and Tricks, T SQL, Technology Tagged: Logical Query Processing Phases ]]> http://blog.sqlauthority.com/2009/10/10/sql-server-download-logical-query-processing-poster/feed/ 1 pinaldave http://blog.sqlauthority.com/2009/10/08/sql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup-part-2/ http://blog.sqlauthority.com/2009/10/08/sql-server-query-optimization-remove-bookmark-lookup-remove-rid-lookup-remove-key-lookup-part-2/#comments Thu, 08 Oct 2009 01:30:53 +0000 pinaldave http://blog.sqlauthority.com/?p=6967 ]]>

This article is follow up of my previous article SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup. Please do read my previous article before continuing further.

I have described there two different methods to reduce query execution cost. Let us compare the performance of the SELECT statement of the previous query.

We have created two different indexes on the table.

Method 1: Creating covering non-clustered index.

In this method, we will create a non-clustered index that contains the columns used in the SELECT statement along with the column used in the WHERE clause.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Cover] ON [dbo].[OneIndex]
(
City, FirstName, ID
) ON [PRIMARY]
GO

Method 2: Creating included column non-lustered index.

In this method, we will create a nonclustered index that includes the columns used in the SELECT statement along with the column used in the WHERE clause. Here, we will use new syntax introduced in SQL Server 2005. An index with included nonkey columns can significantly improve query performance when all the columns in the query are included in the index.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Include] ON [dbo].[OneIndex]
(
City
) INCLUDE (FirstName,ID) ON [PRIMARY]
GO

Let us compare the performance of our query by running the following query with two different indexes.

SELECT ID, FirstName
FROM OneIndex WITH (INDEX(IX_OneIndex_Cover))
WHERE City = 'Las Vegas'
GO
SELECT ID, FirstName
FROM OneIndex WITH (INDEX(IX_OneIndex_Include))
WHERE City = 'Las Vegas'
GO

Let us examine the execution plan and compare the query costs.

In fact, in this example, the performance of both the queries is quite same. In this case, we can use either of the query and obtain the same performance.

I have mentioned in my previous article that I prefer the Method 2. The reason is that method 2 has many advantages over method 1.

1) Index can exceed the 900-byte limitation of the index key.

2) Index can include datatypes that are not allowed as key columns – varchar(max), nvarchar(max) or XML.

3) Size of the key index can be reduced, which improves the overall performance of the index operation.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Related Post:

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3

Today, I would like to share one very quick tip about how to remove bookmark lookup or RID lookup. Let us first understand Bookmark lookup or RID lookup. Please note that from SQL Server 2005 SP1 onwards, Bookmark look up is known as Key look up.

When a small number of rows are requested by a query, the SQL Server optimizer will try to use a non-clustered index on the column or columns contained in the WHERE clause to retrieve the data requested by the query. If the query requests data from columns not present in the non-clustered index, SQL Server must go back to the data pages to get the data in those columns. Even if the table has a clustered index or not, the query will still have to return to the table or clustered index to retrieve the data.

In the above scenario, if table has clustered index, it is called bookmark lookup (or key lookup); if the table does not have clustered index, but a non-clustered index, it is called RID lookup. This operation is very expensive. To optimize any query containing bookmark lookup or RID lookup, it should be removed from the execution plan to improve performance. There are two different ways to remove bookmark/RID lookup.

Before we understand these two methods, we will create sample table without clustered index and simulate RID lookup. RID Lookup is a bookmark lookup on a heap that uses a supplied row identifier (RID).

USE tempdb
GO
-- Create Table OneIndex with few columns
CREATE TABLE OneIndex (ID INT,
FirstName VARCHAR(100),
LastName VARCHAR(100),
City VARCHAR(100))
GO
-- Insert One Hundred Thousand Records
INSERT INTO OneIndex (ID,FirstName,LastName,City)
SELECT TOP 100000 ROW_NUMBER() OVER (ORDER BY a.name) RowID,
'Bob',
CASE WHEN ROW_NUMBER() OVER (ORDER BY a.name)%2 = 1 THEN 'Smith'
ELSE 'Brown' END,
CASE
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%1000 = 1 THEN 'Las Vegas'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 1 THEN 'New York'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 5 THEN 'San Marino'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 3 THEN 'Los Angeles'
ELSE 'Houston' END
FROM sys.all_objects a
CROSS JOIN sys.all_objects b
GO

Now let us run following select statement and check the execution plan.

SELECT ID, FirstName
FROM OneIndex
WHERE City = 'Las Vegas'
GO

As there is no index on table, scan is performed over the table. We will create a clustered index on the table and check the execution plan once again.

-- Create Clustered Index
CREATE CLUSTERED INDEX [IX_OneIndex_ID] ON [dbo].[OneIndex]
(
[ID] ASC
) ON [PRIMARY]
GO

Now, run following select on the table once again.

SELECT ID, FirstName
FROM OneIndex
WHERE City = 'Las Vegas'
GO

It is clear from execution plan that as a clustered index is created on the table, table scan is now converted to clustered index scan. In either case, base table is completely scanned and there is no seek on the table.

Now, let us see the WHERE clause of our table. From our basic observation, if we create an index on the column that contains the clause, a performance improvement may be obtained. Let us create non-clustered index on the table and then check the execution plan.

-- Create Index on Column City As that is used in where condition
CREATE NONCLUSTERED INDEX [IX_OneIndex_City] ON [dbo].[OneIndex]
(
[City] ASC
) ON [PRIMARY]
GO

After creating the non-clustered index, let us run our select statement again and check the execution plan.

SELECT ID, FirstName
FROM OneIndex
WHERE City = 'Las Vegas'
GO

As we have an index on the WHERE clause, the SQL Server query execution engine uses the non-clustered index to retrieve data from the table. However, the columns used in the SELECT clause are still not part of the index, and to display those columns, the engine will have to go to the base table again and retrieve those columns. This particular behavior is known as bookmark lookup or key lookup.

There are two different methods to resolve this issue. I have demonstrated both the methods together; however, it is recommended that you use any one of these methods for removing key lookup. I prefer Method 2.

Method 1: Creating non-clustered cover index.

In this method, we will create non-clustered index containing the columns, which are used in the SELECT statement, along with the column which is used in the WHERE clause.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Cover] ON [dbo].[OneIndex]
(
City, FirstName, ID
) ON [PRIMARY]
GO

Once the above non-clustered index, which covers all the columns in query, is created, let us run the following SELECT statement and check our execution plan.

SELECT ID, FirstName
FROM OneIndex
WHERE City = 'Las Vegas'
GO

From the execution plan, we can confirm that key lookup is removed the only index seek is happening. As there is no key lookup, the SQL Server query engine does not have to go to retrieve the data from data pages and it obtains all the necessary data from index itself.

Method 2: Creating an included column non-clustered index.

Here, we will create non-clustered index that also includes the columns, which are used in the SELECT statement, along with the column used in the WHERE clause. In this method, we will use new syntax introduced in SQL Server 2005. An index with included nonkey columns can significantly improve query performance when all columns in the query are included in the index.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Include] ON [dbo].[OneIndex]
(
City
) INCLUDE (FirstName,ID) ON [PRIMARY]
GO

From the execution plan, it is very clear that this method  also removes the key lookup as well.

In summary, Key lookup, Bookmark lookup or RID lookup reduces the performance of query, and we can improve the performance of query by using included column index or cover index.

I will cover few additional concepts related to the optimal method in another article.

Reference : Pinal Dave (http://blog.sqlauthority.com)

Related Post:

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3

SQL Server never stops to amaze me. As regular readers of this blog already know that besides conducting corporate training, I work on large-scale projects on query optimizations and server tuning projects. In one of the recent projects, I have noticed that a Junior Database Developer used the query hint Force Order; when I asked for details, I found out that the basic concept was not properly understood by him.

Today, let us try to understand its working and the effect of this hint. Further, we will see the extent of difference in performance created by this one query hint. I also have one interesting question for all of you as well; I will give the answer in one of my later posts.

USE AdventureWorks
GO
SELECT COUNT(*) CountEmployee -- 290 Rows
FROM HumanResources.Employee
GO
SELECT COUNT(*) CountEmployeeAddress -- 290 Rows
FROM HumanResources.EmployeeAddress
GO
SELECT COUNT(*) CountEmployeeDepartmentHistory -- 296 Rows
FROM HumanResources.EmployeeDepartmentHistory
GO
SELECT *
FROM HumanResources.Employee e
INNER JOIN HumanResources.EmployeeAddress ea ON ea.EmployeeID = e.EmployeeID
INNER JOIN HumanResources.EmployeeDepartmentHistory edh ON edh.EmployeeID = e.EmployeeID
GO
SELECT *
FROM HumanResources.Employee e
INNER JOIN HumanResources.EmployeeAddress ea ON ea.EmployeeID = e.EmployeeID
INNER JOIN HumanResources.EmployeeDepartmentHistory edh ON edh.EmployeeID = e.EmployeeID
OPTION (FORCE ORDER)
GO


In above example, I have used three tables and their row count is listed as well. Employee and EmployeeAddress – both have same number rows, and EmployeeDepartmentHistory table has around 6 rows more than the other two tables. Now let us run the query without using OPTION (FORCE ORDER) and run it along with the query hint and check the execution plan. You will find a difference in the query cost.

We all accept that the table with least number of rows should be listed as the base table, and the same is done here. We have two such tables with the least number of rows, which are listed as base tables. Now before we further explain this, let us see the execution plan for the same.

It is very clear from the above execution plan that when I order is forced the query cost goes high. This proves that the SQL Server has already made a good decision with regard to the optimized query plan. When plan is forced in the case of the joins more than 2 table the performance matters. Let us see the execution order of the table in both the cases.

Now, from the example, it is very clear when you force order the query, it evaluates the table Employee first and when it is not forced, it evaluates EmployeeAddress first. Even though both the tables have same number of rows, the query optimizer processes them differently and uses different types of join logic. When the order is not forced, it uses hash join; further, in case of forced order, it uses nested loop – this creates a significant difference in the query cost.

The conclusion of this whole exercise is very simple.

• SQL Server Query Execution Engine is pretty smart to decide the best execution plan with least query cost for any query.
• Order of the tables in any query can make a significant impact on the query.

Now the question for you: We have seen that using query hint of OPTION (FORCE ORDER) reduces the performance; give an example wherein we can use this hint to improve the performance?

Please leave your comment here. I will publish the answer to this question with due credit and with my own example in a later post.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Recently, I was working on an optimization project for one of the large organizations. While working on one of the queries, we came across a very interesting observation. We found that there was a query on the base table and when the query was run, it used the index, which did not exist in the base table. On careful examination, we found that the query was using the index that was on another view. This was very interesting as I have personally never experienced a scenario like this. In simple words, “Query on the base table can use the index created on the indexed view of the same base table.”

If you have skipped the first paragraph, I suggest you go over the first paragraph one more time and try to understand the intended meaning. According to me, this is a very interesting observation and we rarely come across such an instance. I have made very small attempt to recreate this particular behavior.

USE tempdb
GO
-- Create Table
CREATE TABLE IndexViewAggr (ID INT)
GO
-- Insert One Hundred Thousand Records
INSERT INTO IndexViewAggr (ID)
SELECT TOP 100000 ROW_NUMBER() OVER (ORDER BY a.name) RowID
FROM sys.all_objects a
CROSS JOIN sys.all_objects b
GO
-- Create View on the table
CREATE VIEW ViewAggr WITH SCHEMABINDING
AS
SELECT ID, COUNT_BIG(*) CountID
FROM dbo.IndexViewAggr
WHERE ID BETWEEN 1 AND 1000
GROUP BY ID
GO
-- Following query is very identical to View
-- Run the query on base table
-- Please note that in execution plan there is Table Scan
SELECT ID, COUNT_BIG(*) CountID
FROM dbo.IndexViewAggr
WHERE ID BETWEEN 1 AND 1000
GROUP BY ID
GO
-- Create Index on the View
-- Please note that this index is on View and not on table
CREATE UNIQUE CLUSTERED INDEX IndexView ON ViewAggr(ID)
GO
-- Run the query on base table again
-- Please note that in execution plan there is Index Scan
SELECT ID, COUNT_BIG(*) CountID
FROM dbo.IndexViewAggr
WHERE ID BETWEEN 1 AND 1000
GROUP BY ID
GO
-- Please note that there are difference in "Query Cost"
-- Clean up Database
DROP VIEW ViewAggr
GO
DROP TABLE IndexViewAggr
GO

Now let us quickly observe two performance comparison of two of the selects. One object is created before the creation of the index over the view, and the other object is created after the creation of the index over the view. Please note that we are talking about two different objects here. Let me explain this schematically.

This is a very interesting behavior as the index created on view affects the query that runs on the base table. Let me show you the execution plan of the query before and after index creation.

From this example, it is very clear that there is a very big difference between the query costs when the index is created on the table. In fact, there is a very simple explanation of this behavior in the white paper Improving Performance with SQL Server 2008 Indexed Views.

The reason for this behavior is that the query optimizer works when it checks for the index on view and finds that the execution plan of the query is similar to that of the view.

Let me know your thoughts about this article. I do understand that this concept is slightly complicated. Further, this type of behavior is rarely encountered in real world. I hope that this article will give you a better insight to such a situation.

Reference : Pinal Dave (http://blog.sqlauthority.com)

Posted in Pinal Dave, SQL, SQL Authority, SQL Index, SQL Optimization, SQL Performance, SQL Puzzle, SQL Query, SQL Scripts, SQL Server, SQL Tips and Tricks, SQL White Papers, SQLServer, T SQL, Technology ]]> http://blog.sqlauthority.com/2009/09/24/sql-server-interesting-observation-index-on-index-view-used-in-similar-query/feed/ 14 pinaldave http://blog.sqlauthority.com/2009/09/17/sql-server-converting-stored-procedure-into-table-valued-function/ http://blog.sqlauthority.com/2009/09/17/sql-server-converting-stored-procedure-into-table-valued-function/#comments Thu, 17 Sep 2009 01:30:27 +0000 pinaldave http://blog.sqlauthority.com/?p=6798 ]]>

In one of my recent articles, I mentioned the use of Table Valued Function (TVF) instead of Stored Procedure (SP). I received a follow up email asking what type of SP can be converted into a TVF. This is indeed a very interesting question! In fact, not all the SPs qualify to be converted to a TVF.

Please note that I am not encouraging to convert all the SPs to TVFs. Each SPs have their own usage and need. Here, I shall discuss about the type of SP that can be converted to a TVF.

First of all, you need to convert the SP that you are using to retrieve data and to insert it into any other table. Please read my original post wherein I have explained the scenario for the same. If you do not have to insert the data from the SP into another table, then there is no need to convert the SP to a TVF. This conversion will not necessarily enhance the performance.

In general, SP that returns only one resultset qualifies for conversion to TVF. If there are more than one resultset, then the SP cannot be converted to TVF as the TVF always returns one resultset. Moreover, if the SP has DML statements such as DELETE, UPDATE, INSERT or MERGE, it cannot be converted to a TVF. The usage of TVF is limited to the SELECT statement only. If the SP must be executed by using EXEC, it cannot be converted to TVF.

Do you know any other reason why a SP needs to be converted to a TVF?

Reference: Pinal Dave (http://blog.sqlauthority.com)

Today we have a very interesting subject to look at. I tried to look for help online but have not found any other documentation besides what we have from the Book Online.

Let us try to understand what are the different kinds of hints available in SQL Server and how they are helpful.

What is a Hint?

Hints are options and strong suggestions specified for enforcement by the SQL Server query processor on DML statements. The hints override any execution plan the query optimizer might select for a query.

Before we continue to explore this subject, we need to consider one very important fact and say some words of caution. SQL Server Query optimizer is a very smart tool and it makes a best selection of execution plan. Suggesting hints to the Query Optimizer should be attempted when absolutely necessary and by experienced developers who know exactly what they are doing (or in development as a way to experiment and learn).

There are three different kinds of hints. Let us understand the basics of each of them separately.

Please continue reading article here.

Reference : Pinal Dave (http://blog.sqlauthority.com)

How to improve performance of SQL Server Queries is a common topic of discussion among many of us. Much has been said, much has been discussed. Few days back, I had an interesting discussion with one of the Junior developers regarding performance improvement of SQL Server Queries. We discussed on how by using a separate hard drive for several database objects can right away improve performance. I suggested him that non clustered index and tempdb can be created on a separate disk to improve performance.

No sooner had I given my suggestion than I received a question – What will happen if we can create clustered index on a separate drive from the table on which it is built.

My answer is : No! It is not possible at all.

Let us first be clear about the difference between a clustered and a non clustered index.

Clustered Index

• Only 1 allowed per table
• Physically rearranges data in the table to conform to the index constraints
• For use on columns that are frequently searched for ranges of data
• For use on columns with low selectivity

Non-Clustered Index

• Up to 249 (for SQL Server 2005) and 999 (for SQL Server 2008) allowed per table
• Creates a separate list of key values with pointers to the location of the data in the data pages
• For use on columns that are searched for single values
• For use on columns with high selectivity

A table devoid of primary key index is called heap, and here data is not arranged in a particular order, which gives rise to issues that adversely affect performance. Data must be stored in some kind of order. If we put clustered index on it then the order will be forced by that index and the data will be stored in that particular order.

Reference : Pinal Dave (http://blog.sqlauthority.com)

While cleaning up my bookmarks this week, I stumbled upon a very small interesting thing. I can proudly call myself a pro at finding stuffs, but after continuously hunting online I could not gather comprehensive information about this topic. I was actually looking for a practical example for Query Optimizer Hint “ROBUST PLAN”. Before I seek help from you, let us first try to understand what query optimizer hints is and then we will move on to the concept of “ROBUST PLAN”.

To put it simply, Query hints is a T-SQL clause which on running directs T-SQL query to run in a particular manner using suggested index, joins or other logic. Query hints can force the kind of execution plan SQL Server needs. In general, SQL Server query optimizer selects the best execution plan itself to run any query; it is a good idea to just keep it at its default setting.

“ROBUST PLAN” is a kind of query hint which works quite differently than other hints. It does not improve join or force any indexes to use; it just makes sure that a query does not crash due to over the limit size of row. Let me elaborate upon it. In a table, there is an outsized row that crashes any operation going over it. T-SQL has specific query hint where query engine ignores such an outsized row which will potentially crash it and stop from working. Using ROBUST PLAN, query optimizer can be forced to ignore such errors that are generated because of outsized rows. This usually does not happen with SQL Server 2005 because of its tremendous capacity of row size (2 GB). However, here we are not talking about row size beyond the limit; we are talking about SQL operation that considers the given row to be too large for this operation, even if the row is in MB or less.

Now, after having thoroughly gone through the above two explanations, I request you all to find out which operation can generate this error and compel us to use ROBUST PLAN hint. I am disappointed after my extensive online search. I could not find a single example or information by anyone where they have suggested that they needed this hint. According to me, this is a useful feature which can be used wisely, as this hint reduces the performance as well in some cases. For queries where the entire execution is not working, this is at least a better option where you can expect to obtain some answer.

I am looking for a query which runs fine with option “ROBUST PLAN” but without it the query fails to produce any result and generates error.

If you know the answer, then please share it with me. I will post your answer here with due credit.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

Of late, I have been using Twitter quite frequently, and I am gradually discovering its usefulness. I received a Direct Message (or DM in terms of twitter) asking if I can comment on the effect of normalization on Index and its performance in one twit! Now honestly speaking, this was new for me. I never expected to be quizzed like this. If you are using Twitter, then you must be aware that one twit contains only 140 characters. I was supposed to give answer on such a big subject in just 140 letters. An interesting fact is that normalization and Index are not really closely related. The right question should have been – what is the effect of normalization on performance?

In any case, let us go over my twit answer.

“Normalization increases count of table along with clustered index which makes every single entity with its independent index improving perf.”

Within seconds of my posting the above twit I received a reply from  Mladen Prajdic – “which you loose on joins :)”

This whole scenario is very intriguing. I will try to give a comprehensive explanation. Prior to normalization, usually data is in a large table. As it gets normalized, all the redundant information is removed and data is categorized and placed in their respective tables. It is a common practice to have PK in tables that leads to additional clustered indexes on the database. This means normalized database has more clustered indexes. As there is no redundancy, each table contains a unique entry for data. Because of this all the clustered indexes have fewer rows, which effectively increase the speed of database operations. Besides, as there is no redundancy, normalized database requires less storage and maintenance. Now when query is intricate and involves numerous tables, lots of joins are introduced. These joins effectively diminish performance. However, this loss of performance is less compared to the gain which was a result of normalization.

In case you have skipped the above paragraph and straightaway reading this sentence, I urge you to go back and pay attention to the previous paragraph. This paragraph provides a good explanation to help you understand the logical backstage theory of normalization and performance relations.

I requested Mladen Prajdic for his opinion in one twit and given below is his response. You can follow Mladen Prajdic on  Twitter here.

“@pinaldave Normalization is a data modularization concept. Indexes are data storage and retrieval concept. http://bit.ly/YMJJD” .

Let me know what you think about this subject. If possible, participate here by giving your explanation within 140 characters. Let us see what you all can come up with. Follow me on Twitter.

Reference : Pinal Dave (http://blog.sqlauthority.com)

Index Defragmentation is one of the key processes to significantly improve performance of any database. Index fragments occur when any transaction takes place in database table.  Fragmentation typically happens owing to insert, update and delete transactions. Having said that, fragmented data can produce unnecessary reads thereby reducing performance of heavy fragmented tables.

I have often been asked to share my personal Index Defragmentation Script. Well, I use Automated Index Defragmentation Script created by my friend – a SQL Expert – Michelle Ufford (a.k.a SQLFool). Michelle is a SQL Server Developer, DBA, a humble blogger, and an absolute geek! She is also the President of Eastern Iowa PASS Chapter, known as the I380 Corridor Professional Association of SQL Server. Currently, she is working with large, high volume, high performance SQL Server databases at GoDaddy.com. Her nickname might be SQLFool but honestly speaking, she is one of the best Gurus of SQL.  You can read her  blog here.

Michelle has written an excellent article about Automated Index Defragmentation Script. I recommend this wonderful write-up to all those database developers who are searching for a good solution to improve database performance. Read her article Automated Index Defragmentation Script to equip yourself with better understanding on how to improve database performance.

Here, I would to mention an interesting question taken from the above-mentioned article, which will clearly show how powerful this Automated Index Defragmentation Script is.

How long will this Automated Index Defragmentation Script  take to run?
Well, it depends. I don’t necessarily recommend running it without specifying a database; at least, not unmonitored. You *can* do that, but it could take a while. For example, to run sys.dm_db_index_physical_stats for all databases and tables, totaling 2TB, took me 4.5 hours; that doesn’t even count the actual defrags.

You can download the script from here.

Please feel free to contact Michelle or me if you have any questions and doubts regarding this script.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

Posted in Best Practices, Database, Pinal Dave, Readers Contribution, SQL, SQL Authority, SQL Index, SQL Optimization, SQL Performance, SQL Query, SQL Scripts, SQL Server, SQL Tips and Tricks, SQLAuthority Website Review, SQLServer, T SQL, Technology ]]> http://blog.sqlauthority.com/2009/04/04/sql-server-automated-index-defragmentation-script/feed/ 10 pinaldave http://blog.sqlauthority.com/2009/03/25/sqlauthority-news-ahmedabad-sql-server-user-group-meeting-review-march-21-2009/ http://blog.sqlauthority.com/2009/03/25/sqlauthority-news-ahmedabad-sql-server-user-group-meeting-review-march-21-2009/#comments Wed, 25 Mar 2009 01:30:34 +0000 pinaldave http://blog.sqlauthority.com/?p=4059 ]]>

We had fun session with Ahmedabad SQL Server Usre Group last week on March 21, 2009. It was short session but one interesting one. We discussed about how query profiler works and how to find most popular query from SQL Server instance. We had also prepared Trace Template as well query which can ran to identify longest running query along with popular query. I received nearly 10 questions after my session and lots of time was spent answering them. The whole session was very interactive.

I want to congratulate everybody who attended it, if you need my Profiler Template and Query to identify longest running query as well popular query, please let me know and I will send them to you.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

Every batch (T-SQL, SP etc) when ran creates execution plan which is stored in system for re-use. Due to this reason large number of query plans are stored in system. However, there are plenty of plans which are only used once and have never re-used again. One time ran batch plans wastes memory and resources.

SQL Server 2008 has feature of optimizing ad hoc workloads. Before we move to it, let us understand the behavior of SQL Server without optimizing ad hoc workload.

Please run following script for testing. Make sure to not to run whole batch together. Just run each command separately to really see the effect of subject of article.

Download complete script of this article here.

/* Test 0 */
/* Clean Cache and Buffers */
DBCC FREEPROCCACHE
DBCC DROPCLEANBUFFERS
GO
USE AdventureWorks
GO
/* Run Adhoc Query First Time */
SELECT * FROM HumanResources.Shift
GO
/* Check if Adhoc query is cached.
It will return one result */
SELECT usecounts, cacheobjtype, objtype, TEXT
FROM sys.dm_exec_cached_plans
CROSS APPLY sys.dm_exec_sql_text(plan_handle)
WHERE usecounts > 0 AND
TEXT LIKE '%SELECT * FROM HumanResources.Shift%'
ORDER BY usecounts DESC;
GO

Now let us check result of this script. It is clear from result that when we ran query once it cached its plan in memory. If we never run this again in future or if we have just ran as part of building longer query the cache plan of this query is just waste of memory.

Let us now enable the option of optimizing ad hoc workload. This feature is available in all the versions of SQL Server 2008.

sp_CONFIGURE 'show advanced options',1
RECONFIGURE
GO

sp_CONFIGURE ‘optimize for ad hoc workloads’,1
RECONFIGURE
GO

We will now run the code for Test 1 which is almost same as Test 0. Make sure to clean the cache and buffer before running the query to create real life scenario of live case.

/* Test 1 */
/* Clean Cache and Buffers */
DBCC FREEPROCCACHE
DBCC DROPCLEANBUFFERS
GO
USE AdventureWorks
GO
/* Run Adhoc Query First Time */
SELECT * FROM HumanResources.Shift
GO
/* Check if Adhoc query is cached.
It will not return any result */
SELECT usecounts, cacheobjtype, objtype, TEXT
FROM sys.dm_exec_cached_plans
CROSS APPLY sys.dm_exec_sql_text(plan_handle)
WHERE usecounts > 0 AND
TEXT LIKE 'SELECT * FROM HumanResources.Shift%'
ORDER BY usecounts DESC;
GO

We can clear see now as we have advance option enabled we do not have query cache planed stored in database.

We are interested to know now that if we run the batch more than 1 time it will cache its execution plan.  With advance option of optimizing ad hoc workload.

/* Test 2 */
/* Clean Cache and Buffers */
DBCC FREEPROCCACHE
DBCC DROPCLEANBUFFERS
GO
/* Run Adhoc Query two  Time */
SELECT * FROM HumanResources.Shift
GO 5
/* Check if Adhoc query is cached.
It will return result with Adhoc Query ran two times*/
SELECT usecounts, cacheobjtype, objtype, TEXT
FROM sys.dm_exec_cached_plans
CROSS APPLY sys.dm_exec_sql_text(plan_handle)
WHERE usecounts > 0 AND
TEXT LIKE '%SELECT * FROM HumanResources.Shift%'
ORDER BY usecounts DESC;
GO

From our image it is quite clear that when the batch is ran for more than 1 time it caches its execution plan. This is generic behavior with or without turning on advance option.

This may be very simple to see from the top but if you are using SQL Server 2008 and have millions of ad hoc query running every day you wil realize how important this feature is. This feature improves performance by relieving memory pressure by not storing the single time used compiled plans.

Reference : Pinal Dave (http://blog.sqlauthority.com)

More than a year ago I had written article SQL SERVER – Union vs. Union All – Which is better for performance? I have got many request to update this article. It is not fair to update already written article so I am rewriting it again with additional information.

UNION
The UNION command is used to select related information from two tables, much like the JOIN command. However, when using the UNION command all selected columns need to be of the same data type. With UNION, only distinct values are selected.

UNION ALL
The UNION ALL command is equal to the UNION command, except that UNION ALL selects all values.

The difference between Union and Union all is that Union all will not eliminate duplicate rows, instead it just pulls all rows from all tables fitting your query specifics and combines them into a table.

A UNION statement effectively does a SELECT DISTINCT on the results set. If you know that all the records returned are unique from your union, use UNION ALL instead, it gives faster results.

Run following script in SQL Server Management Studio to see the result between UNION ALL and UNION. Download complete script from here.

/* Declare First Table */
DECLARE @Table1 TABLE (ColDetail VARCHAR(10))
INSERT INTO @Table1
SELECT 'First'
UNION ALL
SELECT 'Second'
UNION ALL
SELECT 'Third'
UNION ALL
SELECT 'Fourth'
UNION ALL
SELECT 'Fifth'
/* Declare Second Table */
DECLARE @Table2 TABLE (ColDetail VARCHAR(10))
INSERT INTO @Table2
SELECT 'First'
UNION ALL
SELECT 'Third'
UNION ALL
SELECT 'Fifth'
/* Check the data using SELECT */
SELECT *
FROM @Table1
SELECT *
FROM @Table2
/* UNION ALL */
SELECT *
FROM @Table1
UNION ALL
SELECT *
FROM @Table2
/* UNION */
SELECT *
FROM @Table1
UNION
SELECT *
FROM @Table2
GO

In our example we have two tables: @Table1 and @Table2.

Now let us run UNION ALL and UNION together and see the resultset as well as Execution Plan compared to complete set of query. You can always turn on actual execution plan using CTRL+M.

We can see from the resultset of UNION ALL that it returns everything from both the table but from UNION it is very clear that only DISTINCT rows from both the table is only retrieved.

Additionally, when comparing the execution plan of UNION ALL and UNION it is also quite clear that UNION ALL is way less expensive than UNION as it does not have DISTINCT SORT operation.

Let me know what do you think about this article. If you have any suggestion for improvement please let me know and I will update articles according to that.

Reference : Pinal Dave (http://blog.SQLAuthority.com)