It is very easy to find out some basic details of any table using the following Stored Procedure.

USE AdventureWorks
GO
EXEC sp_spaceused [HumanResources.Shift]
GO

Above query will return following resultset

The above SP provides basic details such as rows, data size in table, and Index size of all the indexes on the table.

If we look at this carefully, a total of three indexes can be found on the table HumanResources.Shift.

USE AdventureWorks
GO
SELECT *
FROM sys.indexes
WHERE OBJECT_ID = OBJECT_ID('HumanResources.Shift')
GO

The above query will give result with query listing all the index on the table.

There is a small puzzle for all of you here. The puzzle is to write a query that will return the size for each index that is listed in above query. We need a query that will return an additional column in the above listed query and it should contain the size of the index. In our case, we will have three different sizes, which should add up to a total of 40 KB as shown in earlier query, where the total size is displayed.

I will publish the solution with due credit on this blog.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Script of Backup, Integrity Check and Index Optimization are the most important scripts for any developer. SQL Expert and true SQL enthusiast Ola Hallengren is known for his excellent scripts.

Please try it out and let me know what you think. The documentation is available on http://ola.hallengren.com/Documentation.html and the script can be downloaded from http://ola.hallengren.com.

Here is brief documentation sent by Ola himself for his script in his own words.

Backup Maintenance

I think that most of you have experienced the error messages “BACKUP LOG cannot be performed because there is no current database backup.” and “Cannot perform a differential backup for database “”, because a current database backup does not exist.”.

This usually happens when you have created a new database or when you have changed recovery model of a database from Simple to Full.

The consequence is that the database is not getting backed up, until a full backup (or a differential backup for the BACKUP LOG error message if that can be done) has been performed. It will also create some noise in your monitoring system.

The solution is to check if a differential or transaction log backup can be performed before doing the backup. This can be done by checking sys.master_files.differential_base_lsn and sys.database_recovery_status.last_log_backup_lsn.

The backup solution that I have developed has a parameter called @ChangeBackupType.

EXECUTE dbo.DatabaseBackup @Databases = 'USER_DATABASES', @Directory = 'C:\Backup', @BackupType = 'LOG', @ChangeBackupType = 'Y'

If it’s a new database in Full recovery model, the backup type for that database and job run will be changed to full. If it’s a database that was newly changed to Full recovery model, then the backup type for that database and job run will be changed to differential. The next time the job runs a transaction log backup will be performed for that database.

New databases start getting backed up quickly and no more “BACKUP LOG cannot be performed because there is no current database backup.” and “Cannot perform a differential backup for database “”, because a current database backup does not exist.”.

Index Optimization

The design idea is categorize all indexes based on their fragmentation level (High, Medium or Low) and whether there are columns with LOB (Large Object) data types. For each category you can define an action. The possible actions are to rebuild indexes online or offline, reorganize indexes, update statistics, reorganize indexes and update statistics or to do nothing.

Here’s an example.

EXECUTE dbo.IndexOptimize
@Databases = 'USER_DATABASES',
@FragmentationHigh_LOB = 'INDEX_REBUILD_OFFLINE',
@FragmentationHigh_NonLOB = 'INDEX_REBUILD_ONLINE',
@FragmentationMedium_LOB = 'INDEX_REORGANIZE',
@FragmentationMedium_NonLOB = 'INDEX_REORGANIZE',
@FragmentationLow_LOB = 'NOTHING',
@FragmentationLow_NonLOB = 'NOTHING',
@FragmentationLevel1 = 5,
@FragmentationLevel2 = 30,
@PageCountLevel = 1000


Indexes with a fragmentation above 30% are to be rebuilt, online if possible (no LOB columns), otherwise offline (LOB columns). Indexes with a fragmentation between 5% and 30% are to be reorganized. Indexes with a fragmentation below 5% or a size below 1000 pages are not to be touched.

If you are using partitioning IndexOptimize has a parameter, @PartitionLevel to do index rebuilds and reorganizations on the partition level. If you prefer to do sort operations in tempdb you can do that with the parameter @SortInTempdb and if you would like to set a fillfactor you can do that with the parameter @FillFactor.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Foreign Key and Check Constraints are two types of constraints that can be disabled or enabled when required. This type of operation is needed when bulk loading operations are required or when there is no need to validate the constraint. The T-SQL Script that does the same is very simple.

USE AdventureWorks
GO
-- Disable the constraint
ALTER TABLE HumanResources.Employee
NOCHECK CONSTRAINT CK_Employee_BirthDate
GO
-- Enable the constraint
ALTER TABLE HumanResources.Employee
WITH CHECK CHECK CONSTRAINT CK_Employee_BirthDate
GO

It is very interesting that when the constraint is enabled, the world CHECK is used twice – WITH CHECK CHECK CONSTRAINT. I often ask those to find the mistake in this script when they claim to know the syntax very well.

Have you ever disabled and enabled constraints in your production environment? I would like to know why you did so.

Reference: Pinal Dave (http://blog.sqlauthority.com)

This is very short note announcing details about my course details for ‘MS SQL Server 2005/2008 Query Optimization And Performance Tuning‘.

The course is scheduled to be delivered from 19th Nov, 2009 and 21st Nov, 2009 in Pune, India.

This three day course is an intensive course designed to give attendees an in-depth look at the query optimization and performance tuning concepts and methods found in SQL Server 2005/2008. This course is designed to prepare the SQL Server developers and administrators for a transition to SQL Server 2005/2008 while discussing best practices for a variety of topics.

Day 1 : Strengthen the basics along with SQL Server 2005/2008 New Features

Module 01: Subqueries, Ranking Functions, Joins and Set Operations
Module 02: Table Expressions
Module 03: TOP and APPLY
Module 04: SQL Server 2008 Enhancements

Day 2: Query Optimization & Performance Tuning 1

Module 05: Logical Query Processing
Module 06: Query Tuning
Module 07:  Introduction to the Query Processor
Module 08:  Review of common query coding which causes poor performance

Day 3: Query Optimization & Performance Tuning 2

Module 09:  SQL Server Indexing and index maintenance
Module 10:  Plan Guides, query hints, UDFs, and Computed Columns
Module 11:  Understanding SQL Server Execution Plans
Module 12: Real World Index and Optimization Tips

The space are limited and I encourage you to register as soon as possible.

Call to register – +91 98817 95739, +91 98508 43664, +91 99229 33319

Email to register – workshop “at” seedinfotech.com

If due to any reason, you can not attend this course, please call or email on details above for future schedule for the course and location.

Click to Enlarge

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Recently, I have been working on Query Optimization project. While working on it, I found the following interesting observation. This entire concept may appear very simple, but if you are working in the area of query optimization and server tuning, you will find such useful hints.

Before we start, let us understand the difference between Seek Predicate and Predicate. Seek Predicate is the operation that describes the b-tree portion of the Seek. Predicate is the operation that describes the additional filter using non-key columns. Based on the description, it is very clear that Seek Predicate is better than Predicate as it searches indexes whereas in Predicate, the search is on non-key columns – which implies that the search is on the data in page files itself.

Earlier, I had posted regarding how to remove key lookup and bookmark lookup in the following:

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3

Let us see an example where we will remove the bookmark lookup first using the covering index. On removing the bookmark lookup, it resulted in Index Scan, which is not good for performance. When Index Scan is converted to Index Seek, it provides  a significant improvement in performance.

Run following SELECT, which is based on the database AdventureWorks.

USE AdventureWorks
GO
-- CTRL + M Enforces Key Lookup
-- Try 1
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = 14417807
GO

Let us check the execution plan for the same. The execution plan consists of the key lookup because there are columns that we are trying to retrieve in SELECT as well in WHERE clause.

Let us create a covering index on this table HumanResources.Employee.

-- Create Non clustered Index
CREATE NONCLUSTERED INDEX [IX_HumanResources_Employee_Example] ON HumanResources.Employee
(
NationalIDNumber ASC, HireDate, MaritalStatus
) ON [PRIMARY]
GO

After creating above index, let us run the same SELECT statement again.

-- CTRL + M Removes Key Lookup, but it still enforces Index Scan
-- Try 2
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = 14417807
GO

Due to non-clustered index, Key Lookup is removed along with Nested Loops but Index Scan still remains, and this is not good for performance.

I tried to remove the scan, but my attempts were unsuccessful. I finally looked at the datatype of the NationalIDNumber. All this time, I was assuming that this datatype is INT, but on a careful check, I found that the datatype of NationalIDNumber is nvarchar(15).

In the SELECT statement, we were comparing the datatype of NVARCHAR to INT, and this was forcing the predicate operation while executing the query.

As discussed earlier, due to predicate operation, there has to be explicit conversion on the side of NationalIDNumber, which forces the query optimizer to not use the index and instead it has to scan the complete data in table to get the necessary data. This is not the desired solution. Index Scan reduces performance. The reason for this conversion is because I am using INT value in WHERE clause instead of NVARCHAR.

I changed my WHERE clause and passed STRING as the parameter instead of INT.

-- CTRL + M Removes Key Lookup and it enforces Index Seek
-- Try 3
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = '14417807'
GO

After running the query with the changed WHERE clause, the Index Scan is now converted into Index Seek.

Index Seek is definitely the most optimal solution in this particular scenario.

When the detail execution plan was checked, I found the following two notable points. First, the predicate is converted to seek predicate, which is the reason for performance improvement, as described earlier. Instead of scanning data in the table, Index Seek is performed. Second, as the datatype of the NationalIDNumber is NVARCHAR and the parameters are passed as VARCHAR, the conversion happens on the parameters instead of NationalIDNumber column, and this forces Index Scan to Index Seek.

If we pass the parameter as NVARCHAR instead of VARCHAR, the execution plan will remain the same, but CONVERT_IMPLICIT will not be required any more. Let us run the following query, which has NVARCHAR as the parameter.

-- CTRL + M Removes Key Lookup and it enforces Index Seek and no CONVERT_IMPLICIT
-- Try 4
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = N'14417807'
GO

The execution plan of the above query is very similar to that in which we had passed the parameter as VARCHAR.

Now let us check the execution plan for the same.

In the WHERE condition, the operators we have on both the sides of “=” are of NVARCHAR. NationalIDNumber and parameter passed – both are NVARCHAR, which has removed CONVERT_IMPLICIT. However, there are no changes in the execution plan.

In summary, when Key Lookup is removed and index seek replaces index scan, the performance is tuned up. Let us quickly compare the execution plan of the above four options. I have included the complete code here for easy reference.

USE AdventureWorks
GO
-- CTRL + M Enforces Key Lookup
-- Try 1
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = 14417807
GO
-- Create Non clustered Index
CREATE NONCLUSTERED INDEX [IX_HumanResources_Employee_Example] ON HumanResources.Employee
(
NationalIDNumber ASC, HireDate, MaritalStatus
) ON [PRIMARY]
GO
--WAITFOR DELAY '00:00:30'
-- CTRL + M Removes Key Lookup, but it still enforces Index Scan
-- Try 2
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = 14417807
GO
-- CTRL + M Removes Key Lookup and it enforces Index Seek
-- Try 3
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = '14417807'
GO
-- CTRL + M Removes Key Lookup and it enforces Index Seek and no CONVERT_IMPLICIT
-- Try 4
SELECT NationalIDNumber, HireDate, MaritalStatus
FROM HumanResources.Employee
WHERE NationalIDNumber = N'14417807'
GO
/* What is the reason for difference between Try 2 and Try 3?
Check the exeuction plan
*/
-- Drop Index
DROP INDEX [IX_HumanResources_Employee_Example] ON HumanResources.Employee WITH ( ONLINE = OFF )
GO

Let us look at the execution plan.

Summary points:

• In general, Index Seek is better than Index Scan (I am not talking about it depends conditions)
• Understand Predicates and Seek Predicates and see if you have only Seek Predicates.
• In case of key lookup or bookmark lookup, see if you can create a covering index or included column index.
• Use the datatype wisely even though there is no change in the resultset.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Many times I have seen that the index is disabled when there is large update operation on the table. Bulk insert of very large file updates in any table using SSIS is usually preceded by disabling the index and followed by enabling the index. I have seen many developers running the following query to disable the index.

USE AdventureWorks
GO
----Diable Index
ALTER INDEX [IX_StoreContact_ContactTypeID] ON Sales.StoreContact DISABLE
GO

While enabling the same index, I have seen developers using the following INCORRECT syntax, which results in error.

USE AdventureWorks
GO
----INCORRECT Syntax Index
ALTER INDEX [IX_StoreContact_ContactTypeID] ON Sales.StoreContact ENABLE
GO

Msg 102, Level 15, State 1, Line 1
Incorrect syntax near ‘ENABLE’.

This is because once the index is disabled, it cannot be enabled, but it must be rebuilt. The following syntax will enable and rebuild the index with optimal performance.

USE AdventureWorks
GO
----Enable Index
ALTER INDEX [IX_StoreContact_ContactTypeID] ON Sales.StoreContact REBUILD
GO

I hope that now you have understood why enabling this syntax in the index throws an error and also how to enable an index with optimal performance.

If you want to read just the syntax, I have written a similar article earlier SQL SERVER – Disable Index – Enable Index – ALTER Index.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Community TechDays at Ahmedabad was a great successful event. In fact, this can be considered the biggest event held in Ahmedabad thus far along with the community. I have posted a detail report of the same at SQLAuthority News – Community TechDays in Ahmedabad – A Successful Event. After the event, I received many emails requesting the comic strip I had shown in my presentation.

I had presented two different comic strips: one before the presentation and the other after the presentation. Both the comic strips are shown here; if you click on the images, you can see the larger version. Let me know what you think of them. If you are interested in the whole session of “The other side of Index,” let me know so that I can present the same at your UG event as well.

Before Presentation

Click on the image to see the enlarged version

After Presentation

Click on the image to see the enlarged version

Let me know what you think about these.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Earlier I have written two different articles on the subject Remove Bookmark Lookup. This article is as part 3 of original article. Please read first two articles here before continuing reading this article.

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

We read in above articles that we can remove bookmark lookup using covering index. Covering Index is the index, which contains all the columns used in SELECT as well in JOINs and WHERE conditions. In our example we have created clustered index first.

-- Create Clustered Index
CREATE CLUSTERED INDEX [IX_OneIndex_ID] ON [dbo].[OneIndex]
(
[ID] ASC
) ON [PRIMARY]
GO

Based on clustered index we have created following non clustered index. Please note that we do not have to create both the index together. We can create either covering index or included column index along with it. Please note, I am suggesting to create either of it, not both of it.

In earlier article I have suggested to include all columns but in reality in any non clustered index there is no need to include columns included in clustered index. All non clusteredindex automatically contains pointers to clustered index any way.

We should create index described in earlier article as following.

Method 1: Creating covering non-clustered index.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Cover] ON [dbo].[OneIndex]
(
City, FirstName
) ON [PRIMARY]
GO

Method 2: Creating included column non-lustered index.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Include] ON [dbo].[OneIndex]
(
City
) INCLUDE (FirstName) ON [PRIMARY]
GO

Let us examine the execution plan and compare the query costs and also verify that if both the index usages are forcing index seek instead of index scan.

As discussed in the example, any non clustered index does not need to include columns which are included in the clustered index.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Related Post:

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3

This article is follow up of my previous article SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup. Please do read my previous article before continuing further.

I have described there two different methods to reduce query execution cost. Let us compare the performance of the SELECT statement of the previous query.

We have created two different indexes on the table.

Method 1: Creating covering non-clustered index.

In this method, we will create a non-clustered index that contains the columns used in the SELECT statement along with the column used in the WHERE clause.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Cover] ON [dbo].[OneIndex]
(
City, FirstName, ID
) ON [PRIMARY]
GO

Method 2: Creating included column non-lustered index.

In this method, we will create a nonclustered index that includes the columns used in the SELECT statement along with the column used in the WHERE clause. Here, we will use new syntax introduced in SQL Server 2005. An index with included nonkey columns can significantly improve query performance when all the columns in the query are included in the index.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Include] ON [dbo].[OneIndex]
(
City
) INCLUDE (FirstName,ID) ON [PRIMARY]
GO

Let us compare the performance of our query by running the following query with two different indexes.

SELECT ID, FirstName
FROM OneIndex WITH (INDEX(IX_OneIndex_Cover))
WHERE City = 'Las Vegas'
GO
SELECT ID, FirstName
FROM OneIndex WITH (INDEX(IX_OneIndex_Include))
WHERE City = 'Las Vegas'
GO

Let us examine the execution plan and compare the query costs.

In fact, in this example, the performance of both the queries is quite same. In this case, we can use either of the query and obtain the same performance.

I have mentioned in my previous article that I prefer the Method 2. The reason is that method 2 has many advantages over method 1.

1) Index can exceed the 900-byte limitation of the index key.

2) Index can include datatypes that are not allowed as key columns – varchar(max), nvarchar(max) or XML.

3) Size of the key index can be reduced, which improves the overall performance of the index operation.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Related Post:

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3

Today, I would like to share one very quick tip about how to remove bookmark lookup or RID lookup. Let us first understand Bookmark lookup or RID lookup. Please note that from SQL Server 2005 SP1 onwards, Bookmark look up is known as Key look up.

When a small number of rows are requested by a query, the SQL Server optimizer will try to use a non-clustered index on the column or columns contained in the WHERE clause to retrieve the data requested by the query. If the query requests data from columns not present in the non-clustered index, SQL Server must go back to the data pages to get the data in those columns. Even if the table has a clustered index or not, the query will still have to return to the table or clustered index to retrieve the data.

In the above scenario, if table has clustered index, it is called bookmark lookup (or key lookup); if the table does not have clustered index, but a non-clustered index, it is called RID lookup. This operation is very expensive. To optimize any query containing bookmark lookup or RID lookup, it should be removed from the execution plan to improve performance. There are two different ways to remove bookmark/RID lookup.

Before we understand these two methods, we will create sample table without clustered index and simulate RID lookup. RID Lookup is a bookmark lookup on a heap that uses a supplied row identifier (RID).

USE tempdb
GO
-- Create Table OneIndex with few columns
CREATE TABLE OneIndex (ID INT,
FirstName VARCHAR(100),
LastName VARCHAR(100),
City VARCHAR(100))
GO
-- Insert One Hundred Thousand Records
INSERT INTO OneIndex (ID,FirstName,LastName,City)
SELECT TOP 100000 ROW_NUMBER() OVER (ORDER BY a.name) RowID,
'Bob',
CASE WHEN ROW_NUMBER() OVER (ORDER BY a.name)%2 = 1 THEN 'Smith'
ELSE 'Brown' END,
CASE
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%1000 = 1 THEN 'Las Vegas'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 1 THEN 'New York'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 5 THEN 'San Marino'
WHEN ROW_NUMBER() OVER (ORDER BY a.name)%10 = 3 THEN 'Los Angeles'
ELSE 'Houston' END
FROM sys.all_objects a
CROSS JOIN sys.all_objects b
GO

Now let us run following select statement and check the execution plan.

SELECT ID, FirstName
FROM OneIndex
WHERE City = 'Las Vegas'
GO

As there is no index on table, scan is performed over the table. We will create a clustered index on the table and check the execution plan once again.

-- Create Clustered Index
CREATE CLUSTERED INDEX [IX_OneIndex_ID] ON [dbo].[OneIndex]
(
[ID] ASC
) ON [PRIMARY]
GO

Now, run following select on the table once again.

SELECT ID, FirstName
FROM OneIndex
WHERE City = 'Las Vegas'
GO

It is clear from execution plan that as a clustered index is created on the table, table scan is now converted to clustered index scan. In either case, base table is completely scanned and there is no seek on the table.

Now, let us see the WHERE clause of our table. From our basic observation, if we create an index on the column that contains the clause, a performance improvement may be obtained. Let us create non-clustered index on the table and then check the execution plan.

-- Create Index on Column City As that is used in where condition
CREATE NONCLUSTERED INDEX [IX_OneIndex_City] ON [dbo].[OneIndex]
(
[City] ASC
) ON [PRIMARY]
GO

After creating the non-clustered index, let us run our select statement again and check the execution plan.

SELECT ID, FirstName
FROM OneIndex
WHERE City = 'Las Vegas'
GO

As we have an index on the WHERE clause, the SQL Server query execution engine uses the non-clustered index to retrieve data from the table. However, the columns used in the SELECT clause are still not part of the index, and to display those columns, the engine will have to go to the base table again and retrieve those columns. This particular behavior is known as bookmark lookup or key lookup.

There are two different methods to resolve this issue. I have demonstrated both the methods together; however, it is recommended that you use any one of these methods for removing key lookup. I prefer Method 2.

Method 1: Creating non-clustered cover index.

In this method, we will create non-clustered index containing the columns, which are used in the SELECT statement, along with the column which is used in the WHERE clause.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Cover] ON [dbo].[OneIndex]
(
City, FirstName, ID
) ON [PRIMARY]
GO

Once the above non-clustered index, which covers all the columns in query, is created, let us run the following SELECT statement and check our execution plan.

SELECT ID, FirstName
FROM OneIndex
WHERE City = 'Las Vegas'
GO

From the execution plan, we can confirm that key lookup is removed the only index seek is happening. As there is no key lookup, the SQL Server query engine does not have to go to retrieve the data from data pages and it obtains all the necessary data from index itself.

Method 2: Creating an included column non-clustered index.

Here, we will create non-clustered index that also includes the columns, which are used in the SELECT statement, along with the column used in the WHERE clause. In this method, we will use new syntax introduced in SQL Server 2005. An index with included nonkey columns can significantly improve query performance when all columns in the query are included in the index.

CREATE NONCLUSTERED INDEX [IX_OneIndex_Include] ON [dbo].[OneIndex]
(
City
) INCLUDE (FirstName,ID) ON [PRIMARY]
GO

From the execution plan, it is very clear that this method  also removes the key lookup as well.

In summary, Key lookup, Bookmark lookup or RID lookup reduces the performance of query, and we can improve the performance of query by using included column index or cover index.

I will cover few additional concepts related to the optimal method in another article.

Reference : Pinal Dave (http://blog.sqlauthority.com)

Related Post:

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 2

SQL SERVER – Query Optimization – Remove Bookmark Lookup – Remove RID Lookup – Remove Key Lookup – Part 3

SQL Server never stops to amaze me. As regular readers of this blog already know that besides conducting corporate training, I work on large-scale projects on query optimizations and server tuning projects. In one of the recent projects, I have noticed that a Junior Database Developer used the query hint Force Order; when I asked for details, I found out that the basic concept was not properly understood by him.

Today, let us try to understand its working and the effect of this hint. Further, we will see the extent of difference in performance created by this one query hint. I also have one interesting question for all of you as well; I will give the answer in one of my later posts.

USE AdventureWorks
GO
SELECT COUNT(*) CountEmployee -- 290 Rows
FROM HumanResources.Employee
GO
SELECT COUNT(*) CountEmployeeAddress -- 290 Rows
FROM HumanResources.EmployeeAddress
GO
SELECT COUNT(*) CountEmployeeDepartmentHistory -- 296 Rows
FROM HumanResources.EmployeeDepartmentHistory
GO
SELECT *
FROM HumanResources.Employee e
INNER JOIN HumanResources.EmployeeAddress ea ON ea.EmployeeID = e.EmployeeID
INNER JOIN HumanResources.EmployeeDepartmentHistory edh ON edh.EmployeeID = e.EmployeeID
GO
SELECT *
FROM HumanResources.Employee e
INNER JOIN HumanResources.EmployeeAddress ea ON ea.EmployeeID = e.EmployeeID
INNER JOIN HumanResources.EmployeeDepartmentHistory edh ON edh.EmployeeID = e.EmployeeID
OPTION (FORCE ORDER)
GO


In above example, I have used three tables and their row count is listed as well. Employee and EmployeeAddress – both have same number rows, and EmployeeDepartmentHistory table has around 6 rows more than the other two tables. Now let us run the query without using OPTION (FORCE ORDER) and run it along with the query hint and check the execution plan. You will find a difference in the query cost.

We all accept that the table with least number of rows should be listed as the base table, and the same is done here. We have two such tables with the least number of rows, which are listed as base tables. Now before we further explain this, let us see the execution plan for the same.

It is very clear from the above execution plan that when I order is forced the query cost goes high. This proves that the SQL Server has already made a good decision with regard to the optimized query plan. When plan is forced in the case of the joins more than 2 table the performance matters. Let us see the execution order of the table in both the cases.

Now, from the example, it is very clear when you force order the query, it evaluates the table Employee first and when it is not forced, it evaluates EmployeeAddress first. Even though both the tables have same number of rows, the query optimizer processes them differently and uses different types of join logic. When the order is not forced, it uses hash join; further, in case of forced order, it uses nested loop – this creates a significant difference in the query cost.

The conclusion of this whole exercise is very simple.

• SQL Server Query Execution Engine is pretty smart to decide the best execution plan with least query cost for any query.
• Order of the tables in any query can make a significant impact on the query.

Now the question for you: We have seen that using query hint of OPTION (FORCE ORDER) reduces the performance; give an example wherein we can use this hint to improve the performance?

Please leave your comment here. I will publish the answer to this question with due credit and with my own example in a later post.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Community Tech Days are here Tomorrow in Ahmedabad on Oct 3, 2009. I will be presenting the session ‘SQL Server 2008 – The Other Side of Index’. I will be available there whole day if you want to meet and discuss SQL. I will be starting my session with following cartoon. You will have to attend the session in person to see what I am going to cover in the session.

I suggest you do not miss session of Vinod Kumar and Jacob Sebastian. They are technology rock stars. You can read about them in following posts.

SQLAuthority News – Community Tech Days – Oct 3, 2009 – SQL Server 2008 – The Other Side of Index

SQLAuthority News – Community Tech Days Oct 3, 2009 – Ahmedabad

Reference: Pinal Dave (http://blog.sqlauthority.com)

Posted in Pinal Dave, SQL, SQL Authority, SQL Humor, SQL Index, SQL Query, SQL Server, SQL Tips and Tricks, SQL User Group, SQLAuthority News, SQLServer, T SQL, Technology ]]> http://blog.sqlauthority.com/2009/10/02/sqlauthority-news-sql-server-2008-the-other-side-of-index-live-presentation-in-ahmedabad/feed/ 4 pinaldave http://blog.sqlauthority.com/2009/09/30/sqlauthority-news-community-tech-days-oct-3-2009-sql-server-2008-the-other-side-of-index/ http://blog.sqlauthority.com/2009/09/30/sqlauthority-news-community-tech-days-oct-3-2009-sql-server-2008-the-other-side-of-index/#comments Wed, 30 Sep 2009 01:30:48 +0000 pinaldave http://blog.sqlauthority.com/?p=6924 ]]>

Microsoft Community Tech Days are here! Dive deep into the world of Microsoft technologies at the Community TechDays and get trained on the latest from Microsoft.

Community Tech Days are coming to Ahmedabad on Oct 3, 2009. I will be presenting the session ‘SQL Server 2008 – The Other Side of Index’. I will be talking about the other side of Index where we will be thinking out of the typical way of creating Indexes. Take a look at the following common conversation.

Person 1: My Query is running slow.
Person 2: How about create an index on it?
Person 1: Good point!

Every time I listen to such conversations, I sense a lack of complete understanding on how the Index affects performance. I will be starting the session with the above mentioned conversation and will continue to explain how index and performance are related. We will not talk about the regular creation of index but will explore the different sides of Index. We will also discuss about common concepts that are related to index, such as deadlock, execution plan, indexed view, and few other issues. I assure you that this whole session will be a two-way interaction between the attendees and me.

Again, we will have a lot of interesting demos, examples and conversations. If you have not registered for this event, I suggest that you do the same right away.

Please note that there is no video recording of this event or live telecast. This event should be attended in person.

I am not only attending this event to deliver my tech session, but also to listen to two of my colleagues.

I am an avid listener of Vinod Kumar’s sessions and I make sure that I do not miss any of them. I always feel that he is capable of instilling his passion in all who listen. He is well known for his community site ExtremeExperts.com and also is the top evangelist from Microsoft. He will be presenting two excellent sessions. I am looking forward to listen both the sessions – Tips and Tricks with Office 2010 and Windows 7 – A Dream Operating System for IT Professionals.

Jacob Sebastian is an SQL Server MVP and also my friend. We work together on all the community-related aspects. We are leaders of Ahmedabad SQL Server User Group and Gandhinagar SQL Server User Group. Jacob is going to speak at PASS Global Summit in November 2009. He will be presenting an excellent session on the Best Practices for Exception Handling and Defensive Programming in Microsoft SQL Server.

I insist all of you to register for this event to make the maximum use of it. We can meet after the session and talk about some good SQL.

Reference : Pinal Dave (http://blog.sqlauthority.com)

Working with SQL Server has never seems to be monotonous – no matter how long one has worked with it. Quite often, I come across some excellent comments that I feel like acknowledging them as blog posts. Recently, I wrote an article on SQL SERVER – Execution Plan and Results of Aggregate Concatenation Queries Depend Upon Expression Location, which is well received in community.

Before you read this article further, I request you to read original article. I received very interesting comments from Bob on the blog, where he explained why this is happening. Further, he talked about a similar kind of example. Let us first read his comment, and then we will execute his example and see the resultset.

Comment from Bob

pinaldave,

If you want to see what’s going on here, I think you need to shift your point of view from an implementation-centric view to an ANSI point of view. ANSI does not guarantee processing order. Figure 2 is interesting, but it will be potentially misleading if you don’t understand the ANSI rule-set SQL Server operates under in most cases. Implementation thinking can certainly be useful at times when you really need that multi-million row query to finish before the backups fire off, but in this case, it’s counterproductive to understanding what is going on.

First off, your statements all return a single row, which were properly sorted according to your ORDER BY expression; which is to say that there was really no sorting to be done because only a single row is returned. So let’s forget about the ORDER BY. Forget that the execution plan shows 2 rows being sorted — that’s implementation specific. A different vendor’s product could have an optimizer smart enough to see that only a single row can ever be returned from this query and therefore, avoid the unnecessary sort altogether.

Secondly, you are not guaranteed that the string concatenation will occur in any particular order. The ORDER BY clause only pertains to the final results from an ANSI perspective (at least, for these queries). MS SQL Server is right in line with ANSI and will not guarantee you anything different. Don’t expect to get AB in every case.

Finally, SQL works using relational sets, so the construct varchar = varchar + varchar forms a relational set that is evaluated accordingly. I have seen this type of construct used many times and it works sometimes – until it doesn’t. This is why you get only B in one example where I suspect you wanted AB. After thinking about it a bit, I built this example which I hope is just different enough to show you the same mechanism is at work here:
DECLARE @sum INT
SELECT @sum = 0
SELECT @sum = @sum + num FROM (
SELECT 1 [num] UNION SELECT 2 [num]
) [a] ORDER BY (num * num)
SELECT @sum [Is it 3]

I tried to show two things here. The first is that the ORDER BY isn’t doing what you think it’s doing with respect to the order of concatenation in your query. The order here is nonsensical. The second thing I think the example shows is that 1+2 doesn’t equal 3 with this @ = @ + col construct in all cases. It’s a bad construct to use. Remove the ORDER BY and you get 3, leave it in and you get 2. Heck, someone else could get 1 in theory. There’s a case I see all the time where this is used, but I can’t quite recall it. I’ll post it if I think of it again.

I must admit that he has made excellent point and explained the concept very well as well. Just for clarity, we will see the implementation of the example mentioned by him to understand how it works.

DECLARE @sum INT
SELECT @sum = 0
SELECT @sum = @sum + num FROM (
SELECT 1 [num] UNION SELECT 2 [num]
) [a] ORDER BY (num * num)
SELECT @sum WithOrderBy
GO
DECLARE @sum INT
SELECT @sum = 0
SELECT @sum = @sum + num FROM (
SELECT 1 [num] UNION SELECT 2 [num]
) [a]
SELECT @sum WithoutOrderBy
GO

Let us see the result when we run the above two queries with and without ORDER BY clause.

Again, it is interesting to see how results are different from each other when there is only change of inclusion of ORDER BY clause. Bob has explained why this is happening in his comment earlier. He also explains further in another comment in the same article that.

Comment from Bob:

One more item to note is that if you built:

SELECT @Str0 = @Str0 + C1 FROM T1

into a table expression (i.e. VIEW, FUNCTION, FROM (SELECT)), you may get the results you want over and over until the one day where you call the view or function with the combination of syntax that triggers the unexpected behavoir.

pinaldave,

On my final point, I think it’s unclear/weak. I’d like to add that the most clear reason as to why you can’t expect to get AB or BA, but might in fact get just A or just B, is that the construct @=@ +col is non-deterministic and therefore, for the same range of input values, is not guaranteed to produce the same result in every instance.

Bob, Thank you very much for excellent explanation and your participation in the community.

Reference : Pinal Dave (http://blog.sqlauthority.com)

Today I have question for you. Just a day ago I was reading whitepaper Improving Performance with SQL Server 2008 Indexed Views. Following is question and answer I read in the white paper.

Q. Why can’t I use OUTER JOIN in an indexed view?

A. Rows can logically disappear from an indexed view based on OUTER JOIN when you insert data into a base table. This makes incrementally updating OUTER JOIN views relatively complex to implement, and the performance of the implementation would be slower than for views based on standard (INNER) JOIN.

Here I would like to ask you one question, do you have example for kind of OUTER JOIN where you insert data into base table, it will make a row disappear from query?

Please post your answer as comment.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Recently, I was working on an optimization project for one of the large organizations. While working on one of the queries, we came across a very interesting observation. We found that there was a query on the base table and when the query was run, it used the index, which did not exist in the base table. On careful examination, we found that the query was using the index that was on another view. This was very interesting as I have personally never experienced a scenario like this. In simple words, “Query on the base table can use the index created on the indexed view of the same base table.”

If you have skipped the first paragraph, I suggest you go over the first paragraph one more time and try to understand the intended meaning. According to me, this is a very interesting observation and we rarely come across such an instance. I have made very small attempt to recreate this particular behavior.

USE tempdb
GO
-- Create Table
CREATE TABLE IndexViewAggr (ID INT)
GO
-- Insert One Hundred Thousand Records
INSERT INTO IndexViewAggr (ID)
SELECT TOP 100000 ROW_NUMBER() OVER (ORDER BY a.name) RowID
FROM sys.all_objects a
CROSS JOIN sys.all_objects b
GO
-- Create View on the table
CREATE VIEW ViewAggr WITH SCHEMABINDING
AS
SELECT ID, COUNT_BIG(*) CountID
FROM dbo.IndexViewAggr
WHERE ID BETWEEN 1 AND 1000
GROUP BY ID
GO
-- Following query is very identical to View
-- Run the query on base table
-- Please note that in execution plan there is Table Scan
SELECT ID, COUNT_BIG(*) CountID
FROM dbo.IndexViewAggr
WHERE ID BETWEEN 1 AND 1000
GROUP BY ID
GO
-- Create Index on the View
-- Please note that this index is on View and not on table
CREATE UNIQUE CLUSTERED INDEX IndexView ON ViewAggr(ID)
GO
-- Run the query on base table again
-- Please note that in execution plan there is Index Scan
SELECT ID, COUNT_BIG(*) CountID
FROM dbo.IndexViewAggr
WHERE ID BETWEEN 1 AND 1000
GROUP BY ID
GO
-- Please note that there are difference in "Query Cost"
-- Clean up Database
DROP VIEW ViewAggr
GO
DROP TABLE IndexViewAggr
GO

Now let us quickly observe two performance comparison of two of the selects. One object is created before the creation of the index over the view, and the other object is created after the creation of the index over the view. Please note that we are talking about two different objects here. Let me explain this schematically.

This is a very interesting behavior as the index created on view affects the query that runs on the base table. Let me show you the execution plan of the query before and after index creation.

From this example, it is very clear that there is a very big difference between the query costs when the index is created on the table. In fact, there is a very simple explanation of this behavior in the white paper Improving Performance with SQL Server 2008 Indexed Views.

The reason for this behavior is that the query optimizer works when it checks for the index on view and finds that the execution plan of the query is similar to that of the view.

Let me know your thoughts about this article. I do understand that this concept is slightly complicated. Further, this type of behavior is rarely encountered in real world. I hope that this article will give you a better insight to such a situation.

Reference : Pinal Dave (http://blog.sqlauthority.com)

Posted in Pinal Dave, SQL, SQL Authority, SQL Index, SQL Optimization, SQL Performance, SQL Puzzle, SQL Query, SQL Scripts, SQL Server, SQL Tips and Tricks, SQL White Papers, SQLServer, T SQL, Technology ]]> http://blog.sqlauthority.com/2009/09/24/sql-server-interesting-observation-index-on-index-view-used-in-similar-query/feed/ 14 pinaldave http://blog.sqlauthority.com/2009/08/24/sql-server-index-seek-vs-index-scan-diffefence-and-usage-a-simple-note/ http://blog.sqlauthority.com/2009/08/24/sql-server-index-seek-vs-index-scan-diffefence-and-usage-a-simple-note/#comments Mon, 24 Aug 2009 01:30:18 +0000 pinaldave http://blog.sqlauthority.com/?p=6648 ]]>

In this article we shall examine the two modes of data search and retrieval using indexes- index seeks and index scans, and the differences between the two.

Firstly, let us revisit indexes briefly. An index in a SQL Server database is analogous to the index at the start of a book. That is, its function is to allow you to quickly find the data you are searching for inside the book; in the case of a database, the “book” is a table.

An index scan means that SQL Server reads all rows in a table, and then returns only those rows that satisfy the search criteria. When an index scan is performed, all the rows in the leaf level of the index are scanned. This essentially means that all of the rows of the index are examined instead of the table directly. This is sometimes contrasted to a table scan, in which all the table data is read directly. However, there is usually little difference between an index scan and a table scan.

You may wonder why the Query Optimizer may choose to do an index or table scan. Surely it is much faster to first look up data using an index than to go through all the rows in a table? In fact, for small tables data retrieval via an index or table scan is faster than using the index itself for selection. This is because the added overhead of first reading the index, then reading the pages containing the rows returned by the index, does not offer any performance improvement for a table with only a few rows.

Other reasons to use an index scan would be when an index is not selective enough, and when a query will return a large percentage (greater than 50%) of rows from the table. In such cases the additional overhead of first using the index may result in a small degradation of performance.

An index seek, on the other hand, means that the Query Optimizer relies entirely on the index leaf data to locate rows satisfying the query condition. An index seek will be most beneficial in cases where a small percentage (less than 10 or 15%) of rows will be returned. An index seek will only affect the rows that satisfy a query condition and the pages that contain these qualifying rows; this is highly beneficial, in performance terms, when a table has a very large number of rows.

It is also worth noting that it is usually not worthwhile to create indexes on low-cardinality columns as they would rarely be used by the Query Optimizer. A low-cardinality column is one that contains a very small range of distinct values, for example a ‘Gender’ column would have only two distinct values- Male or Female. An example of a high-cardinality column is of course the primary key column, in which each value is distinct.

In summary, the Query Optimizer generally tries to perform an index seek. If this is not possible or beneficial (for example when the total number of rows is very small) then an index scan is used instead.

Reference : Pinal Dave (http://blog.sqlauthority.com)

About a week ago, SQL Server Expert, Imran Mohammed, provided a script, which will list all the missing identity values of a table in a database. In this post, I asked my readers if any could write a similar or better script. The results were interesting. While no one provided a new script, my question sparked a very active discussion that is still ongoing.

When providing the script, Imran asked me if I knew of any specific circumstances in which this kind of query could be useful, as he could not think of an instance where it would be necessary to find a missing identity. I was unable to think of a single reason for listing missing identities in a table. I posted Imran’s script on the assumption that someone would come up with an improved script, but as mentioned earlier, nobody did. Instead, we have been able to follow a very interesting discussion on subject of the need, if any, for listing Missing Identity values.

So, the question is this: “Do you know a real-world scenario where a Missing Identity value in any table can create problems?“

I have already received some extremely interesting comments from many experts, and all have posed the above question in one form or another. At this moment, I am still trying to think of an example from my own experience, but have yet to find one. Imran has since come up with one good example. Here is what he and other experts have suggested so far.

Jacob Sebastian – IDENTITY values are not expected to be sequential and there are all chances of having missing identity values, the most common cause is transaction rollbacks.

Simon Worth – The identity column is basically just a random number – even though they come sequentially. A developer making an assumption that the next record inserted will have an identity that is 1 more than the last inserted record. And if this is the case – then there are flaws in the logic of the developer.

Jacob Sebastian – What if the value in my table is 1, 2, 3, 5, 6 etc where “4″ is missing from the sequence. So what is the importance of knowing whether a table has missing identity values or not?

Imran Mohammed – If you use Identity property as your most unique column and Transaction Identifier, then definitely you would want to know why few transaction did not completely, Is there any specific fashion these transaction fails (Can be found out looking at missing values of identity)… Could be helpful to debug.

Now it is your turn. Let us have your thoughts.

Reference : Pinal Dave (http://blog.sqlauthority.com)

TechEd on Road Ahmedabad, June 20, 2009, was a huge success. This grand event saw over 200 attendees actively participating in the sessions. We had attendees traveling from far and wide, including Delhi, Mumbai, Jaipur, Kerala, Baroda, Himmatnagar, Rajkot, among other cities from India. This enthusiastic participation made the event truly grand. It was a moment of bliss for me as I had not anticipated such tremendous positive response!

Although the Official time to commence the event was at 1:45 PM we were really excited to see the attendees entering the hall before the official time. We were more than happy to quickly arrange a special session for those attendees who came in early to keep them engaged. They won loads of gifts while waiting for the official sessions to start. We had a brainstorming quiz session. I was happy to see that people in Ahmedabad are now taking keen interest in enhancing their knowledge.

One of the questions was – What is the maximum number of Index per table? I received lots of answers to this question but only two answers are correct. Let us now take a look at them.

For SQL Server 2005:
1 Clustered Index + 249 Nonclustered Index = 250 Index
http://msdn.microsoft.com/en-us/library/ms143432(SQL.90).aspx

For SQL Server 2008:
1 Clustered Index + 999 Nonclustered Index = 1000 Index
http://msdn.microsoft.com/en-us/library/ms143432.aspx

I would like to thank one of the attendees of Ahmedabad TechEd Kruti Kansara for the above comment.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

How to improve performance of SQL Server Queries is a common topic of discussion among many of us. Much has been said, much has been discussed. Few days back, I had an interesting discussion with one of the Junior developers regarding performance improvement of SQL Server Queries. We discussed on how by using a separate hard drive for several database objects can right away improve performance. I suggested him that non clustered index and tempdb can be created on a separate disk to improve performance.

No sooner had I given my suggestion than I received a question – What will happen if we can create clustered index on a separate drive from the table on which it is built.

My answer is : No! It is not possible at all.

Let us first be clear about the difference between a clustered and a non clustered index.

Clustered Index

• Only 1 allowed per table
• Physically rearranges data in the table to conform to the index constraints
• For use on columns that are frequently searched for ranges of data
• For use on columns with low selectivity

Non-Clustered Index

• Up to 249 (for SQL Server 2005) and 999 (for SQL Server 2008) allowed per table
• Creates a separate list of key values with pointers to the location of the data in the data pages
• For use on columns that are searched for single values
• For use on columns with high selectivity

A table devoid of primary key index is called heap, and here data is not arranged in a particular order, which gives rise to issues that adversely affect performance. Data must be stored in some kind of order. If we put clustered index on it then the order will be forced by that index and the data will be stored in that particular order.

Reference : Pinal Dave (http://blog.sqlauthority.com)