I am using threading in my .net application and which is causing duplicates in my database, so when two threads are creating duplicates at the same time, in this situation will this IGNORE_DUP_KEY = ON work successfully?

Data
——–
insert into test values (’2012-10-01′,’First-1′);
insert into test values (’2012-10-01′,’First-2′);

insert into test values (’2012-11-01′,’First-1′);
insert into test values (’2012-11-01′,’First-2′);

When I select (select * from test order by 1) it returns
2012-10-01 First-1
2012-10-01 First-2
2012-11-01 First-1
2012-11-01 First-2

And when I again insert
insert into test values (’2012-10-01′,’First-3′);
insert into test values (’2012-10-01′,’First-0′);

and select (select * from test order by 1), it returns
2012-10-01 First-0
2012-10-01 First-1
2012-10-01 First-2
2012-10-01 First-3
2012-11-01 First-1
2012-11-01 First-2

I want to get the data as
2012-10-01 First-1
2012-10-01 First-2
2012-10-01 First-3
2012-10-01 First-0

2012-11-01 First-1
2012-11-01 First-2

Please let me know if this is possible to sort the data by first column and second column get sorted by the insert sequence.

Thanks,
Yunus

Loved your explanation.

I have a situation where I need to select data joining many tables, filter them using various rules and calculate some values out of them. I select data into table valued variable. After that I must update other tables depending what’s in that temp table.

Now I don’t want to have duplicate entries in that temp table. It wouldn’t break anything but there could be A LOT of rows so I would have to do a lot of unnecessary updates and cause major havoc in the db with write locks.

I figured out that I could extract distinct rows out of the temp table but that would cause some extra work elsewhere in my procedure. So instead of that I just added “with (IGNORE_DUP_KEY = ON)” to the tables primary key and it seems to work like a charm :)

– There’s duplicate rows now
SELECT * FROM DupData ORDER BY ID
GO
– Create temp table
CREATE TABLE #DupIndex (ID INT,
FirstName VARCHAR(100),
LastName VARCHAR(100),
City VARCHAR(100))
GO
– And index
CREATE UNIQUE NONCLUSTERED INDEX [IX_DupIndex_FirstName]
ON [dbo].[#DupIndex]
(
[FirstName] ASC
) WITH (IGNORE_DUP_KEY = ON) ON [PRIMARY]
GO
– Select rows into temp table (duplicates are ignored)
INSERT INTO #DupIndex (ID,FirstName,LastName,City)
SELECT ID,FirstName,LastName,City
FROM DupData
ORDER BY ID
GO
– Truncate table
TRUNCATE TABLE DupData;
GO
– Select distinct rows back to table
INSERT INTO DupData (ID,FirstName,LastName,City)
SELECT ID,FirstName,LastName,City
FROM #DupIndex
ORDER BY ID
GO
– Now there’s now duplicate data
SELECT * FROM DupData ORDER BY ID
GO
– Drop temp table
drop table #DupIndex;
GO

Shall i consider this as a new way for removing duplicate records from a table.

SELECT MIN(ID) AS ID, FirstName, LastName, City
FROM DupData
GROUP BY FirstName, LastName, City
ORDER BY ID

It works but it’s not very generic (or efficient) way to do it, I would say.

But I think this is really limited situation where you’d want to copy data from one table to another. Good to know that this kind of possibility exists though.

Aaron

The data specific query can be answered only if you provide table design and existing data in source and destination tables.

Regards,
Pinal Dave

Nice article.

I have one question,

– Insert will throw warning
– Warning will not insert duplicate row
INSERT INTO DupIndex (ID,FirstName,LastName,City)
SELECT ID,FirstName,LastName,City
FROM DupData
ORDER BY ID
GO

From above query, WITH (IGNORE_DUP_KEY = ON) ON [PRIMARY] key, why ID =1 is ignored from table DupData while insertion, why not ignore ID = 3?