Some conversations are very interesting and I would like to bring them to everyone’s attention. Here is one of the conversation happened earlier this week. Let us say the question was asked by Dustin.

Dustin: Can we retrieve last 10 rows from a table?

Pinal: Sure, just get the top rows from the end of the table. For this you will have to order table with Identity Column Descending.

Dustin: I do not have an identity column in table.

Pinal: No problem. Do you have a column which tracks the date and time for the newly inserted row or timestamp?

Dustin: No such thing in my table.

Pinal: Hmmm… That means there is no property which can identify if the rows were inserted last or first.

Dustin: I believe so. Any other thoughts?

Pinal: Well, is there any auditing on the table or any trigger or any other mechanism where you store your data to another table for example with an OUTPUT clause?

Dustin: No, nothing like that. Is there any administrative ways to get the last inserted row?

Pinal: What do you mean?

Dustin: Can I read the logs and know what were the rows inserted?

Pinal: I have personally never tried to read log files and I have no clue how to do that. If you are going do that, I suggest you take backup of your log file and do the experiment on it. There are good chances if you use tool which is not safe you will get your database at very high risk.

Dustin: Do you recommend any tool?

Pinal: No, I do not recommend any tool as of now. If I will find one, I will write a blog post.

Note: Please do not post a comment with third party tools link, I will delete them without notice. I will be doing my research soon and will post a blog about it.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Why use a caching solution?

Let’s quickly recap the many reasons to use data caching with SQL Server:

• To improve data access read/write performance
• To reduce the number queries hitting the of database
• To reduce database load (CPU, IO, Memory)
• To improve application page load response time (the user’s actual experience)
• To distribute repetitive query workloads to separate cache servers
• To share the benefit of cached data on one server across all other servers.
• To increase operational efficiency, by scaling more data and more users on smaller and fewer SQL Servers machines

Another benefit of caching is its impact on database scalability. SQL Server is expensive to scale with bigger hardware, which also can trigger increased license fees. Fast Flash storage systems, which can be helpful, are also very expensive. Whereas adding a new web/caching server can be much cheaper (and potentially more effective).

So, now that we know why we want to use caching with SQL SERVER, what’s the best way to go about it? Well, you’ll need to decide what’s best for you. Let’s look at three main SQL Server caching options:

1. SQL Server Memory and IO caching
2. Application data caching, a Do It Yourself approach using Memcached/similar APIs
3. SafePeak automated SQL caching

By the way, I’d also like to learn from your experiences, so I look forward to your comments and thoughts on what I discuss here.

SQL Server: Doesn’t it already have memory cache?

To start with, we need to remember that SQL Server has its own memory cache for objects in the database. When data is retrieved, SQL Server maintains its cache and will (if necessary) pull the row from its memory and not hit the disk.

So if SQL Server has its own cache, what’s the benefit of data caching layers such as Memcached, SafePeak or similar?

The answer is this: SQL Server only caches:

1. Query plans
2. Pages from the database files.

SQL Server does NOT cache results from a query.

This is the important distinction.

For example, imagine if you have a complex query which uses some aggregation on a lot of data (e.g.: how many different countries we have in our customer database: SELECT DISTINCT Country from Customers GROUP BY country). SQL Server will scan the WHOLE customer table, but your result-set will only be a few entries long. When you reissue your query, SQL Server will reuse the query plan and will rescan the customer table, (and if you are lucky the pages are still in memory)

When you use application cache, you store your result-sets in Memcached RAM. Then reuse them over and over again without connecting to the database server, thus offloading workloads from your database server.

Do-It-Yourself application data caching using Memcached / similar APIs

Application data caching is quite easy to start. However, changing (or building) code for large applications with effective data caching is challenging for a few reasons:

1. Correctly scoping required code changes;
2. Delivering and testing actual code changes required to handle caching of all (or most) SQL queries;
3. Preserving data integrity: how to invalidate all relevant cache items and to handle all kinds of data change events. And in real-life both READs / WRITEs can be complex involving stored-procedures, triggers, views, table joins etc.; and
4. Maintaining the additional code layer.

One thing to keep in mind is that because DIY and Memcached approaches need to touch app code, they can’t be used to accelerate any off-the-shelf software that relies on SQL Server.

So, while it can be complicated, application performance benefits are always good, so let’s dive into what you would do using Memcached.

About Memcached data caching
Memcached is a big hash table: a key/value store that lives entirely in RAM on multiple servers. It allows you to use RAM from multiple servers as single memory caching space.

Basic data caching using Memcached

Below is a basic tutorial showing (via pseudocode) how you can get started with integrating Memcached into your application. If you’re an application developer, it isn’t something you just “turn on” and then your site goes faster. You have to pay attention. For every query you want to be cached, you have to put (set()) it into cache, get it from cache (get()) and (probably most complicated) make sure you keep the data in the cache correct by removing from cache (delete()) when the data is updated.

The following pseudocode of Memcached example is written in #Perl but can be done in same way in most other languages, including .net, java and php – etc (memcached clients)

Initializing a Memcached Client with a list of your pre-configured Memcached servers:

# perl example
my $memclient = Cache::Memcached->new({ servers => [ '10.0.0.10:11211', '10.0.0.11:11211' ]});

Wrapping an SQL Query
Memcached is famous for reducing load on SQL databases. Here is some pseudocode showing how to wrap a database query access with a memcached caching layer, by implementing check-in-cache (function get()) and place-into-cache (function set()):

# Define a query and use it as a key for the Memcached:
sql = "SELECT * FROM user WHERE user_id = ?"
key = 'SQL:' . user_id . ':' . md5sum(sql)
# We check if the value is 'defined' (or in cache), since '0' or 'FALSE' can be legitimate values!
if (defined result = memcli:get(key)) {
        return result
} else {
        # Query is not in Cache, Get query resultset from your database server and convert it to resultset array
        handler = run_sql(sql, user_id)
        rows_array = handler:turn_into_an_array

        # Cache it for five minutes
        memcli:set(key, rows_array, 5 * 60)
        return rows_array
}

Notice that the SQL query result-set was entered to cache with five-minute expiration time, as an example.

Stale data in cache and invalidation of cached items
Unless you actively invalidate the cached item, when a user (or your database) makes a change, it can take up to five minutes (or more, depending on cache item expiration time) for users to see the correct new data. This is a key issue when implementing a Do-It-Yourself caching strategy.

When a user comes along and edits data, you can keep the cache in sync in two main ways:

1. You can update the item in cache; or
2. Delete the old item.

Expiration
The most basic invalidation happens by defining the Expiration period. Even if you‘re actively deleting or overwriting cached data, you’ll still want to have the cache expire occasionally.

 # Cache it for five minutes
  memcli:set(key, rows_array, 5 * 60)

Expiration times can be set from 0, meaning “never expire”, to 30 days. Any time higher than 30 days is interpreted as a timestamp date.

delete()
The simplest method of invalidation is to simply delete an entry and have your website re-cache the data the next time it’s fetched. For example, when a user updates her bio, you want her to see her latest info when she reloads the page:

 sql = "SELECT * FROM user WHERE user_id = ?"
 key = 'SQL:' . user_id . ':' . md5sum(sql)

 memcli:delete(key)

The next time this query will be requested it will be fetched from the database and repopulate the cache.

set()
A more efficient idea is to actively update your cache as your data changes. When our user updates her bio, take the bio object and move it into the cache via ‘set’. You can pass the new data into the same routine that normally checks for data, or however you want to structure it.

Summary about DYI application data caching

Data caching is considered a key component of a high performance and scalable application architecture. Fast page load response time is a one of the key metrics for user satisfaction. Memcached and similar distributed caching technologies are used today in most (if not all) well-known big websites (Linkedin, Facebook others). The challenge is doing all the extra work custom DIY caching represents for a development team to implement, test, and maintain.

If you think I’m painting a dark picture for Do-It-Yourself, home-grown caching, you’re right. Why? Because nowadays there are alternatives to use a commercial smart caching layer. The cost to purchase third-party caching software is much more accurately defined, than the unknown cost of developing caching code yourself.

But, you need to decide for yourself if that’s the best option for your situation.

SafePeak’s “Automated Dynamic Caching”

SafePeak is pre-packaged software that is specifically designed to cache SQL Server queries. It automates many of the steps that developers would try to do manually with Memcached. And it doesn’t require any changes to application code. This makes it quick to deploy and useful to accelerate the performance of both 3^rd-party applications (like SharePoint, CRM Dynamics) and your own custom applications.

In my opinion, this makes Safepeak’s approach much easier for software and database architects and developers to use: you can literally “plug-in” SafePeak to production SQL Server systems. All queries are analyzed and managed by SafePeak automatically: their logic and dependent objects (like tables, views, triggers, procedures, etc.) are identified and analyzed, write commands are monitored and cache invalidation applied in real-time.

And its app acceleration encompasses all of an app’s queries and procedures traffic immediately on deployment.

SafePeak’s self-learning algorithms manage most of configuration; the rest is done via a dashboard tool that combines cache management with real-time monitoring and statistics. By analyzing both the database schemas and the SQL traffic, SafePeak creates optimized caching patterns specific to your applications and improves them over time. SafePeak cache supports read-write database transactions and is dynamically kept in sync with any data changes as they happen.

Here’s what the SafePeak dashboard looks like (screenshot from SafePeak’s whitepaper):

What Next?

You have to decide what approach or technology is best for your situation.

An effective DYI caching strategy, using libraries like Memcached or Velocity, has some advantages (better granular cache invalidation control, ability to store additional types of data besides database row-sets and others). But it can be tricky to accurately scope the work needed, hard to code correctly and comes at a real cost of time and focus away from developing application capabilities.

SafePeak offers an easier way to accelerate SQL Server based applications, with all caching benefits without any app re-programming. In a way, SafePeak’s design reminds me of a combination of Memcached caching and an In-Memory database, but enhanced with machine learning intelligence.

Caching technology is designed to maximize operational efficiency of your existing infrastructure, so that you can scale more data and more workloads.

I like SafePeak’s “plug-and-play” approach as it gets results fastest and with minimum risk. If you’d like to check it out, they offer a free trial (www.safepeak.com/download) that includes one-on-one technical training & assistance support.

Now I’d like to hear from you. Have you deployed a SQL caching strategy? What have you done? How has it worked out?

Reference: Pinal Dave (http://blog.sqlauthority.com)

Let us create the following table:

CREATE TABLE items(item_id INT, item_description VARCHAR(100));
INSERT INTO items VALUES (1,'Television');
INSERT INTO items VALUES (2,'Mobile');
INSERT INTO items VALUES (3,'laptop');
INSERT INTO items VALUES (4,'Cables');
INSERT INTO items VALUES (5,'Camera');
INSERT INTO items VALUES (6,'jewels');
INSERT INTO items VALUES (7,'shirt');
INSERT INTO items VALUES (8,'Cup');
INSERT INTO items VALUES (9,'Pen');
INSERT INTO items VALUES (10,'Pencil');

1 Find out item_description that starts with c
SELECT item_description FROM items
WHERE item_description regexp '^c';

Result :

Item_description
 Cables
 Camera
 Cup

2 Find out item_description that ends with s
SELECT item_description FROM items
WHERE item_description regexp 's$';

Result :

Item_description
 Cables
 jewels

3 Find out item_description that starts with c or ends with s
SELECT item_description FROM items
WHERE item_description regexp '^c|s$';

Result :

Item_description
 Cables
 Camera
 jewels
 Cup

4 Find out item_description that contains the alphabet a
SELECT item_description FROM items
WHERE item_description regexp '[a]';

Result :

Item_description
 laptop
 Cables
 Camera

5 Find out item_description that contains the alphabet c or p

SELECT item_description FROM items
WHERE item_description regexp '^[cp]';

Result :

Item_description
 Cables
 Camera
 Cup
 Pen
 Pencil

Reference: Pinal Dave (http://blog.sqlauthority.com)

“What is the difference between latches and locks?”

The answer to this question is not an easy one. Here is a quick answer from the MS white paper.

A page in SQL Server is 8KB and can store multiple rows. To increase concurrency and performance, buffer latches are held only for the duration of the physical operation on the page, unlike locks which are held for the duration of the logical transaction. Latches are internal to the SQL engine and are used to provide memory consistency, whereas locks are used by SQL Server to provide logical transactional consistency.

Above answer is taken from the old, but extremely relevant white paper written in the year 2011. It was written for SQL Server 2008 R2, however, almost all the concepts from this Whitepaper are still very accurate.

If you want to learn more about Latches and how to diagnose and resolve problems around latch contention, I strongly suggest following whitepaper.

Diagnosing and Resolving Latch Contention on SQL Server

Let me know your thoughts about this subject.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Q: What is the default Port SQL Server running on?

A: 1433 for TCP/IP and 1434 for USD connection.

I always get the correct answer for this question. However, when I ask a follow up question, so far I have yet not got a single correct answer.

Q: Great, now assumes that there is more than one instance of SQL Server running on the same server. What will be the port of the second SQL Server running on the server as the default port is already used by default instant?

A: (I have yet to get the correct answer for this one in interview).

Because of the same reason, I have decided to blog about this.

Here are different ways one can figure out on which port the second instance of SQL Server is running.

Method 1: using xp_readerrorlog

Execute following stored procedure on the instance where you want to find out port on which SQL Server is running.

USE MASTER
GO
xp_readerrorlog 0, 1, N'Server is listening on'
GO


The above query will return results something very similar to the following:

Method 2: Configuration Properties

Go to Windows >> Start >> SQL Server Program Folder >> SQL Server Configuration Manager

Now go to SQL Server Network Configurations >> Protocols for Your Server and go to TCP/IP and right click over it.

Now over here when you scroll down you will notice Port Details.

It is that easy to find the port of the second instance of SQL Server.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Could not allocate space for object ‘<object_name>’ in database ‘<database_name>’ because the ‘PRIMARY’ filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.

I was a bit surprised as this was not a production server or even real server. It was my laptop with SQL Server installed in it for testing and presentation purpose. I was a bit shocked and decided to figure out what is wrong with my SQL Server. Just like everyone I searched online and I end up having few solutions, however, after a while I realized then what exactly I was missing. However, I encourage all of you to read various steps I took to reach to a solution to this problem.

Solution / Fix :

Step 1: I checked if my Hard Drive has more space available.

If your Hard Drive is full, please empty out any unnecessary stuff. This may not directly fix your problem, but it is a necessary exercise to do before you do Step 2.

Step 2: Delete unnecessary object from your file group.

If your primary filegroup (in most cases) or filegroup which has generated this error is restricted to fix size, you may get this error. You have two workarounds here.

Step 2a: Delete objects from your filegroup which you do not need. You can move objects to another filegroup if you have multiple filegroups.

Step 2b: Increase the fixed size of your primary filegroup.

Step 3: Check your SQL Server version.

SQL Server Express version 2008 and earlier had the limitation of 4GB maximum Database size. SQL Server Express 2008 R2 and onwards have limitation 10GB maximum database size.

In my case, I was running SQL Server Express version, hence I had faced above error. I moved my database to standard edition and everything went fine.

Reference: Pinal Dave (http://blog.sqlauthority.com)

In this episode of the Notes from the Field series I asked SSIS Expert Andy Leonard a very crucial question – How to troubleshoot SSIS data flow? It is such a complicated problem that everyone thinks they know, but not everyone can resolve this issue. Any Leonard is a world renowned expert, explains in this blog post how to troubleshoot SSIS data flow.

SQL Server Integration Services (SSIS) is designed to move data. Several SSIS tasks can move data but the Data Flow Task is arguably used most to accomplish moving data. When bad things happen to good developers, it helps to know how to troubleshoot. Writing about all the ways to troubleshoot an SSIS Data Flow Task will take more than one post. So this is Part 1.

Building a Quick Demo Package

If you are interested in working through some demos with me, create a new SSIS project named DataFlowTests. Add a Data Flow Task to the Control Flow as shown in Figure 1:

You can use any source data you desire, but I am going to use a file containing weather data captured from my weather station here in Farmville, Virginia beginning in December 2008. You can obtain this data here. Click the Data Flow tab to edit the Data Flow Task. Add a Flat File Connection Manager configured to consume the sensor1-all.csv file containing the December 2008 weather data. Add a Flat File Source adapter and link it to the Flat File Connection Manager. You Data Flow should appear as shown in Figure 2:

To test the Flat File Source adapter, we need to connect an output to another component. Add a Union All transformation and connect the output of the Flat File Source to it, as shown in Figure 3:

You can now test-execute the SSIS package. You should see results similar to those shown in Figure 4:

This data flow isn’t doing very much. The Flat File Source adapter is coupled to the Flat File Connection Manager. The Flat File Connection Manager reads data from the sensor1.all.csv file you downloaded from andyweather.com. Weather data is read into the data flow from the file via the Flat File Connection Manager and the Flat File Source adapter. The 106 rows of weather data are then passed into the Data Flow Path connecting the Flat File Source adapter output to the first Union All Input (Union All Input 1).

Insert a Warning

Let’s create an issue that will raise a warning. Open the Flat File Connection Manager Editor, click on the Advanced page, select the Min T column, and change its DataType property to “four-byte signed integer [DT_I4] as shown in Figure 5:

When you close the Flat File Connection Manager Editor, a warning icon displays on the Flat File Source adapter. If you hover over the Flat File Source adapter with your mouse, a tooltip will display some of the warning text as shown in Figure 6:

You can see more of the Warning if you execute the package and the view the Progress (if the package is running the SSIS Debugger) or Execution Results tab (if the package has been executed in the Debugger and then the Debugger has been stopped), as shown in Figure 7:

The warning we injected by changing the DataType property of the Min T column in the Flat File Connection Manager does not cause an error. The warning stems from the fact that there is a difference between the data type of the Min T column flowing into the Flat File Source adapter from the Flat File Connection Manager and the data type of the Min T column flowing out of the Flat File Source adapter. You can correct this warning by opening the Flat File Source adapter and clicking the Columns page. When opening the Editor (or Advanced Editor) of the Flat File Source adapter, you will be prompted as shown in Figure 8:

Click the “Yes” button to synchronize the Min T column’s data type in the Flat File Source adapter’s output (the metadata of the “output column”) with the Min T column’s data type supplied from the Flat File Connection Manager (the metadata of the “external column”).

Insert an Error

One way to generate an error in our simple data flow task is to configure the Flat File Connection Manager to perform an impossible data type coercion. Every value in the Min HI column contains “—“. Let’s configure the Min HI column in the Flat File Connection Manager and the Flat File Source adapter as data type Integer.

First, open the Flat File Connection Manager Editor and change the Min HI DataType property to DT_I4 as shown in Figure 9:

When you click the “OK” button, we see the same warning displayed in Figure 6 – the external columns are out of synchronization with the data source columns. Right-click the Flat File Source adapter and click Show Advanced Editor as shown in Figure 10:

When you attempt to open the Advanced Editor, you will be prompted to fix the data type mismatch between the output columns and the external columns. Click the “No” button to proceed, as shown in Figure 11:

When the Advanced Editor for the Flat File Source adapter opens, click the Input and Output Properties tab. Expand the Flat File Source Output node, and then expand the Output Columns node. Click on the Min HI column and change the DataType property to “four-byte signed integer [DT_I4] as shown in Figure 12:

Click the “OK” button to close the Advanced Editor. Note the Union All now displays a validation error as shown in Figure 13:

What has happened? The tooltip reveals an issue with a metadata mismatch, but the error is truncated. We can view the full error message in the Error List by clicking ViewàError List as shown in Figure 14:

Note: The Error List option is missing from the View menu in some versions of SSDT-BI. In those environments, use the keystrokes. Hold down Ctrl and press \ followed by E.

The Error List window displays the full text of errors, warnings, and other informational messages, and is shown in Figure 15:

There are a couple ways to fix this error. The easy way is to delete the Union All and add a new Union All transformation in its place (the hard way is to edit the package’s XML).

Conclusion

We are going to stop here. We have introduced an error in the Flat File Source Adapter, though. And we will begin there in Part 2.

If you want to get started with SSIS with the help of experts, read more over at Fix Your SQL Server.

Reference: Pinal Dave (http://blog.sqlauthority.com)

STRAIGHT_JOIN is actually an INNER JOIN but it forces the order in which data should be processed.

As per MySQL’s documentation at, the definition is given below

“STRAIGHT_JOIN is similar to JOIN, except that the left table is always read before the right table. This can be used for those (few) cases for which the join optimizer puts the tables in the wrong order.”

Let us create the following data

CREATE TABLE items(item_id INT, item_description VARCHAR(100));
CREATE TABLE sales(sales_id INT auto_increment KEY,item_id INT, sales_date DATETIME, sales_amount DECIMAL(12,2));
INSERT INTO items VALUES (1,'Television');
INSERT INTO items VALUES (2,'Mobile');
INSERT INTO items VALUES (3,'laptop');
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (1,'2014-01-01',1200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-01-02',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (3,'2014-01-09',1700);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (3,'2014-01-29',1700);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (3,'2014-02-11',1700);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (1,'2014-02-16',1200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-02-16',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-02-20',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-02-20',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-02-22',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (3,'2014-02-24',1700);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (1,'2014-02-24',1200);


STRAIGHT_JOIN can be used in two methods

Method 1 Join tables using STRAIGHT_JOIN

SELECT t1.item_description,t2.sales_id,t2.sales_date,t2.sales_amount
FROM sales AS t2 STRAIGHT_JOIN items AS t1  ON t1.item_id=t2.item_id;


Method 2 Use INNER JOIN to join tables and use STRAIGHT_JOIN in the SELECT clause itself

SELECT STRAIGHT_JOIN t1.item_description,t2.sales_id,t2.sales_date,t2.sales_amount
FROM sales AS t2 INNER JOIN items AS t1  ON t1.item_id=t2.item_id;


Both the queries produce the following result

item_description sales_id sales_date sales_amount
 Television 1 1/1/2014 0:00 1200
 Mobile 2 1/2/2014 0:00 200
 laptop 3 1/9/2014 0:00 1700
 laptop 4 1/29/2014 0:00 1700
 laptop 5 2/11/2014 0:00 1700
 Television 6 2/16/2014 0:00 1200
 Mobile 7 2/16/2014 0:00 200
 Mobile 8 2/20/2014 0:00 200
 Mobile 9 2/20/2014 0:00 200
 Mobile 10 2/22/2014 0:00 200
 laptop 11 2/24/2014 0:00 1700
 Television 12 2/24/2014 0:00 1200

However the real test is when we execute the STRAIGHT_JOIN syntax with EXPLAIN command before SELECT statement and observe the order of table joined.

Method STRAIGHT JOIN

EXPLAIN SELECT STRAIGHT_JOIN t1.item_description,t2.sales_id,t2.sales_date,t2.sales_amount
FROM sales AS t2 INNER JOIN items AS t1 ON t1.item_id=t2.item_id;

Now let us see the result where we observe the order of tables used in Join.

You will notice that Sales Table which is aliased as t2 is first used in the result set.

Method INNER JOIN (without Straight Join)

EXPLAIN SELECT t1.item_description,t2.sales_id,t2.sales_date,t2.sales_amount
FROM sales AS t2 INNER JOIN items AS t1 ON t1.item_id=t2.item_id;

Now let us see the result where we observe the order of tables used in Join.

You will notice that Sales Table which is aliased as t2 is second used in the result set. That means MySQL have used internal optimizer and re-ordered the table order for maximum performance.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Before we jump into the contest, let us quickly understand what Clustrix is all about in four simple points:

• ClustrixDB can scale reads, writes, updates and analytics, — near linearly — as you add nodes. The scale-out architecture of the cloud means that the new cloud applications use scale-out NoSQL or SQL databases, and most often a combination of both.
• Real-time analytics is analytics on your live operational database, up to date to the current moment. ClustrixDB allows real-time insights into your business and fast, current reports for your business and self-serve customers.
• ClustrixDB has powered mission-critical business applications for more than three years, with trillions of transactions per month running on ClustrixDB. Proven in massive transaction volume environments with an appliance form factor, ClustrixDB is now being used by multiple customers across the globe as software and on public clouds such as AWS.
• Several databases and data management platforms are now on the market. We created a series of landscapes to help our customers determine the right solution to their problem. Whether primary or analytics, SQL or NoSQL–comparing features gives clarity.

If you have directly skipped to this statement, I encourage to read the four bullet points above, it is indeed interesting to know that there are solution exists in the market which can help our mission critical problems. Now let us jump to details of the contest.

Contest Details

Step 1: Download and Install ClustrixDB here using the community license

Click Here to Download

Step 2: Do one thing with ClustrixDB 
ClustrixDB is nearly plug-and-play compatible with MySQL. You can do any of the following tasks:

• Load Data
• Create a Table
• Create an online schema change
• Run an Analytics query
• Or any other task which is a core database task

Step 3: Post it on the ClustrixDB forum (http://support.clustrix.com/forums/):

1. The current database you use
2. Snapshot (or screenshot) of what you did with ClustrixDB

That’s it! It is that simple and it will also give you exposure how to get going with ClustrixDB as well enable you to Win Amazon Gift Card.

Prize for Winner (Total worth USD 200)

• 10 Amazon Gift Cards each of USD 20
• 3 Gift Cards will go to the first 3 posts and the remaining 7 will be raffled among the other posts.
• Post your entry before March 20th Midnight EST.

Well, if you are an early bird, indeed there is a guaranteed prize for you as you can read the second bullet point above.

Hints

Well, if you want few helps regarding how to do any of the above tasks. Here are a few articles which you can follow:

• Getting Started on ClustrixDB Software
• Importing Data
• Creating the Database and Loading Data
• Best Practices for Loading Data Onto Clustrix

Let me know what you think of this contest and ClustrixDB. I have a surprise prize for one of the participation in this contest. I will announce it with the winner list after March 20th.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Let us create the following tables
CREATE TABLE items(item_id INT, item_description VARCHAR(100));
CREATE TABLE sales(sales_id INT auto_increment KEY,item_id INT, sales_date DATETIME, sales_amount DECIMAL(12,2));
INSERT INTO items VALUES (1,'Television');
INSERT INTO items VALUES (2,'Mobile');
INSERT INTO items VALUES (3,'laptop');
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (1,'2014-01-01',1200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-01-02',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (3,'2014-01-09',1700);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (3,'2014-01-29',1700);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (3,'2014-02-11',1700);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (1,'2014-02-16',1200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-02-16',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-02-20',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-02-20',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (2,'2014-02-22',200);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (3,'2014-02-24',1700);
INSERT INTO sales(item_id,sales_date,sales_amount) VALUES (1,'2014-02-24',1200);


By using the NATURAL JOIN, we can select only distinct columns by using * in the SELECT statement.
SELECT * FROM items NATURAL JOIN sales;

The result is

item_id Item_description sales_id sales_date sales_amount
 1 Television 1 2014-01-01 00:00:00 1200.00
 2 Mobile 2 2014-01-02 00:00:00 200.00
 3 laptop 3 2014-01-09 00:00:00 1700.00
 3 laptop 4 2014-01-29 00:00:00 1700.00
 3 laptop 5 2014-02-11 00:00:00 1700.00
 1 Television 6 2014-02-16 00:00:00 1200.00
 2 Mobile 7 2014-02-16 00:00:00 200.00
 2 Mobile 8 2014-02-20 00:00:00 200.00
 2 Mobile 9 2014-02-20 00:00:00 200.00
 2 Mobile 10 2014-02-22 00:00:00 200.00
 3 laptop 11 2014-02-24 00:00:00 1700.00
 1 Television 12 2014-02-24 00:00:00 1200.00

As you have noticed, * does not return all the columns from both the tables. The common column item_id is chosen first, followed by other columns in the tables items and sales. But if you use INNER JOIN with SELECT * all the columns will be selected which will result to duplicate columns in the resultset.

You can also use the USING clause with INNER JOIN to produce the same result.

SELECT * FROM items INNER JOIN sales USING (item_id);

In the above statement item_id is used to map both the tables.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Let us use the following data

CREATE TABLE mysql_testing(db_names VARCHAR(100));
INSERT INTO mysql_testing
SELECT 'SQL Server' UNION ALL
SELECT 'MySQL' UNION ALL
SELECT 'SQL Server' UNION ALL
SELECT 'MongoDB' UNION ALL
SELECT 'SQL Server' UNION ALL
SELECT 'MySQL' UNION ALL
SELECT 'Oracle' UNION ALL
SELECT 'PostGreSQL' UNION ALL
SELECT 'MongoDB' UNION ALL
SELECT 'SQL Server' UNION ALL
SELECT 'SQL Server' UNION ALL
SELECT 'MongoDB' UNION ALL
SELECT 'MySQL' UNION ALL
SELECT 'Oracle' UNION ALL
SELECT 'MongoDB' UNION ALL
SELECT 'PostGreSQL';

Now you can generate the row number using two variables and reset it for each group of db_names in two methods. The idea is to use two variables one for incrementing the numbers and other that can be used to reset the number in to 1 whenever group value changes.

Method 1 : Set a variable and use it in a SELECT statement and reset whenever group value changes

SET @row_number:=0;
SET @db_names:='';
SELECT @row_number:=CASE WHEN @db_names=db_names THEN @row_number+1 ELSE 1 END AS row_number,@db_names:=db_names AS db_names
FROM mysql_testing
ORDER BY db_names;

Method 2 : Use a variable as a table and cross join it with the source table and reset whenever group value changes

SELECT @row_number:=CASE WHEN @db_names=db_names THEN @row_number+1 ELSE 1 END AS row_number,@db_names:=db_names AS db_names
FROM mysql_testing, (SELECT @row_number:=0,@db_names:='') AS t
ORDER BY db_names;

Both the above methods return the following result

row_number db_names
 1 MongoDB
 2 MongoDB
 3 MongoDB
 4 MongoDB
 1 MySQL
 2 MySQL
 3 MySQL
 1 Oracle
 2 Oracle
 1 PostGreSQL
 2 PostGreSQL
 1 SQL Server
 2 SQL Server
 3 SQL Server
 4 SQL Server
 5 SQL Server

Reference: Pinal Dave (http://blog.sqlauthority.com)

MySQL does not have any system function like SQL Server’s row_number () to generate the row number for each row. However, it can be generated using the variable in the SELECT statement.

The following table has five rows.
CREATE TABLE mysql_testing(db_names VARCHAR(100));
INSERT INTO mysql_testing
SELECT 'SQL Server' UNION ALL
SELECT 'MySQL' UNION ALL
SELECT 'Oracle' UNION ALL
SELECT 'MongoDB' UNION ALL
SELECT 'PostGreSQL';


Now you can generate the row number using a variable in two methods

Method 1 : Set a variable and use it in a SELECT statement

SET @row_number:=0;
SELECT @row_number:=@row_number+1 AS row_number,db_names FROM mysql_testing
ORDER BY db_names;


Method 2 : Use a variable as a table and cross join it with the source table

SELECT @row_number:=@row_number+1 AS row_number,db_names FROM mysql_testing, (SELECT @row_number:=0) AS t
ORDER BY db_names;

Both the above methods return the following result

row_number db_names
 1 MongoDB
 2 MySQL
 3 Oracle
 4 PostGreSQL
 5 SQL Server

Reference: Pinal Dave (http://blog.sqlauthority.com)

“How big my table name can be?”

The answer is simple 128.  You can figure this out by executing stored procedure sp_server_info on SQL Server.

Execute following stored procedure in SQL Server.

EXEC sp_server_info

It will return following resultset.

attribute_id attribute_name attribute_value
1 DBMS_NAME Microsoft SQL Server
2 DBMS_VER Microsoft SQL Server 2012 - 11.0.3000.0
10 OWNER_TERM owner
11 TABLE_TERM table
12 MAX_OWNER_NAME_LENGTH 128
13 TABLE_LENGTH 128
14 MAX_QUAL_LENGTH 128
15 COLUMN_LENGTH 128
16 IDENTIFIER_CASE MIXED
17 TX_ISOLATION 2
18 COLLATION_SEQ charset=iso_1 sort_order=nocase_iso charset_num=1 sort_order_num=52
19 SAVEPOINT_SUPPORT Y
20 MULTI_RESULT_SETS Y
22 ACCESSIBLE_TABLES Y
100 USERID_LENGTH 128
101 QUALIFIER_TERM database
102 NAMED_TRANSACTIONS Y
103 SPROC_AS_LANGUAGE Y
104 ACCESSIBLE_SPROC Y
105 MAX_INDEX_COLS 16
106 RENAME_TABLE Y
107 RENAME_COLUMN Y
108 DROP_COLUMN Y
109 INCREASE_COLUMN_LENGTH Y
110 DDL_IN_TRANSACTION Y
111 DESCENDING_INDEXES Y
112 SP_RENAME Y
113 REMOTE_SPROC Y
500 SYS_SPROC_VERSION 11.00.3000

In the result set you can notice there is a row with information TABLE_LENGTH and the value is 128. There are few other information related to server is also listed in the result set.

Though, a regular tablename can be 128 character long, a temporary table cannot be longer than 116 character. You can read the detailed blog post over here Maximum Allowable Length of Characters for Temp Objects is 116 – Guest Post by Balmukund Lakhani.

Reference: Pinal Dave (http://blog.sqlauthority.com)

In this episode of the Notes from the Field series database expert David Klee explains a very crucial issue DBAs and Developer faces in their career – how to collect performance statistics. Linchpin People are database coaches and wellness experts for a data driven world. Read the experience of John in his own words.

Performance statistics are critically vital for the long-term health and capacity management of an enterprise environment. Ongoing performance metric collection ensures that you, the DBA, have the metrics of a system’s operating state whenever a performance problem is reporter – either real-time or historically. Too many times when I am troubleshooting a well-known performance problem, the client has no metrics about the system state during the time window where end users reported performance problems.

Let’s fix that. We need some performance statistic collection.

Why Collect Around the Clock?

Are you able to definitively state that, at 10:48pm tonight, you will have unusually high disk latency on your SQL Server archival data volume? If so, become a consultant and the world is yours! For the rest of us, our reality would be an email tomorrow morning stating that the application was ‘slow’ last night and the customer service reps received complaints. We would normally not have any method to get a clear picture of the state of our system last night, and without the numbers, are now left to start looking at the current system state to see if the issue reported is still lingering in the system.

Constant, around-the-clock performance statistic sampling is the simplest method of achieving this goal. Wouldn’t you like to be able to generate charts like the following at any time to get a simple picture of what your servers were doing at any given time?

A picture says a thousand words, and this simple graph quickly tells the reader the times where the system was busy. The underlying data was collected with Windows Perfmon on a five-minute interval throughout the week, and loaded into a database where an SSRS report is emailed out to the administrators every morning. The reports are quickly reviewed each morning as the DBA performs the usual morning review, and if an anomaly is present, would be quickly discovered.

Methods for Collection

Quite a number of third-party utilities from your favorite tooling vendors exists to assist you with ongoing performance statistic collection. They work great and I highly endorse any tool that makes you more efficient. But, if you do not have one of these utilities, or the price is prohibitive, you already have a tool that can handle this task for you. Windows Performance Monitor, or Perfmon, is built into the Windows Server operating system, and can be configured to constantly record performance statistics for future use.

Use the directions at Technet (link to http://technet.microsoft.com/en-us/library/cc749249.aspx) to set up an ongoing performance data collector set to collect information around the clock. I generally use a five-minute polling interval to start with. If you know you are experiencing undiagnosed performance issues, increase the polling interval.

What to Collect

The following list of Windows and SQL Server WMI counters should get you the basics needed to help with troubleshooting current performance problems. Feel free to modify this list according to your environment and your specific needs. The following web site contains a list of the performance counters and their descriptions.

http://technet.microsoft.com/en-us/library/cc768048.aspx

Note: if a counter has multiple instances of a selected object, make sure you select <All instances> so the counters are separated appropriately. This option helps you get individual statistics for objects like CPU cores or disk drive letters.

Windows Server Counters

SQL Server Counters

You can also build custom collectors for SQL Server-specific items that WMI will not serve. Use your favorite job scheduler to routinely execute queries against system DMOs and store the results into custom tables in a utility database. To get a good idea of the sort of metrics that you can collect, reference some of the many objects in Glenn Berry’s fantastic diagnostic queries, available at http://sqlserverperformance.wordpress.com. Items such as disk stall rates, instance and database level CPU, memory, and disk consumption metrics, and background CPU activity can all be sampled with these queries.

Now what?

Now, what do you do with this data that you are collecting? Let’s make some sense of it! Stay tuned until next time when we discuss how to start sifting through the great quantities of data that we are gathering.

If you want to get started with performance statistics SQL Servers with the help of experts, read more over at Fix Your SQL Server.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Along with the opportunities they provide, the traditional printed documentation still plays an important role by allowing to conveniently operate the required information by means of reports.

Creation of such reports, depending on the degree of complexity, can take a long time, therefore, to accelerate their creation, visual editors are traditionally used.

Some of these editors have only basic functionality. Others have extended reporting capabilities. The latter group of products includes the SQL report builder built-in to dbForge Studio for SQL Server.

The editor’s capabilities allow you to build a report according your SQL query, literally, within a few minutes. For this, it’s enough to call the Report Designer command from the main menu.

Select a report type in the opened dialog.

Select a connection and a data source.

Herewith, tables and views can be selected as a source.

In case if data, for example, from several tables should be returned by a query, a user query can be written.

After selecting a data source, specify the columns, by which the report will be built.

You should also specify how this data will be presented in the report.

At the final stage, it remains just to specify the name for the report and click the Finish button.

After the wizard is finished, the report editor opens, in which it is possible to fine-tune the appearance of the report.

To preview the report, there is the Preview button provided.

After the report is generated, it can be both printed and exported to one of the supported formats.

As you can see, creating and editing the report in SQL report builder does not take a lot of time, allowing to streamline the process and reduce development time.

Reference: Pinal Dave (http://blog.sqlauthority.com)

A Developer’s Wish

Have you ever built a database application where you are driving your navigation with the help of Data?

Have you ever realized that you are running query much faster on your local server, but when you move them to your test/staging server they are suddenly running slow?

Have you ever wished that you can sync your local machine with the master database so you can run your queries in your near real time scenario so you do not have surprise when you go live on production?

The matter of the fact is that every developer faces such situation when they wish that they can directly work on the live data instead of old historical data on their local machine. If an organization care about the experience of the developer and their product, they will always thrive to provide a tool which makes their developer’s life very easy.

Wish Granted

If you ask me what is a good tool which can help you to synchronize data across multiple databases and servers, I will suggest you to take a look at the DB Change Manager. It is a very powerful tool which can do quite a number of different tasks. Here are a few of your daily chores which you can rely on DB Change Manager to comply.

• Schema Archive and Compare
• Synchronization and Roll-back
• Schema Auto-Synchronization
• Data Compare and Synchronization
• Data Masking

Click, Click and Go!

If you are a developer and given a task to make a tool to synchronize data across two different databases, how difficult it would be creating such a tool. Honestly, very difficult as we need to take care of so many different things besides data, for example – schema, configuration, security and what not. However, when we are using the Embarcadero DB Change Manager all the problems are taken care of by the tool.

If you have followed my earlier blog post about Synchronize Schema across Multiple Environment – How to Overcome Database Migration Challenge. All the steps are very similar to that demonstration. The only difference between Schema Compare and Data Compare is the very first step when we start the job. Here is the image where I demonstrate how we can start the Data Comparison Job.

Once you start the Data Comparison Job, it will take you to a very simple wizard with few steps. Each step is extremely easy to follow and most of the time, you just have to accept everything as a default and click on go.

Once you follow the few quick steps of the wizard, you will land upon the final report page where you will notice a grid is placed with details related your change data. You can export the difference or synchronize the data across your server.

That’s it!

I wish I can write more about this entire process, but as I mentioned earlier, this tool is extremely simple to use and follow. Most of the time, I just do Click, Click and Go and I am done!

You can download Embarcadero DB Change Manager from here. I am eagerly waiting for your feedback about this product.

Reference: Pinal Dave (http://blog.sqlauthority.com)

That’s not to say however this approach doesn’t have its advantages: Once stored any media is protected by the database integrity so you don’t find yourself suffering from corrupt or lost files and media. Not to mention all your precious media is protected by the inherent security of the database. That being said most developers I know prefer avoid BLOB’ing and instead link to the media file location in the database. This brings us naturally to the core concern for Business and Developers: Coding an effective Media Management web tool.

It’s going to need an upload UI with restrictions on file types and sizes. It will need to save files in either local or cloud storage like Amazon S3 or Azure (and create an authorized download URL if you want the files stored on private storage). The upload should also save the appropriate link in the database for the frontend application as well. You may also need the capability to run additional processing on the uploaded image such as image manipulation or assign it a name or path according to additional information in the row.

Take a simple example: You have an employee with a photo. You want to save each photo with the employee name and the auto identifier key of the row so that every time an employee the employee uploads his photo it will overwrite his old photo – but ensure than employees with the same name will not overwrite each other. On top of that, you want to manage your media files with all the other textual content of the frontend app, implement additional business rules much like the photo example and edit and filter the database as well. An effective media management web tool deployed in a business environment will also require robust role security if it’s to be of any use. What initially started out as seemingly simple development task has quite quickly become a complicated, never mind time-consuming and expensive project.

Many businesses don’t have the resources to implement an effective web management solution (never mind the needed in-house expertise). It’s why the  Back& platform provides a web management for databases including media files. For one thing it’s simple: Here’s a how-to video where set up takes a few clicks.  In addition to managing media files it’s replete with additional features like smart entry forms that automatically take into consideration the relations and data types, charts reports and dashboards, data driven actions and a ton of other great options that set business up with an instant ready to use web administration for an application database. A service like this is perfect for me compared to when I have to develop an admin tool myself. I have to adjust it each time I/ add a new feature to my frontend app. With Back& however, changes in the schema are done automatically. Further adjustments like label renaming, colour schema, column orders and so much more can be easily adjusted by the user himself. Not only does it save me a ton of development time it gives my users a far better administration tool. Empowering users to do their work easier and more effectively is after all the core concern for developers and businesses alike. 

Why spend money and time away from front end business concerns developing tools when a feature rich platform like Back& is just a free click away?

Reference: Pinal Dave (http://blog.sqlauthority.com)

Here is a quick script how you can use stored procedure sp_pkeys to list the primary key. This stored procedure takes a minimum of two parameters a) name of the table, b) name of the schema.

USE AdventureWorks2012;
GO
EXEC sp_pkeys @table_name = N'Employee',
@table_owner = N'HumanResources';

Here is the output generated from the script listed above:

Here is an alternate way to list the keys as well - Two Methods to Retrieve List of Primary Keys and Foreign Keys of Database.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Here is a quick script how you can use stored procedure sp_fkeys to list all the foreign key relationship. This stored procedure takes a minimum of two parameters a) name of the table, b) name of the schema.

USE AdventureWorks2012;
GO
EXEC sp_fkeys @pktable_name = N'Employee',
@pktable_owner = N'HumanResources';

Here is the output generated from the script listed above:

Here is an alternate way to list the keys as well - Two Methods to Retrieve List of Primary Keys and Foreign Keys of Database.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Q: The series which you have written two years ago, is it still relevant to latest SQL Server?
A: Yes, absolutely. Everything which I have written earlier is still relevant to the latest version of SQL Server. The matter of the fact, most of it will remain relevant forever.

Q: Is there anyway, I can read everything together in an eBook format?
A: Yes, you can get SQL Wait Stats on Kindle over here.

Q: Is SQL Wait Stats a good logical starting point for SQL Performance Tuning?
A: I believe so. It gives you a good idea where exactly your bottleneck is in your server.

Q: I have previously not learned about SQL Wait Stats, can I start now?
A: Absolutely, Yes, here are the links:

SQL SERVER – Introduction to Wait Stats and Wait Types – Wait Type – Day 1 of 28

SQL SERVER – Signal Wait Time Introduction with Simple Example – Wait Type – Day 2 of 28

SQL SERVER – DMV – sys.dm_os_wait_stats Explanation – Wait Type – Day 3 of 28

SQL SERVER – DMV – sys.dm_os_waiting_tasks and sys.dm_exec_requests – Wait Type – Day 4 of 28

SQL SERVER – Capturing Wait Types and Wait Stats Information at Interval – Wait Type – Day 5 of 28

SQL SERVER – CXPACKET – Parallelism – Usual Solution – Wait Type – Day 6 of 28

SQL SERVER – CXPACKET – Parallelism – Advanced Solution – Wait Type – Day 7 of 28

SQL SERVER – SOS_SCHEDULER_YIELD – Wait Type – Day 8 of 28

SQL SERVER – PAGEIOLATCH_DT, PAGEIOLATCH_EX, PAGEIOLATCH_KP, PAGEIOLATCH_SH, PAGEIOLATCH_UP – Wait Type – Day 9 of 28

SQL SERVER – IO_COMPLETION – Wait Type – Day 10 of 28

SQL SERVER – ASYNC_IO_COMPLETION – Wait Type – Day 11 of 28

SQL SERVER – PAGELATCH_DT, PAGELATCH_EX, PAGELATCH_KP, PAGELATCH_SH, PAGELATCH_UP – Wait Type – Day 12 of 28

SQL SERVER – FT_IFTS_SCHEDULER_IDLE_WAIT – Full Text – Wait Type – Day 13 of 28

SQL SERVER – BACKUPIO, BACKUPBUFFER – Wait Type – Day 14 of 28

SQL SERVER – LCK_M_XXX – Wait Type – Day 15 of 28

SQL SERVER – Guest Post – Jonathan Kehayias – Wait Type – Day 16 of 28

SQL SERVER – WRITELOG – Wait Type – Day 17 of 28

SQL SERVER – LOGBUFFER – Wait Type – Day 18 of 28

SQL SERVER – PREEMPTIVE and Non-PREEMPTIVE – Wait Type – Day 19 of 28

SQL SERVER – MSQL_XP – Wait Type – Day 20 of 28

SQL SERVER – Guest Posts – Feodor Georgiev – The Context of Our Database Environment – Going Beyond the Internal SQL Server Waits – Wait Type – Day 21 of 28

SQL SERVER – Guest Post – Jacob Sebastian – Filestream – Wait Types – Wait Queues – Day 22 of 28

SQL SERVER – OLEDB – Link Server – Wait Type – Day 23 of 28

SQL SERVER – 2000 – DBCC SQLPERF(waitstats) – Wait Type – Day 24 of 28

SQL SERVER – 2011 – Wait Type – Day 25 of 28

SQL SERVER – Guest Post – Glenn Berry – Wait Type – Day 26 of 28

SQL SERVER – Best Reference – Wait Type – Day 27 of 28

SQL SERVER – Summary of Month – Wait Type – Day 28 of 28

Reference: Pinal Dave (http://blog.SQLAuthority.com)

In this episode of the Notes from the Field series database expert John Sterrett (Group Principal at Linchpin People) discusses about SQL Agent Job History in this blog post. Linchpin People are database coaches and wellness experts for a data driven world. Read the experience of John in his own words.

In this tip of the week, I want to cover a common problem I see too often in the field. Multiple times I have been asked, “Dude, where is the SQL Agent Job History?” I had a job fail but I cannot view the history and get details towards why it failed. Today, I am going to cover why the job history is most likely disappearing on you.

Typically there are two reasons why your SQL Agent Job History is missing. First, someone or a process is running sp_purge_jobhistory to purge the history. Typically, I find that this isn’t the reason why your history is disappearing randomly.  Normally, it’s the default settings for the SQL Agent Job History that is purging your history. In this week’s tip were going to focus on the cause and effect of the default settings focusing on why you might need to change them.

Setup Demo

This tip is going to utilize two SQL Agent jobs that you need to download if you want to walk through step by step through the demo. The first job is “JobHistoryDemo – Divide by Zero” and the second is “JobHistoryDemo – Batch Process.” In this demo, were going to execute a job that will fail (Divide by Zero) and then were going to execute another job (Batch Process) over a 1,000 times.  For the purpose of this demo, the batch process job just does “SELECT 1” from tempdb.

For the purpose of this demo, were going to tweak the default setting for max rows per job. By default, this is set to 100. We are modifying this to be 1000 so we only need two jobs for the purpose of this demo. We will go into more detail about SQL Agent properties later in this tip.

USE [msdb]
GO
EXEC msdb.dbo.sp_set_sqlagent_properties @jobhistory_max_rows_per_job=1000
GO


Run Demo Code

After downloading and running the scripts you will have the two new “JobHistoryDemo” SQL Agent Jobs shown below.

Run the following script it will take about 30 minutes to complete. You will notice that it will start to automatically purge the history for all the SQL Agent jobs. We will dive into why during the next section of the tip.

/* Job fails */
EXEC sp_start_job @job_name = 'JobHistoryDemo - Divide by Zero';
DECLARE @i INT;
SET @i = 1200; -- default history retention for job history records
WHILE @i > 0 BEGIN
EXEC sp_start_job @job_name ='JobHistoryDemo - Batch Process'
WAITFOR DELAY '00:00:02';
SET @i=@i-1
END


If you want to monitor the progress you can execute the following script. You will notice that the history gets purged once the “JobHistoryDemo – Divide by Zero” executes 1,000 times.

SELECT j.name, COUNT(*) Executions
FROM msdb.dbo.sysjobs j
INNER JOIN msdb.dbo.sysjobhistory h
ON j.job_id = h.job_id
GROUP BY j.name
ORDER BY Executions DESC

Here is a screen shot just right before we hit 1,000 executions.

Here is a screen shot just right after we had a 1,000 executions. Notice, that just 1,000 total SQL Agent job executions triggers the purging process. It doesn’t have to be limited to just a specific job.

Where did the history go?

Now we have the scenario described in the first paragraph of this tip. In this scenario, we have a controlled scenario where job history is missing. This can be very frustrating if you have multiple jobs executing frequently as it will seem like the data is being purged randomly.

Let’s take a look at the SQL Agent Job Activity monitor.   We can clearly see that the “JobHistoryDemo – Divide by Zero” job failed. Right click on the job and select view history.

You will notice all the job history is gone. Next we will show you most likely why you don’t see the job history.

How do we resolve the problem?

The first step to solving the problem is to identify the root cause. Reading along you might already know why the history is gone. Let’s drill down to the properties of the SQL Agent and focus on the history tab.

You will notice that the default setting is to only keep the last 1,000 history records for all the SQL Agent Jobs. The SQL Agent will also only keep 100 history records for a single job. It’s why we modified the Maximum job history rows per job property above for this demo.

Now that we have identified a root cause to this problem here is how you resolve it. Disable the limit size of job history log and utilize sp_purge_jobhistory to purge as needed to meet your storage and business requirements for maintaining agent job history.

Finally, if you need to maintain the results of SQL Agent jobs I recommend using the native functionality built-in to store message details. We will cover this in a future tip.

Are your servers running at optimal speed or are you facing any SQL Server Performance Problems? If you want to get started with the help of experts read more over here: Fix Your SQL Server.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Pinal: It is easy to find anything in Object Explorer but it is not possible to find everything in Object Explorer.

Vinod: What do you mean? Explain me with example.

Pinal: For example, if I want to find anything related to any schema or table, I can go to Object Explorer and can search about that schema and find relevant information.

However, if I want every single object where any string exists, how do I find all of those objects.

Vinod: That is easy as well, remember SQL Server is Microsoft product and will follow the simple rules of the search which you are familiar. Just try to wrap your string with % and see the magic.

Pinal: Never thought of it. Let me try.

After the conversation was over, I tried to wrap my search string with % and it works, indeed it works. SSMS retrieved every single object where the string appeared.

Thanks Vinod. It is so cool that when we have questions we can reach out to experts.

Reference: Pinal Dave (http://blog.sqlauthority.com)

However, a general rule can be pointed out for all types of DBMS: the fewer server resources are expended when executing a query, the more effective will be the query itself.

To detect bottlenecks, MySQL provides the EXPLAIN command. It will help tell you what exactly the database server does, when you execute the query.

To obtain data on executing a query, it is enough to set EXPLAIN before each targeted query. However, putting EXPLAIN for a large number of constructs can become rather tedious.

To simplify search for a problem query let us use MySQL Profiler in dbForge Studio for MySQL. It provides such commands as a Query Profiling Mode and Generate Execution Plan.

To demonstrate work with execution plan, let’s consider a small sample. Suppose we need to find vendors of PCs, but not laptops, basing on the following data:

CREATE TABLE models (
model_id INT(11) NOT NULL PRIMARY KEY,
model_type VARCHAR(10) NOT NULL
) ENGINE = INNODB;
CREATE TABLE products (
maker_id INT(11) NOT NULL,
model_id INT(11) NOT NULL,
PRIMARY KEY (maker_id, model_id),
CONSTRAINT FK_products_models_model_id
FOREIGN KEY (model_id)
REFERENCES models (model_id)
ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE = INNODB;
INSERT INTO models(model_id, model_type)
VALUES
(1, 'PC'), (2, 'Laptop'), (3, 'Tablet'),
(4, 'Phone'), (5, 'PC'), (6, 'Laptop');
INSERT INTO products(maker_id, model_id)
VALUES
(1, 1), (1, 2), (1, 3), (2, 4),
(4, 4), (2, 5), (3, 5), (3, 6);


This task can be solved quite trivial:

SELECT DISTINCT p.maker_id
FROM products AS p
JOIN models m ON p.model_id = m.model_id
WHERE m.model_type = 'PC'
AND NOT EXISTS (
SELECT p2.maker_id
FROM products AS p2
JOIN models m2 ON p2.model_id = m2.model_id
WHERE m2.model_type = 'Laptop'
AND p2.maker_id = p.maker_id
);


When designing a query, one always  ought to pay attention to re-readings from the same tables, since they bear an additional load on the database server disk subsystem:

The query can be rewritten with no re-readings, using aggregating functions:

SELECT p.maker_id
FROM products AS p
JOIN models AS m ON p.model_id = m.model_id
WHERE m.model_type IN ('PC', 'Laptop')
GROUP BY p.maker_id
HAVING COUNT(CASE WHEN m.model_type = 'PC' THEN 1 END) > 0
AND COUNT(CASE WHEN m.model_type = 'Laptop' THEN 1 END) = 0;


However, having a large number of records, JOIN to the models table can still affect performance.

To speed up the query, denormalization of the scheme can be performed by duplicating data from the model_type column of the products table:

UPDATE products AS p
JOIN models AS m ON p.model_id = m.model_id
SET p.model_type = m.model_type;


Also, pay attention to the fact that the model_type column is filtered by, so it is recommended to index it.

At that,  selection of columns, that will be included to the primary key,  should be approached carefully. Since columns from the primary key are included to the nonclustered index, this can increase database size.

As a result, selection is significantly simplified:

SELECT maker_id
FROM products
GROUP BY maker_id
HAVING COUNT(CASE WHEN model_type = 'PC' THEN 1 END) > 0
AND COUNT(CASE WHEN model_type = 'Laptop' THEN 1 END) = 0;


At this, the plan becomes simple at all:

However, the root of performance problems should not always be searched for in queries – it is very important to correctly configure database server settings. For this, the Server Variables tab can be used.

In this case, all the settings, for convenience, can be grouped according to their target facilities:

There is no perfect advice how to optimize a query.  However, using the profiler embedded into dbForge Studio for MySQL, there is a probability to detect all the performance bottlenecks.

Optimal settings for MySQL server can also vary depending on the specificity of performed tasks. Therefore, detailed consideration of the server configuration process will be revealed in a separate article.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Inceptions of Notes from the Field

There are two aspects of any technology 1) Theoretical Knowledge and 2) Practical Knowledge. I have been blogging over 7 years over here and over the years, I have shared my theoretical and practical knowledge over here on this blog. However, I am primarily contributor to my blog and hence there was a monotonous tone to this blog. I always wanted to bring the perspective of other Database Experts to the blog readers. In the real world, we have thousands of the servers deployed, and each server has its own stories to tell. When I have worked on performance tuning projects, I have realized that it is not possible to figure out everything by a single expert. When we have more than one opinion for any server, it helps us to figure out what is the best possible solution for that server. Following the same logic, I have decided to reach out to few of my fellow industry experts to help me write a series Notes from the Field.

Contest

Here are details of quick contest. We have so far published 16 different episodes of Notes from the Field. We believe each of them are extremely valuable and have lots of interesting information from the real world. We personally cannot decide which one is better than another, as they all are exceptional posts. With said that when we check the statistics of each of the blog post, we see different interest level for each of them. We decided to have a quick contest for all of you, which will help us to come up with better notes from the field.

Answer following question in the comments – What would you like to learn next in the Notes from the Field series? 

One lucky winner will get USD 50 Amazon gift card. We will select lucky winner based on the random algorithm.

Roundup from Notes from the Field

My friends at Linchpin People are expert at Anything and Everything about SQL. You can reach out to them when you need an urgent care specialist for your database. If you believe your servers are healthy, that is a great thing, but it does not hurt to read the advice of the experts in their own words.

Here is the roundup from Notes from the Field. Each of the note is shared by experts at Linchpin People.

• Transaction Log Full – Transaction Log Larger than Data File – Notes from the Field #001
• Root Cause to Performance Problems – Notes from the Field #002
• vCPUs – How Many Are Too Many CPU for SQL Server Virtualization ? – Notes from the Field #003
• SQL Server Configuration Checking – A Must Do for Every DBA – Notes from the Field #004
• Finding Frequently Running Query and Elapsed Time – Notes from the Field #005
• Optimal Memory Settings for SQL Server – Notes from the Field #006
• SQL Server Alerts and How to Use Them – Notes from the Field #007
• Overall SQL Server Performance – Not Just Query Performance – Notes from the Field #008
• Tools for Proactive DBAs – Central Management Server – Notes from the Field #009
• Auditors, Passwords and Exceptions – Notes from the Field #010
• Check for Database Integrity – Notes from the Field #011
• Tools for Proactive DBAs – Policy Based Management – Notes from the Field #012
• Virtualized SQL Server Performance and Storage System – Notes from the Field #013
• Learning SSIS – Where Do I Start? – Notes from the Field #014
• Expanding Your Skills – Notes from the Field #015

• Finding the Last Backup for All Databases – Notes from the Field #016

If you are busy with Christmas, I suggest you bookmark this post and read it later on in the new year.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Windows Azure SQL Database and SQL Server – Performance and Scalability Compared and Contrasted 
This white paper is recently updated and one of the most important article to know the difference between SQL Server and Windows Azure SQL Database (SQL Database). There are lots of similarities and differences between SQL Database and SQL Server and they are often confused among each other. This article takes on the vary subject and explains the differences between them very clearly and with the help of examples. Here are few important sections of this white paper.

• Standard Application Scenarios
• Several Common Expections
• Classic Patterns
• General Performance Evaluation
• Evalute Performance of SQL Database

Click here to read Windows Azure SQL Database and SQL Server – Performance and Scalability Compared and Contrasted 

Reference: Pinal Dave (http://blog.sqlauthority.com)