SQL SERVER – COLUMNSTORE and COLUMNSTORE_ARCHIVE data compression

In earlier blogs on “Fundamentals of Columnstore Index”, I took a simple approach on taking a tour of how to create a simple Columnstore index. Having said that, SQL Server 2014 supports columnstore and columnstore archival compression for columnstore tables and indexes. Columnstore tables and indexes are always stored with columnstore compression. Most data warehouses have some data that is frequently accessed and some that is accessed more infrequently.

You might be wondering the use case scenario for these commands. For example, in reality the data may be partitioned by date and the most recent data is accessed much more frequently than older data. In such cases the older data can benefit from additional compression at the cost of slower query performance. I talk to a lot of customers about this special feature:

  • To enable this scenario SQL Server added support for archival compression of SQL Server column stores from SQL Server 2014.
  • The archival compression option is enabled on a per-table or partition (object) basis; a column store can contain objects with archival compression applied and objects without. To allow for easy extensibility of existing on-disk structures, archival compression is implemented as an extra stream compression layer on top of Columnstore index implementation.
  • The Columnstore compression implementation transparently compresses the bytes being written to disk during the column store serialization process and transparently decompresses them during the deserialization process. Stream decompression is always applied when data is read from disk. Data is not cached in memory with stream compression.

The further reduction obtained by archival compression is substantial, ranging anywhere from 25% to 60%+ depending on the data. These are rough numbers based on what I have seen at customer interactions in the past.

  • Use COLUMNSTORE_ARCHIVE data compression to compress columnstore data with archival compression.

Let us look at how this can be achieved using TSQL commands.

To Apply Compression:

The simple TSQL commands to play with ColumnStore and columnStore Archive bits in single ALTER script:

-- Adding Archive bit for single Partition
ALTER TABLE tbl_myColumnStore
REBUILD PARTITION
= 1 WITH (DATA_COMPRESSION =  COLUMNSTORE_ARCHIVE) ;
-- Adding Archive bit for all Partition
ALTER TABLE tbl_myColumnStore
REBUILD PARTITION
= ALL WITH (DATA_COMPRESSION =  COLUMNSTORE_ARCHIVE) ;
-- Adding Columnstore for all parition and Archive for few Partition
ALTER TABLE tbl_myColumnStore
REBUILD PARTITION
= ALL WITH (DATA_COMPRESSION =  COLUMNSTORE_ARCHIVE ON PARTITIONS (1,3)) ;

Use COLUMNSTORE data compression to decompress archival compression. This resulting data will continue to be compressed with columnstore compression.

To Remove the Archive Compression

Similar to above commands, we can also use to remove our archival compression on specific partitions or on objects. A representative command looks like:

-- Turning Archive bit off to Columnstore for all Partitions
ALTER TABLE tbl_myColumnStore
REBUILD PARTITION
= ALL WITH (DATA_COMPRESSION =  COLUMNSTORE) ;

As I wrap up this blog, I would like to know how many of you out there are using the archival columnstore in your environments. How much compression are you getting and what is the space savings because of this? I would love to hear from your experience as we learn together.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Interview Question of the Week #008 – Write Scripts to Convert String to Title Case or Proper Case

Just received a question in email:

“I just got out of the interview and I was asked to write scripts to convert a string to Title Case. I was also asked to refer to the website over here to validate my answer after I complete writing script.

Do you think it is not fair to ask such questions in an interview?”

Well, this time the question is not about writing scripts, but more about if this kind of questions is a good question for interview or not. The question is a bit open ended and my answer can be vague or full of philosophy of it depends. However, I will try to answer it bit more honest and clear.

I believe all questions are good question in the interview. It is never about the question’s validity, but more about quality of answer and attitude decides the capability of the user. If I ever face a situation where I am asked questions to write script to convert a string to title case, I would happily sit down and attempt to write it if I want the job and I am up for the challenge. If I do not know how to write it, I will honestly say I do not know and let the interviewer decide the next step. If you believe this kind of question is inappropriate for the job which you have applied, well, you can always mention that after you solve it.

Well – in my early career, I have written similar script and I have posted that on this blog post well. You can refer that in this blog post – SQL SERVER – UDF – Function to Convert Text String to Title Case – Proper Case. I strongly suggest that you read the comments as those are packed with the wealth of the information. Here is the script from the blog post.

CREATE FUNCTION udf_TitleCase (@InputString VARCHAR(4000) )
RETURNS VARCHAR(4000)
AS
BEGIN
DECLARE
@Index INT
DECLARE
@Char CHAR(1)
DECLARE @OutputString VARCHAR(255)
SET @OutputString = LOWER(@InputString)
SET @Index = 2
SET @OutputString =
STUFF(@OutputString, 1, 1,UPPER(SUBSTRING(@InputString,1,1)))
WHILE @Index <= LEN(@InputString)
BEGIN
SET
@Char = SUBSTRING(@InputString, @Index, 1)
IF @Char IN (' ', ';', ':', '!', '?', ',', '.', '_', '-', '/', '&','''','(')
IF @Index + 1 <= LEN(@InputString)
BEGIN
IF
@Char != ''''
OR
UPPER(SUBSTRING(@InputString, @Index + 1, 1)) != 'S'
SET @OutputString =
STUFF(@OutputString, @Index + 1, 1,UPPER(SUBSTRING(@InputString, @Index + 1, 1)))
END
SET
@Index = @Index + 1
END
RETURN
ISNULL(@OutputString,'')
END

You can use this script as follows:

SELECT dbo.udf_TitleCase('This function will convert this string to title case!')

Reference: Pinal Dave (http://blog.sqlauthority.com)

SQL SERVER – Script to Find Strings with Same Characters

One of my blog followers asked me if it is possible to check whether all the characters of a column value is same. For example if there are values like KKK and KKT, the value KKK should be the output as all characters are same.

There is a method without using WHILE Loop

Create the following data set

CREATE TABLE #TEMP(COL VARCHAR(20))
INSERT INTO #TEMP
SELECT 'AAAAA' AS DATA UNION ALL
SELECT 'NKKKKKKKKKK' UNION ALL
SELECT '32000000' UNION ALL
SELECT '11111111' UNION ALL
SELECT '1002' UNION ALL
SELECT 'OOOOO'UNION ALL
SELECT 'M'

Run the following code

SELECT
COL FROM #TEMP
WHERE
REPLACE(COL,LEFT(COL,1),'')=''

The result is

COL
--------------------
AAAAA
11111111
OOOOO
M

The logic is to get first character and replace that character by empty string in the column. If the final result is empty string, it means that all characters are same. Otherwise all characters are not same.

Reference: Pinal Dave (http://blog.sqlauthority.com)

SQL SERVER – FIX – Error: 905, Severity: 21, State: 1 – Database ‘xxx’ cannot be started in this edition of SQL Server because it contains a partition function ‘xxx’.

Recently one of my blog reader contacted me via mail and told that after upgrading from Enterprise Evaluation Edition to Standard Edition, the database was not accessible. To troubleshoot, I have asked them to share SQL Server ERRORLOG. There are various ways to read Errorlog. Here is the method to read Errorlog:

  • In Object Explorer, expand a server, expand Management, and then expand SQL Server Logs.
  • Right-click a log and click View SQL Server Log.

Other ways to get Errorlog are explained in detail by Balmukund (b|t) on his blog (Help : Where is SQL Server ErrorLog?)

In the Errorlog, I asked him to look for any error related to the database which is not accessible and soon they shared below messages:

2015-02-20 13:47:36.65 spid7s      Error: 905, Severity: 21, State: 1.
2015-02-20 13:47:36.65 spid7s      Database ‘My_Database’ cannot be started in this edition of SQL Server because it contains a partition function ‘myRangePF1′. Only Enterprise edition of SQL Server supports partitioning.
2015-02-20 13:47:36.65 spid7s      Error: 933, Severity: 21, State: 1.
2015-02-20 13:47:36.65 spid7s      Database ‘My_Database’ cannot be started because some of the database functionality is not available in the current edition of SQL Server.

Explanation

From the error we can see that the database is using partition function which is enterprise only feature. Here is the link which has edition comparison.  The database contains one or more partitioned tables or indexes. Standard edition of SQL Server cannot use partitioning. Therefore, the database cannot be started correctly. Partitioned tables and indexes are not available in every edition of Microsoft SQL Server. Books online topic “Features Supported by the Editions of SQL Server 2012” has complete list of feature.

Solution

I asked him to restore the database MDF and LDF files to another instance which is either developer, enterprise or enterprise evaluation edition. Then get rid of the features which are enterprise only feature. They can easily be found using the DMV sys.dm_db_persisted_sku_features

USE <DatabsaeName> -- Change the name to the database where we need to check
GO
SELECT *
FROM sys.dm_db_persisted_sku_features

Note that the query has to be run in the database which is under question, not the master database. Here are the possible values on SQL Server 2014.

  • TransparentDataEncryption
  • Compression
  • ChangeCapture
  • ColumnStoreIndex
  • InMemoryOLTP
  • Partitioning

Once the object referring to that feature is removed, the backup can be taken and restored on “lower” edition of SQL Server.

Here is another error which can be received while restoring database using compression

Msg 909, Level 21, State 1, Line 1
Database ‘SqlAuthority’ cannot be started in this edition of SQL Server because part or all of object ‘PageCompressionTest’ is enabled with data compression or vardecimal storage format. Data compression and vardecimal storage format are only supported on SQL Server Enterprise Edition.

I always advice to run the DMV and check the features whenever there is a downgrade of edition. Even though evaluation to standard is an upgrade but I would call it as downgrade because there are few feature of enterprise editions would not be available in standard edition.

Have you ever encountered any downgrade scenarios and got some errors?

Reference: Pinal Dave (http://blog.sqlauthority.com)

SQL SERVER – Finding Top Offenders in SQL Server 2012 – Notes from the Field #068

[Note from Pinal]: This is a 68th episode of Notes from the Fields series. Performance tuning gets easier with SQL Server 2012. For the first time, you can capture a workload and find your top offenders inside of Management Studio without having to write a single line of T-SQL or use any third party tools.

In this episode of the Notes from the Field series database expert John Sterrett (Group Principal at Linchpin People) explains a very common issue DBAs and Developer faces in their career – how to find top offenders in SQL Server. Linchpin People are database coaches and wellness experts for a data driven world. Read the experience of John in his own words.


One of the biggest mistakes with performance tuning is not knowing where to start.  Sometimes performance tuning can be like finding a needle in a haystack.  This is because you might not have a great process to help you find your top offenders. In this short five minute video below you will be able to utilize extended events to aggregate and group completed statements so you can find your top offenders without having to write any T-SQL code.

Are your servers running at optimal speed or are you facing any SQL Server Performance Problems? If you want to get started with the help of experts read more over here: Fix Your SQL Server.

Reference: Pinal Dave (http://blog.sqlauthority.com)

SQL SERVER – Notes and Observations on ReadOnly Databases in SQL Server

In the past couple of weeks, I have written few blogs that revolve around utilizing “readonly” databases. It was fun working on this special case scenario that I had stumbled upon an interesting set of questions from my blog readers. These were not clearly called out in my previous blogs, hence thought of writing them down the responses I had given with examples in this blog. Some of the questions I have been asked were:

  • Can I create temporary tables or global temporary tables when working with ReadOnly databases?
  • Since the database is in ReadOnly mode, we cannot insert any values into the database. How about creation of tables when the DB is marked as ReadOnly?
  • Can we create stored procedures when working with ReadOnly Databases?
  • I have a large database and I see a lot of free space, can I shrink the DB when it is marked as ReadOnly? Is this allowed?

Some of these questions are interesting and require a small script to validate our understanding. A rule of thumb at this point is, this is a ReadOnly database and we need to know we cannot do anything with this database. Temp tables are created in the context of tempdb database and the readonly attribute doesn’t apply to objects created that way.

Next let me build the script to demystify each of the questions asked above:

CREATE DATABASE [ReadOnlyDB]
CONTAINMENT
= NONE
ON  PRIMARY
( NAME = N'ReadOnlyDB', FILENAME = N'C:\Temp\ReadOnlyDB.mdf' , SIZE = 4024KB , FILEGROWTH = 1024KB )
LOG ON
( NAME = N'ReadOnlyDB_log', FILENAME = N'C:\Temp\ReadOnlyDB_log.ldf' , SIZE = 20480KB , FILEGROWTH = 10%)
GO
USE MASTER
GO
ALTER DATABASE [ReadOnlyDB] SET READ_ONLY
GO

The basic script above is creating our database and then setting the attribute to ReadOnly. Let us start our various tests to validate our understanding.

USE ReadOnlyDB
GO
-- Creating our table
CREATE TABLE tbl_SQLAuth (id INT, Longname CHAR(8000))
GO

As per our understanding, this must raise an error as shown below:

Msg 3906, Level 16, State 1, Line 15

Failed to update database “ReadOnlyDB” because the database is read-only.

Temp Tables: As discussed above, the below query in the ReadOnlyDB context willnot raise any error.

-- Creating our Temp Tables
CREATE TABLE #t1 (i INT)
GO
DROP TABLE #t1
GO
-- Creating our Global Temp Tables
CREATE TABLE ##t1 (i INT)
GO
DROP TABLE ##t1
GO

Stored procedure creation: If we try to create any objects in our ReadOnlyDB, we will get the same error of 3906.

– Create the stored procedure inside ReadOnlyDB

-- Create the stored procedure inside ReadOnlyDB
CREATE PROC prc1
AS
BEGIN
SELECT
1
END
GO

Msg 3906, Level 16, State 1, Procedure prc1, Line 30

Failed to update database “ReadOnlyDB” because the database is read-only.

Shrink Database command: Shrink database question was an interesting one for me. But on second look, this is logical. Since shrink database will try to alter the header in our mdf file, in the readonly mode that is not allowed. Hence we will get the error.

DBCC SHRINKDATABASE (ReadOnlyDB, TRUNCATEONLY);
GO

Msg 7992, Level 16, State 1, Line 4

Cannot shrink ‘read only’ database ‘ReadOnlyDB’.

All these errors are simple yet questions that come to many of us. This blog post is a dedication to those who took time in writing a line to me even after close to 7+ years of blogging. I get to learn from each one of you out there. SQL Server is an ocean and you make me complete.

Reference: Pinal Dave (http://blog.sqlauthority.com)

SQL SERVER – Cloud Based Data Integration Made Easy – A Real World Scenario

If you are a DBA, once in a while, you will have a situation where you end up with some tasks which will be boring and annoying. Trust me in my life, I often come across similar scenarios quite often. Here is one such tasks I came across a few days ago.

A Task, I would like to Automate

Just the other day I was assigned a task where I had to take a CSV file from my network and insert into SQL Server which was hosted in remote locations. Well, you may think it as a pretty easy task and I agree with you this is very easy task. The challenge is not about its difficulty, but the challenge was rather about the entire process and my interest. The part which annoyed me the most was that I have to do this every day at 4 PM.

This means, every day I must be at my desk at 4 PM and take a file from the network and upload to remote SQL Server. What about weekends? What about when I have to step away from my desk at 4 PM? What about the situation, when I am busy doing something much more important than this task? Well, as I said, more than task, I have been just one-place with the routine which was associated with it. In simple words, this was an ETL task which needed to be automated, but I can’t depend on my machine always. I have to find a solution which was cloud based and runs efficiently.

Skyvia at Rescue

I was sitting miffed in office and suddenly I remembered that last year I blogged about the tool Skyvia. Here is the blog post Integrate Your Data with Skyvia – Cloud ETL Solution. I quickly referred to my own blog, post and realized I should give Skyvia a try.

What is Skyvia?

Skyvia is a powerful solution for cloud data integration (ETL) and backup. It is a complete online solution and does not require any local software installed except for a web browser. In Skyvia we can create integration packages that define the operations and then we can run them or schedule for automatic execution. An integrated package is a set of data ETL operations that can be stored for future use, executed, modified, or deleted. Skyvia provides several kinds of packages for different integration scenarios. They are Import, Export, Replication, and Synchronization packages.

How did I do it?

Well, here are few of the screenshots of the task which I was assigned.

First, I checked if the table where I have to export data exists or not. As the table was already created, I quickly checked if it contained data or not. The table contained no data.

Next we will open the Skyvia web interface. It is pretty simple and it will list three options on the left bar. We will click Integration there.

In the Integration section, click Create Now under Data Import.

In the data integration screen we will be presented with various options. We will load the CSV file from an FTP server, so we select source as a CSV from FTP and target as SQL Server.

As we will be connecting SQL Server for the very first time we will be creating new connection and that is pretty straight forward procedure.

Then we will configure an FTP connection

Next we will configure CSV options. Here will be providing various options, but in our case all the default options were good enough for us to move next.

Right after that we will select the target table. In our case the target table is actor table.

The next screen will present mapping and we will one more time review various mapping options. We will make sure that all the source and target columns maps correctly.

When we click finish it will bring up the following screen.

Click on Save and now we are back on the following screen. Over here we can execute our task and see if it works or not. Click on the RUN button on the right side of the screen.

In my case the task ran successfully and it shows that it has inserted 200 rows successfully. The time taken to complete this entire task was 35 seconds and it depends on my network connection to the destination server.

We can execute the same select statement which we had executed earlier and see if the table contains the valid data.

Once we commit that our task has worked successfully, we can create a daily schedule.

That’s it! I am done.

Now every day at specific time the task will automatically execute and will log history.

Next Action Items

Team Devart has created Skyvia a feature rich service. One has to experiment with various different options to fully see the vast capability of this amazing product. Here are few things you can consider doing it. Here is the link where you can sign up for Skyvia for totally FREE. Next I will be trying out Skyvia with Salesforce. Skyvia is an all-in-one cloud solution for various Salesforce data integration scenarios. In addition to the standard Salesforce data loader functionality – data import and data export – it offers powerful data replication and synchronization tools and backup functionality with preview and export of backed up data, data search, viewing data changes between backups, and one-click restore.

Sign up for Skyvia for totally FREE.

Reference: Pinal Dave (http://blog.sqlauthority.com)