So what’s the approach?

First, expressor and Melissa Data created a partnership through which expressor Studio users could obtain a key to use the Melissa Data WebSmart Suite .

Second, expressor created a collection of operator templates, one for each service in the WebSmart Suite.

And third, expressor packaged this collection of operator templates into an expressor library that you can easily integrate into any expressor library or project.

Let’s see how this all comes together.

Once you register with on the WEBSMARTexpress Web site, you are given the URL to download the expressor operator template library – WEBSMARTexpress.zip.  Do not extract this archive file.  Rather you simply import the library into an expressor workspace.

• Open the workspace.
• Select the Studio > Import Projects… menu item.
• In the first window of the Import Projects wizard, browse to the file system location where you saved the operator template archive file and select the file for import.
• In the wizard’s second window, select the LIB_WebSmart library and click Finish.  The library is added to your workspace.

If you now expand the library, you will see that it includes composite types, the operator templates, and some Datascript modules.

Each composite type describes the return from one of the WebSmart services.  The scripting that provides the processing logic for each operator template is contained in a dedicated Datascript module.  You will not need to edit the coding within these Datascript modules but you will need to enter your Melissa Data key into the module MDVariables before you can access the WebSmart services.

Double click on the MDVariables entry under the Datascript Modules folder to open this module in the script editor and on line 13 replace num with your key and then save and close the editor.

Now create another library or project that will include the dataflow in which you will use one of the operator templates.  Be certain to reference the LIB_WebSmart library.

When you are laying out the dataflow, note that the operator templates are listed on the Templates tab within the Transformers grouping of operators.  To use one of the operator templates, simply drag it onto the dataflow.  When you are configuring this operator, do not change any of the settings on its Properties panel, but you must open the Rules Editor.  In the screen shots that follow, I’m working with the email validation service, but usage of all the operator templates is identical.

At a minimum, I must connect the incoming attribute that contains the email address to be verified to the Rule_1 input parameter Email.  The outgoing parameters are already bound to their corresponding attributes in the output record.  The structure of the output is predefined by the composite type EmailResponse, which was included in the WEBSMART library.

If I also want to include other incoming attributes in the record emitted by this operator, I must drag them into the Rule_1 input parameter listing, add corresponding attributes to the Output panel, and drag the newly added output attributes into the Rule_1 output parameter listing.

Note that to handle these additional values I did not need to edit the coding in either the operator template or its associated Datascript Module.

And that’s all there is to it.  As your expressor dataflow executes, the email address in each record will be submitted to the WebSmart service for validation and the detailed information returned by the Web service will be added to the record emitted by the operator.

Through their extension framework and operator template functionality, expressor software has provided two very effective approaches for extending the functionality of an expressor Studio installation.  The extension framework provides a mechanism to add new functionality to Studio while the operator template/Datascript Module approach is a simple way to reuse logic that you have previously developed.  And the partnership with Melissa Data demonstrates how straight-forward it is to integrate expressor with functionality or services provided by partner organizations.

expressor is making this integration with Melissa Data available as part of their Desktop and Standard Editions.  You can download a trial of the Desktop Edition by clicking this link.  As always, I am impressed how quickly expressor evolves their product line and adds very valuable new capabilities to it every few months.   I encourage you to download their Desktop Edition trial.

Reference: Pinal Dave (http://blog.sqlauthority.com)

You might know that RDBMS have automatic support for loading CSV files into tables – but it is not quite as easy as one click of a button.  First of all, most databases have a command line interface and you need the file and configuration script in order to load up.  You also need to know enough to write the script – which for novices can be extremely daunting.  On top of all this, if you work with more than one type of RDBMS, you need to know the ins and outs of uploading and writing script for more than one program.

So you might begin to see how useful CSVexpress.com might be!  There are many other tools that enable uploading files to a database.  They can be very fancy – some can generate configuration files automatically, others load the data directly.  Again, novices will be able to tell you why these aren’t the most useful programs in the world.  You see, these programs were created with SQL in mind, not for uploading data.  If you don’t have large amounts of data to upload, getting the configurations right can be a long process and you will have to check the code that is generated yourself.  Not exactly “easy to use” for novices.

That makes CSVexpress.com one of the best new tools available for everyone – but especially people who don’t want to learn a lot of new material all at once.  CSVexpress has an easy to navigate graphical user interface and no scripting or coding is required.  There are built-in constraints and data validations, and you can configure transforms and reject records right there on the screen.  But the best thing of all – it’s free!

That’s right, you can download CSVexpress for free from www.csvexpress.com and start easily uploading and configuring riles almost immediately.  If you’re currently happy with your method of data configuration, keep up with the good work.  For the rest of us, there’s CSVexpress.com.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

SafePeak is in business of accelerating performance of SQL Server applications, as well as their scalability, without making code changes to the applications or to the databases. SafePeak performs database dynamic caching, by caching in memory result sets of queries and stored procedures while keeping all those cache correct and up to date. Cached queries are retrieved from the SafePeak RAM in microsecond speed and not send to the SQL Server. The application gets much faster results (100-500 micro seconds), the load on the SQL Server is reduced (less CPU and IO) and the application or the infrastructure gets better scalability.

SafePeak solution is hosted either within your cloud servers, hosted servers or your enterprise servers, as part of the application architecture. Connection of the application is done via change of connection strings or adding reroute line in the c:\windows\system32\drivers\etc\hosts file on all application servers.

For those who would like to learn more on SafePeak architecture and how it works, I suggest to read this vendor’s webpage: SafePeak Architecture.

More interesting new features in SafePeak 2.1

In my previous review of SafePeak new I covered the first 4 things I noticed in the new SafePeak (check out my article “SQLAuthority News – SafePeak Releases a Major Update: SafePeak version 2.1 for SQL Server Performance Acceleration”):

1. Cache setup and fine-tuning – a critical part for getting good caching results
2. Database templates
3. Choosing which database to cache
4. Monitoring and analysis options by SafePeak

Since then I had a chance to play with SafePeak some more and here is what I found.

5. Analysis of SQL Performance (present and history):

In SafePeak v.2.1 the tools for understanding of performance became more comprehensive. Every 15 minutes SafePeak creates and updates various performance statistics. Each query (or a procedure execute) that arrives to SafePeak gets a SQL pattern, and after it is used again there are statistics for such pattern. An important part of this product is that it understands the dependencies of every pattern (list of tables, views, user defined functions and procs). From this understanding SafePeak creates important analysis information on performance of every object: response time from the database, response time from SafePeak cache, average response time, percent of traffic and break down of behavior. One of the interesting things this behavior column shows is how often the object is actually pdated. The break down analysis allows knowing the above information for: queries and procedures, tables, views, databases and even instances level.

The data is show now on all arriving queries, both read queries (that can be cached), but also any types of updates like DMLs, DDLs, DCLs, and even session settings queries.

The stats are being updated every 15 minutes and SafePeak dashboard allows going back in time and investigating what happened within any time frame.

6. Logon trigger, for making sure nothing corrupts SafePeak cache data

If you have an application with many parts, many servers many possible locations that can actually update the database, or the SQL Server is accessible to many DBAs or software engineers, each can access some database directly and do some changes without going thru SafePeak – this can create a potential corruption of the data stored in SafePeak cache. To make sure SafePeak cache is correct it needs to get all updates to arrive to SafePeak, and if a DBA will access the database directly and do some changes, for example, then SafePeak will simply not know about it and will not clean SafePeak cache.

In the new version, SafePeak brought a new feature called “Logon Trigger” to solve the above challenge. By special click of a button SafePeak can deploy a special server logon trigger (with a CLR object) on your SQL Server that actually monitors all connections and informs SafePeak on any connection that is coming not from SafePeak. In SafePeak dashboard there is an interface that allows to control which logins can be ignored based on login names and IPs, while the rest will invoke cache cleanup of SafePeak and actually locks SafePeak cache until this connection will not be closed. Important to note, that this does not interrupt any logins, only informs SafePeak on such connection.

On the Dashboard screen in SafePeak you will be able to see those connections and then decide what to do with them.

Configuration of this feature in SafePeak dashboard can be done here: Settings -> SQL instances management -> click on instance -> Logon Trigger tab.

Other features:

7. User management

ability to grant permissions to someone without changing its configuration and only use SafePeak as performance analysis tool.

8. Better reports

for analysis of performance using 15 minute resolution charts.

9. Caching of client cursors

10. Support for IPv6

Summary

SafePeak is a great SQL Server performance acceleration solution for users who want immediate results for sites with performance, scalability and peak spikes challenges. Especially if your apps are packaged or 3^rd party, since no code changes are done. SafePeak can significantly increase response times, by reducing network roundtrip to the database, decreasing CPU resource usage, eliminating I/O and storage access.

SafePeak team provides a free fully functional trial www.safepeak.com/download and actually provides a one-on-one assistance during such trial.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

I have used this tool extensively since 2009 at numerous occasion and found it to be very impressive. What separates it from the crowd the most – it is it’s apparent simplicity and speed. When I install SQLBackupAndFTP and configure backups – all in 1 or 2 minutes, my clients are always impressed.

To put it simply, SQLBackupAndFTP is MS SQL Server backup software that performs these tasks:

• Backup SQL Server Database
• Zip the backups
• Encrypt the backups
• FTP the backups to remote FTP server
• Move file to local area network
• Send final status of tasks in email.
• Allows to monitor the history of jobs through the web

SQLBackupAndFTP comes in Free and Paid versions (starting from $29) – see this version comparison chart, but even Free version is fully functional for unlimited ad hoc backups or for scheduled backups of up to two databases – it will be sufficient for many small customers.

But enough theory, let us now see how we can take backup and move it to the desired folder.

Image 1: You start from the main form and  all the settings are very easy as described in the following image. From database selection to job scheduling, you can configure everything on one screen.

On this form you would usually do the following:

1. Press “Connect to SQL Server” button to connect to your SQL Server instance
2. Check databases in the list to select databases to backup
3. If you need to store backups in a Local/Network folder – fill out the path to this folder
4. If you need to send backups to FTP – fill out your FTP server settings
5. Enter you email to receive job notification e-mails
6. Fill out the time when a scheduled daily backup job should run
7. Press “Run Now” button to test your backups

Note that the above covers just the basic scenario when you need full backups once a day. If you need Differential or Transaction Log backups or a schedule more complicated than once per day – press “Advanced Schedule”.

Image 2: Advanced Backup Schedule. Again, what I love about this program – it is how simply it allows you to resolve even quite complicated tasks. Your can click the sample scenario link that matches you needs the most – like “Daily 4 times” and everything is set – you’ll get one full and 3 Differential backups per day. Or select any other scenario that suits you.

Image 3: Advanced Settings screen is optional and allows you set encryption, compression levels, backup file name, temporary folders, if you want to use web logs (more about it below) and more

Image 4: If you checked “Use web log” in Advanced Settings, the log of your jobs is being saved on the web on sqlbackupandftp.com site (see weblog demo) and you can see the history of these jobs (even collected from multiple servers) in a single page – very interesting feature that can be very handy for large organizations

A detailed configuration tutorial is available here if you are interested.

Now, a question that might baffle your mind is – how this tool is better than maintenance tasks available in SQL Server. Well, here goes the answer. Maintenance tasks are easy to set up for backups; however, SQLBackupAndFTP have integrated solution for encryption, FTP and email which make it superior to maintenance tasks in every aspect.

I would like to present my unbiased view about this tool. At the same time, I want to give my readers a broad perspective about backup tools in general.  On comparing this tool with other backup tools I have experienced that it is not the fanciest tool to manage backups or check their health. This is one tool that smoothly performs the job in a very few steps compared to native SQL Server backup, which takes numerous steps to execute the similar job.

Image 5: Along with SQLBackupAndFTP setup gives you the option to install “One-Click SQL Restore” (you can install it stand-alone too) – a basic tool for restoring just Full backups.

However basic, you can drag-and-drop on it the zip file created by SQLBackupAndFTP, it unzips the BAK file if necessary, connects to the SQL server on the start, selects the right database, it is smart enough to restart the server to drop open connections if necessary – very handy for developers who need to restore databases often.

While testing SQLBackupAndFTP I noticed some limitations. It only works reliably on local SQL Server instances. In other words it has to be installed on the SQL server itself. For remote servers it uses scripting which is less reliable. This limitations is actually inherent in SQL server itself as BACKUP DATABASE command  creates backup not on the client, but on the server itself.

This tool is compatible with almost all the known SQL Server versions. It works with SQL Server 2008 (all versions) and many of the previous versions. It is especially useful for SQL Server Express 2005 and SQL Server Express 2008, as they lack built in tools for backup.

SQLBackupAndFTP is rapidly evolving since I first started using it two years ago. I spoke to developers and it seems that their immediate goal is to include cloud storage support – users will be able to backup directly to Microsoft SkyDrive, Amazon S3, DropBox, Box.net and others – very exciting development.

I strongly recommend this tool to all the DBAs. They must absolutely try it as it is free and does exactly what it promises. You can download your free copy of the tool from here.

Please share your experience about using this tool. I am eager to receive your feedback regarding this article. I will also give print book of my SQL Server Wait Stats book to two individual who share their feedback about this product. The book will be sent to any country where Amazon delivers the book.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Given how important performance is these days among SQL Server critical applications, I was excited to look into a new product by SafePeak Technologies (www.SafePeak.com) that aims to immediately resolve, in a plug-and-play way, the performance, scalability and peaks challenges of SQL Server applications on the Cloud, hosting servers and enterprise data centers.

Click to Enlarge

While most SQL Server shops ask me about performance monitoring and alert solutions, to allow them to identify and analyze performance bottlenecks, this unique product actually can resolve them and increase performance in dramatic ways.

SafePeak is a caching proxy that resides between the application and the database. It is unique in the way it requires neither code changes nor database changes. The only change required is routing the application through SafePeak instead of the database.

The benefits of such a solution are obvious: quick integration, immediate ROI, no complex projects and most importantly, minimal downtime for production systems.

As far as I know, this is the only true plug & play caching solution available on the market today and it can provide instant acceleration to any Cloud, hosted or business SQL Server application.

The concept

The idea behind SafePeak is that simple – repeatable queries should not be executed against the database, the result sets can be stored in RAM and can be returned immediately, in fractions of a millisecond, without the need to validate the syntax, calculate the execution plan, read IO blocks from the disk and perform data aggregation.

On transactional applications there are many such repeatable queries and SafePeak can reduce the overhead of executing them against the database over and over again.

How Does SafePeak work?

SafePeak’s solution to the performance problem is to store the result sets returned by the database in RAM, as is. When a recurring query or SP execution with the same parameters is passed through SafePeak, it will reply from RAM in a fraction of a millisecond instead of sending the query to the database, thus significantly reducing resources from the database server and the database storage.

In order to achieve this behavior, SafePeak reads and analyzes the TDS protocol (the protocol between the application and the SQL Server), making it transparent to any application working with SQL Server.

Data integrity, reliability and high availability

The main challenge with database caching is to understand in real time when data is no longer valid and should be evicted from the cache. Data can become obsolete for numerous reasons: DML, DDL, executing complex stored procedures, triggers etc.

SafePeak solved this complicated challenge with patent-pending algorithms with three parts: 1) Automatic analysis algorithm of the database schema and creation of a relations graph between all the objects in the database: tables, views, triggers, functions, stored procedures, etc.; 2) A self-adaptive algorithm that studies the application traffic, creates patterns of queries and understands their dependant objects; 3) Real-time analysis of the TDS and the T-SQL queries upon arrival of data modification event, and then flushing or refreshing in real-time the relevant cache items results.

The data on the SQL Server is always up-to-date and the cache items that remain in SafePeak memory are always up to date.

SafePeak’s team made sure to protect the application and database from potential glitches of software and hardware. Basically, in single-mode installation the worst case of any failure the traffic will be sent directly to the database server. In cluster installation, SafePeak second, the passive server takes over (Virtual IP) and becomes the active one.

Installation and configuration

SafePeak Installation is quick and simple, and is completed in few minutes. It requires a different server than the database with Windows 2008 64bit or Windows 2008R2 (VMware, Microsoft Hyper-V or any cloud virtual instance/server are supported as well).

Adding a database instance to SafePeak requires mixed authentication on the database but again, it is simple and takes few seconds. From this point SafePeak analyzes the schema and creates the relations graph, a procedure that takes between seconds to an hour and is a one-time operation. Once this is completed, you are good to go.

The application should be pointed (via connection string or etc\hosts file or a change of DNS record) to the SafePeak server instead of the database. If you have Jobs running on SQL Server using the agent then you may need to define an “alias” to the agent that will route all the jobs via SafePeak (but only if want SafePeak to be aware on any updates they may perform).

From this point SafePeak is in auto-pilot, learns the traffic patterns and does caching of whatever is possible.

There is fine tuning that can be done to improve performance in relation to stored procedures that perform dynamic SQL (inform SafePeak which objects are being accessed) and the same operation is required for CLR and encrypted objects. The application will continue to work without the configuration, but the effectiveness of SafePeak decreases. Basically within one hour the application will be accelerated.

User Interface

The nice, easy browser-based GUI allows both product management and also provides tools for performance bottlenecks identifications and analysis.

Click to Enlarge

The product provides a structured way to see performance information starting with the instance level all the way through the queries execution.

On each level the product shows the amount read and records the average response time, the cached response time.

Historical information can be viewed using the product reports.

Conclusion

Overall, I am very impressed with SafePeak as a novel product. SafePeak is a great acceleration solution for users who want immediate results for sites with performance, scalability and peak spikes challenges.

While there are specific environments that will clearly benefit from SafePeak, such as SQL Servers with high SELECTivity and read-intensive databases, it is designed for OLTP applications. Such environments that seem to get the most out of SafePeak’s acceleration are E-commerce, Cloud and Hosted apps, large SharePoint and CRM Dynamics implementations, ERP & CRM applications.

SafePeak can significantly increase response times, by reducing network roundtrip to the database, decreasing CPU resource usage, eliminating I/O and storage access

It is simple, plug-and-play, and easily integrates into the current architecture. If you have an important SQL Server website or enterprise application whose performance affects company sales and productivity – then it is definitely worth giving SafePeak a try.

However, while SafePeak works well with SQL Server OLTP apps, it does not play well with OLAP databases. You can still use SafePeak, but you will need to specially configure CLR objects with SafePeak.

You can download a free, fully functional trial here: www.SafePeak.com/download.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

http://www.safepeak.com/download?source=sqlauthority-sidebar

Version control makes it possible to have multiple people developing together and working on same project.  If I am working on something, a fellow ETL developer will not be able to check it out and make changes to it until I am finished with my changes and check those back in.   This addresses the common situation where one developer’s project artifact and code changes clobber that of another developer by accident.

Another reason to have version control is that it gives the developer confidence to try different modifications to the logic without having to worry about a negative impact to the previously developed logic.   Sure, I could always save off my work and create a copy to try things out on.  However, in my experience, I would wind up with various different copies all over the place and it would then become difficult to keep track of what’s what.  Version control allows me to capture and document relevant information about whatever I am checking in.

Finally, version control maintains a history of changes that have been made to the application logic.  In some environments, this information and traceability can prove extremely valuable if not necessary.   Ever wonder when a bug was introduced into the application logic?

Click to Enlarge

The expressor Studio desktop ETL tool allows developers to seamlessly check-in and -out project artifacts to expressor’s Repository, which supports version control and is available as part of their Standard Edition product offering.

Many data integration solutions do not have built-in version control.   Most offer some way to interface to version control systems, but expressor has it built-in with their Standard Edition product offering.   I requested a trial license to play with it and was pleasantly surprised at expressor’s seamless version control integration.  There was no need to do any separate 3^rd party download, install, and configuration.  With expressor, check-in and check-out was almost transparent.  As someone who has developed ETL solutions, I really appreciated the clean and simple approach expressor has taken.   I have no doubt this would help make any ETL project go smoother and faster.

You can request a free 30-day trial of the expressor Standard Edition, or download their free expressor Studio community ETL tool at their site.

Reference:  Pinal Dave (http://blog.SQLAuthority.com)

Introduction to expressor Datascript Modules

5 Tips for improving your data with expressor Studio

expressor 3.2 Release Review

I will soon be blogging about their upcoming 3.4 release to keep you informed about the latest developments around their product. If you haven’t tried yet, consider downloading and test-driving their Studio product – it’s absolutely free.

Pinal Dave (http://blog.SQLAuthority.com)

Suppose you want to execute a one-to-many application in which each incoming record needs to be parsed into multiple output records.  For example, a record containing monthly data over a year period needs to be reworked so that each emitted record contains data from a single month.  The following fragment shows two representative data records, where the first field is the year and the following fields are the monthly data, January through December.

Year,Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec
2010,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012
2009,901,902,903,904,905,906,907,908,909,910,911,912

You want each output record to contain only three fields: Year, Month, and Value.

Year,Month,Value
2010,Jan,1001
2010,Feb,1002
2010,Mar,1003
2010,Apr,1004
...

To illustrate how you can reuse the coding within a Datascript Module, think of the incoming record as a sequence of fields each with a specific attribute name and value.

GroupAttribute,Atr1,Atr2,Atr3,Atr4,Atr5,Atr6,Atr7,Atr8,Atr9,Atr10,Atr11,Atr12

And think of each emitted record as comprised of three fields.

GroupAttribute,Atr,Val

Now you can write generic code within the Datascript Module that can handle any incoming record in a one-to-many application.

• Since you want to use the Datascript Module in multiple applications, create this artifact within an expressor library.
  • In this example, the library is named modules_library.0 and the Datascript Module is named OneToMany.
• Within the OneToMany module, include the following scripting.
  • Note that line 4, which defines the fully qualified name of the module, is entered by Studio when you first create the module.  It is a concatenation of the library name and module file name.
  • The Datascript Module includes a single function named oneToMany, which takes as arguments the name of the attribute holding a value that uniquely identifies the incoming record and the incoming record that needs to be divided into multiple output records.
  • Observe how the code uses the generic field names GroupAttribute, Atr, and Val to populate each output record.  It’s not until the collection of output records is returned to the calling code that the coding uses the actual attribute names (Year, Month, Value) of the output record.
    • Consequently, the actual attribute names used by the incoming and outgoing records are only referenced within the calling code, which is specific to an application.
    • All coding within the Datascript Module uses generic names applicable to any incoming data.

Now, let’s turn our attention to the code that uses this Datascript Module.  In this case, the application is quite simple.  It reads a file that contains the yearly summary records, uses a Transform operator to process each record into multiple output records, and then writes each monthly output record to a file.

• Line 2 cross-references the Datascript Module.  This statement is equivalent to simply copying and pasting the scripting from the module into the operator’s coding.
• Line 5 invokes the oneToMany function defined in the module.
• Lines 10 through 20 iterate through the collection of output records returned by the module’s oneToMany function, emitting each record individually.
  • To emit more than one record from the Transform operator, the scripting uses an iterator function, another new feature introduced in expressor 3.3.

Using modules written by other developers is no more involved.  In fact, expressor 3.3 ships with many modules that your code can utilize.  These include a cURL module that you can use to invoke on Web services or FTP servers, a SAX XML module that you can use to parse an XML document, a socket module that you can use to interact with an external application, a SQL module that allows your code to execute a SQL statement against a remote database, as well as some expressor Datascript Modules that include utility functions useful for parsing CSV files and serializing and de-serializing expressor Datascript tables.

To use any of these modules, simply include the necessary require statement at the beginning of your scripting.  The expressor Community Knowledge Base includes entries that describe use of these modules. The expressor Studio desktop ETL tool is freely downloadable from the expressor website.   I encourage you to give it a try!

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Absolutely, SSIS package can be scheduled in SQL Agent Jobs. Here is quick note on how one can do the same.

First you can create new job from SQL Server Agent Menu.

Create New Step.

Select Type as SQL Server Integration Services Packages. Select Package Source as file system and give package path.

Now click on OK, which will bring you at following screen.

On next screen you can select schedule and configure desired schedule.

You can notice this is very easy process. Let me know if you have any further questions.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

Recently I pressed CTRL+ALT+R and suddenly a web browser opened in SSMS. I later realize that I can also access the same from SSMS >> View >> Other Windows >> Web Browser.

Click to Enlarge

Now user can open web browser and New Query editor and split them vertically (Go to Windows and click on New Vertical Tab Groups) and have both side by side. Very near feature. I am personally going to use it again and again.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

DJ: Pinal, your SQL Server does not like me.
Pinal: Possible, no one likes you!
DJ: I am serious…
Pinal: I am too…
DJ: No really – be serious.
Pinal: Well, you started it, tell me your problem.
DJ: This new SQLCMD thingy does not work. Everytime I am using my username and password it does not log me in.
Pinal: Have you tried with SSMS?
DJ: No, but it works fine. I just logged in my machine using it.
Pinal: Oh, are you trying to login using your Windows Authentication Username and Password?
DJ: Does it matter? I am the admin on my box.
Pinal: It does matter. Tell me your script.
DJ: sqlcmd -S .\sqlent1 -U username -P password. And then I’d been given “error 18456 – login failed for user.”
Pinal: Try sqlcmd -S .\sqlent1 -E
(After a while)
DJ: It works! So you still know SQL Server, what’d you say?
Pinal: I say  you still do not know SQL Server (evil laugh)
DJ: (extends the evil laugh)

Well, it was simple. He was trying to connect to SQL Server which was installed on his local box, where he was the admin. In this case, he can use a simpler script. This is a very specific situation and it was not the production server’s fault. Here is a copy of the scenario on my personal laptop.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

I recently searched for command prompt solution for the same and I learned it today.

If you are going to say this is so 90s. Well, I am late by 20 years :)

Shutdown the Computer using Command Prompt

shutdown /s

Restart the Computer using Command Prompt

shutdown /r

Well this is very simple but I am glad that I know it.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

Once you get past the download and install process, you are greeted by a clean and familiar UI, which feels natural to most users of Windows applications, such as Microsoft Office.  The 3.2 release also introduces an error handling framework, a new SQL query operator, and type constraints on expressor’s metadata layer, which they calls Semantic Types.  The rest of this article will focus on what these three enhancements mean to users.

A welcome addition to expressor’s ETL software is the ability to handle data validation errors related to field and record-level processing within an application.  Users can predefine how to handle errors when they occur at the field or record level. There are up to five error handling options, depending on the operation.  Users can simply abort the entire dataflow, skip the offending record entirely, redirect the record to a file or a different branch of the dataflow for additional processing, and skip or reject all remaining records.  The addition of error handling alone is very powerful, but when combined with Semantic Constraints, you can truly appreciate what expressor brings to the data integration world.

The expressor Studio error handling framework allows users to redirect invalid records for post-processing.

Before we jump into Semantic Type constraints, let me provide you with some high level background on Semantic Types.  Semantic Types are internal to expressor, and they provide a uniform way of expressing data within the application.  When expressor Studio connects to a relational or flat file data source, the application automatically creates an internal Semantic Type for you.  Typically, this is just a direct mapping of your external data format (schema) to their internal format.  Users can also define their own Types and reuse them as shared types across one or more expressor dataflows.  This feature is extremely helpful if your source data changes and you don’t want the changes to affect any subsequent transformations.  Users are able to simply map in new source data to an existing Semantic Type and their application will continue to work as if the source never changed.  The same is true on the target side.

Constraints are simply patterns and rules that you apply to Semantic Types.  For example, you can define rules on how to validate email addresses, birthdays, phone numbers, and SSNs during the mapping process.  Depending on the type of a field, you can specify a regular expression, a min or max value, or allowed values. A replacement value can be specified for use when a constraint definition is not met.  When data does not conform to the constraint and a default value is not used, the error handling framework discussed above manages the erroneous data.

 The expressor Semantic Type editor allows users to set constraints for each field attribute.  As shown in the screenshot above, a simple Regular Expression constraint has been set for “email_address” , which rejects all records that do not include an “@” symbol and do not have a “.com” or “.edu” or “.gov” or .”info” extension.

Last but not least is the addition of an SQL Query operator.  expressor now enables users to execute their own Select statements on the database prior to extracting any data.  What’s interesting about their implementation is that they are still able to create a schema and a Semantic Type automatically no matter how complex the select query is.  This new operator greatly improves expressor’s ability to selectively extract data from a database and even apply some transformation logic to the data during the extraction process.

The editor for the SQL Query operator also users to define SELECT statements.

Overall, I am very impressed with how fast expressor is adding functionality to their product.  They are on pace to deliver significant functionally every 60 days.  expressor Studio is a great application for users who want a simplified user interface for building data integration applications.  If you have a data integration or ETL project that involves SQL Server or any other relational database, or if you are like me and like to explore new technologies, then definitely give expressor Studio a try. You can download by clicking this link – and best of all, It’s free.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

SQL Server 2008 and SQL Server 2008 R2 has script generator wizard which does it for us. I ask you to pay special attention to image #5.

After the script is generated, the next challenge often users face is how to execute this large script as SQL Server Management Studio does not open the file. One can use SQLCMD for the same. See that in the last image of this post.

Pay attention to the option Types of data to script – select option ‘Schema and data’

As the file with data will be very large, use SQLCMD to execute the large script which will create database with schema & data.

Please let me know if you have any question or suggestion on this blog post.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

Specialized operators are fine for basic transformations, but most ETL transformations require logic specific to the particular application.  For that, tools resort to full featured coding tools such as Microsoft Visual Studio.  expressor software has taken a different approach.  The expressor Studio tool provides a light-weight scripting language called expressor Datascript and integrates an editing environment into each programmable operator. These tools allow development of transformation scripts of any level of complexity within the same interface.

As with any powerful programming language, expressor Datascript includes the ability to define and initialize variables, to control logic through if..then..else, while..do, repeat..until structures, to repeatedly run a block of code within a for loop, and to manipulate complex data formats. Extensive documentation for beginning to advanced programmers makes it easy to write specialized functions and to use the many functions built into expressor Datascript.

The expressor Studio interface simplifies script writing by putting functions and expression operators at the user’s fingertips in the Transform Editor.

Visual mapping aids creation of transformations, and the pre-structured assignment statement provides a starting point for the script. The expressor Studio also includes type-completion features to make it even easier to select and layout scripts.

Using expressor Datascript’s built-in functions, complex transformations can be written in the assignment statement. For example, this conditional statement uses the expressor Datascript decision function to perform multiple comparisons similar to an if..elseif..else code block.

Despite its complexity, this above statement was written entirely by selecting function names from drop down lists and simply entering the arguments to each of the nested functions.

The expressor Transform Editor also provides a pallet for writing customized functions and script structures that cannot be written, or the user does not want to write, in a single statement.

expressor’s scripting capabilities are quite powerful and available as part of their free expressor Studio application.   I encourage you to check out expressor’s product if you haven’t done so already.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

I am not the best guy to with command shell so I did it with following setup.

Code of tsql.sql

SELECT 1 ColumnName

Code of command.bat

sqlcmd -S . -i tsql.sql
timeout 100

Code of  AllBatch.bat

start cmd.exe /C “command.bat”
start cmd.exe /C “command.bat”
start cmd.exe /C “command.bat”

Now I ran AllBatch.bat and it run all the three files in parallel and simulated my needed scenario.

I believe there should be simpler way using power-shell. Anybody want to come up with equivalent code which is improvement to this code?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

SQL Server Reporting Services Add-in for SharePoint Technologies

This post is a refresher to me. I knew it was something so familiar since I have used this syntax so many times during my performance tuning project.

Run the following query to find out what the latest query that was executed in the session. There are various methods mentioned in my earlier post, so here I am picking only the one that I use most of the time.

Please use Shortcut – CTRL+T or enable “result to text” in the resultset to get formatted output.

DECLARE @sqltext VARBINARY(128)
SELECT @sqltext = sql_handle
FROM sys.sysprocesses
WHERE spid = (YourSessionID)
SELECT TEXT
FROM sys.dm_exec_sql_text(@sqltext)
GO


You can find a range of active session IDs in your system by running system stored procedure sp_who2.

The following is the resultset where I have selected the session id that is the same as from where I am running above statement.

Additionally, you can use following T-SQL script as well.

SELECT TEXT
FROM sys.dm_exec_connections
CROSS APPLY sys.dm_exec_sql_text(most_recent_sql_handle)
WHERE session_id = (yoursessionID)
GO

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Following text is produced directly from original MSDN page from here.

The SQL Server Management Pack provides the capabilities for Operations Manager 2007 SP1 and R2 to discover SQL Server 2005, 2008, and 2008 R2. It monitors SQL Server components such as database engine instances, databases, and SQL Server agents.

The monitoring provided by this management pack includes performance, availability, and configuration monitoring, performance data collection, and default thresholds. You can integrate the monitoring of SQL Server components into your service-oriented monitoring scenarios.

In addition to health monitoring capabilities, this management pack includes dashboard views, extensive knowledge with embedded inline tasks, and views that enable near real-time diagnosis and resolution of detected issues.

Download SQL Server Monitoring Management Pack Download

Reference: Pinal Dave (http://blog.sqlauthority.com)

Once the tool is launched, select the product you wish to analysis.

Click on Start Scan will take few minutes to analysis the server.

Select the appropriate features to include the analysis in report. I have just selected Engine over here.

Once the analysis is over it will bring three different tabs. Expand the tabs to see your desired results.

You can further double-click on error screen and see more details about analysis. It also lists few added links for further reading.

You can also export the report in XML format and save it for further usage.

Let me know what your thoughts about this tool.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Laerte Junior has previously helped me personally to resolve the issue with Powershell installation on my computer. He did awesome job to help. He has send this another wonderful article regarding performance counter for readers of this blog. I really liked it and I expect all of you who are Powershell geeks, you will like the same as well.

As a good DBA, you know that our social life is restricted to a few movies over the year and, when possible, a pizza in a restaurant next to your company’s place, of course.

So what we have to do is to create methods through which we can facilitate our daily processes to go home early, and eventually have a nice time with our family (and not sleeping on the couch).

As a consultant or fixed employee, one of our daily tasks is to monitor performance counters using Perfmom. To be honest, IDE is getting more complicated. To deal with this, I thought a solution using Powershell. Yes, with some lines of Powershell, you can configure which counters to use. And with one more line, you can already start collecting data. Let’s see one scenario:

You are a consultant who has several clients and has just closed another project in troubleshooting an SQL Server environment.

You are to use Perfmom to collect data from the server and you already have its XML configuration files made with the counters that you will be using- a file for memory bottleneck f, one for CPU, etc.

With one Powershell command line for each XML file, you start collecting. The output of such a TXT file collection is set to up in an SQL Server. With two lines of command for each XML, you make the whole process of data collection.

Creating an XML configuration File to Memory Counters:

Get-PerfCounterCategory -CategoryName "Memory" | Get-PerfCounterInstance  | Get-PerfCounterCounters |Save-ConfigPerfCounter -PathConfigFile "c:\temp\ConfigfileMemory.xml" -newfile

Creating an XML Configuration File to Buffer Manager, counters Page lookups/sec, Page reads/sec, Page writes/sec, Page life expectancy:

Get-PerfCounterCategory -CategoryName "SQLServer:Buffer Manager" | Get-PerfCounterInstance | Get-PerfCounterCounters -CounterName "Page*" | Save-ConfigPerfCounter -PathConfigFile "c:\temp\BufferManager.xml" –NewFile

Then you start the collection:

Set-CollectPerfCounter -DateTimeStart "05/24/2010 08:00:00" -DateTimeEnd "05/24/2010 22:00:00" -Interval 10 -PathConfigFile c:\temp\ConfigfileMemory.xml -PathOutputFile c:\temp\ConfigfileMemory.txt

To let the Buffer Manager collect, you need one more counters, including the Buffer cache hit ratio.

Just add a new counter to BufferManager.xml, omitting the new file parameter

Get-PerfCounterCategory -CategoryName "SQLServer:Buffer Manager" | Get-PerfCounterInstance | Get-PerfCounterCounters -CounterName "Buffer cache hit ratio" | Save-ConfigPerfCounter -PathConfigFile "c:\temp\BufferManager.xml"

And start the collection:

Set-CollectPerfCounter -DateTimeStart "05/24/2010 08:00:00" -DateTimeEnd "05/24/2010 22:00:00" -Interval 10 -PathConfigFile c:\temp\BufferManager.xml -PathOutputFile c:\temp\BufferManager.txt

You do not know which counters are in the Category Buffer Manager? Simple!

Get-PerfCounterCategory -CategoryName "SQLServer:Buffer Manager" | Get-PerfCounterInstance | Get-PerfCounterCounters

Let’s see one output file as shown below. It is ready to bulk insert into the SQL Server.

As you can see, Powershell makes this process incredibly easy and fast. Do you want to see more examples? Visit my blog at Shell Your Experience

You can find more about Laerte Junior over here:

• www.laertejuniordba.spaces.live.com
• www.simple-talk.com/author/laerte-junior
• www.twitter.com/laertejuniordba
• SQL Server Powershell Extension Team: http://sqlpsx.codeplex.com/

Reference: Pinal Dave (http://blog.SQLAuthority.com)

My Scenario

It is quite common in real life that sometimes observing or retrieving older data is necessary; however, it had changed as time passed by. The full database backup was 40 GB in size, and, to restore it on our production server, it usually takes around 16 to 22 minutes, depending on the load server that is usually present. This range in time varies from one server to another as per the configuration of the computer. Some other issues we used to have are the following:

1. When we try to restore a large 40-GB database, we needed at least that much space on our production server.
2. Once in a while, we even had to make changes in the restored database, and use the said changed and restored database for our purpose, making it more time-consuming.

My Solution

I have heard a lot about the Idera’s SQL virtual database tool.. Well, right after we started to test this tool, we found out that it really delivers what it promises. Using this software was very easy and we were able to restore our database from backup in less than 2 minutes, sparing us from the usual longer time of 16–22 minutes. The needful was finished in a total of 10 minutes. Another interesting observation is that there is no need to have an additional space for restoring the database. For complete database restoration, the single additional MB on the drive is not required anymore. We can use the database in the same way as our regular database, and there is no need for any additional configuration and setup.

Let us look at the most relevant points of this product based on my initial experience:

• Quick restoration of the database backup
• No additional space required for database restoration
• virtual database has no physical .MDF or .LDF
• The database which is restored is, in fact, the backup file converted in the virtual database. DDL and DML queries can be executed against this virtually restored database.
• Regular backup operation can be implemented against virtual database, creating a physical .bak file that can be used for future use.
• There was no observed degradation in performance on the original database as well the restored virtual database.
• Additional T-SQL queries can be let off on the virtual database.

Well, this summarizes my quick review. And, as I was saying, I am very impressed with the product and I plan to explore it more. There are many features that I have noticed in this tool, which I think can be very useful if properly understood.

I had taken a few screenshots using my demo database afterwards. Let us see what other things this tool can do besides the mentioned activities. I am surprised with its performance so I want to know how exactly this feature works, specifically in the matter of why it does not create any additional files and yet, it still allows update on the virtually restored database. I guess I will have to send an e-mail to the developers of Idera and try to figure this out from them.

I think this tool is very useful, and it delivers a high level of performance way more than what I expected. Soon, I will write a review for additional uses of SQL virtual database.. If you are using SQL virtual database in your production environment, I am eager to learn more about it and your experience while using it.

The ‘Virtual’ Part of virtual database

When I set out to test this software, I thought virtual database had something to do with Hyper-V or visualization. In fact, the virtual database is a kind of database which shows up in your SQL Server Management Studio without actually restoring or even creating it. This tool creates a database in SSMS from the backup of the same database. The backup, however, works virtually the same way as original database.

Potential Usage of virtual database:

As soon as I described this tool to my teammate, I think his very first reaction was, “hey, if we have this then there is no need for log shipping.” I find his comment very interesting as log shipping is something where logs are moved to another server. In fact, there are no updates on the database from log; I would rather compare it with Snapshot Replication. In fact, whatever we use, snapshot replicated database can be similarly used and configured with virtual database. I totally believe that we can use it for reporting purpose. In fact, after this database was configured, I think the uses of this tool are unlimited. I will have to spend some more time studying it and will get back to you.

Click on images to see larger images.

Please note that all SQL Server MVP gets free license of this software.

Reference: Pinal Dave (http://blog.SQLAuthority.com), Idera (virtual database)

Let us see what kind of set-up he had on his package.

Source Table with Identity column
Destination Table with Identity column
Following checkbox was disabled in Import Expert Wizard (as per the image below)

To solve this issue, we enabled the checkbox as seen in the above image. This way, we were able to fix the problem.

The reason why he was facing this error is because his destination table had the IDENTITY property which will not allow any kind of insert from the user. This value is automatically generated by the system when new values are inserted in the table. However, when the user manually tries to insert a certain value in the table, it this property stops the action and shows an error.

Enabling the  property “Enable Identity Insert” by checking the checkbox allows the values to be inserted in the identity field. This way, the exact identity values are moved from source database to destination table.

Let me know if this blog post is easy to understand, so I can further elaborate the details as necessary.

Reference: Pinal Dave (http://blog.SQLAuthority.com),

VDS Technologies has all the spatial files for every location for free. You can download the spatial file from here. If you cannot find the spatial file you are looking for, please leave a comment here, and I will send you the necessary details. Unzip the file to a folder and it will have the following content.

Then, download Shape2SQL tool from SharpGIS. This is one of the best tools available to convert shapefiles to SQL tables. Afterwards, run the .exe file.

When the file is run for the first time, it will ask for the database properties. Provide your database details.

Select the appropriate shape files and the tool will fill up the essential details automatically. If you do not want to create the index on the column, uncheck the box beside it. The screenshot below is simply explains the procedure. You also have to be careful regarding your data, whether that is GEOMETRY or GEOGRAPHY. In this example,  it is GEOMETRY data.

Click “Upload to Database”. It will show you the uploading process. Once the shape file is uploaded, close the application and open SQL Server Management Studio (SSMS).

Run the following code in SSMS Query Editor.

USE Spatial
GO
SELECT *
FROM dbo.world
GO

This will show the complete map of world after you click on Spatial Results in Spatial Tab.

In Spatial Results Set, the Zoom feature is available. From the Select label column, choose the country name in order to show the country name overlaying the country borders.

Let me know if this tutorial is helpful enough. I am planning to write a few more posts about this later.

Note: Please note that the images displayed here do not reflect the original political boundaries. These data are pretty old and can probably draw incorrect maps as well. I have personally spotted several parts of the map where some countries are located a little bit inaccurately.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

• Disk Usage
• Disk Usage by Top Tables
• Disk Usage by Table
• Disk Usage by Partition
• Backup and Restore Events
• All Transactions
• All Blocking Transactions
• Top Transactions by Age
• Top Transactions by Blocked Transactions Count
• Top Transactions by Locks Count
• Resource Locking Statistics by Objects
• Object Execute Statistics
• Database Consistency history
• Index Usage Statistics
• Index Physical Statistics
• Schema Change history
• User Statistics

Select the Reports with name Index Physical Statistics. Once click, a report containing all the index names along with other information related to index will be visible, e.g. Index Type and number of partitions.

One column that caught my interest was Operation Recommended. In some place, it suggested that index needs to be rebuilt. It is also possible to click and expand the column of partitions and see additional details about index as well. DBA and Developers who just want to have idea about how your index is and its physical statistics can use this tool.

Note: Please note that I will rebuild my indexes just because this report is recommending it. There are many other parameters you need to consider before rebuilding indexes.

However, this tool gives you the accurate stats of your index and it can be right away exported to Excel or PDF writing by clicking on the report.

Update: As Per Paul Randal this is un-killable and lots of IO, be careful about using it.

Reference : Pinal Dave (http://blog.SQLAuthority.com)