Have you ever wanted a feature in the BIDS to quickly identify package variables which have become obsolete?  Well, I’ve been involved in the cleanup of existing packages and I can tell you, that feature would be very handy!

After a search on the internet, it turns out that one of the planned features for the BIDS Helper contains exactly that.  The request is a bit wider than what I need, but at least “highlight unused variables” is part of it.

All I’m asking now is just a minute of your time to vote for that feature request.

Have fun, and thank you!

Valentino.

So, a couple of weeks ago I decided to post my first bug to Microsoft Connect, an issue which I had discovered in SQL Server 2012 RC0.  Because this process took me more time than I’d expected, I‘m now blogging about my experience.

The hardest part about the whole experience was finding the actual page that needs to be used to enter and submit the content.

Submitting Content To Microsoft Connect

Are you in for a ride on the Connect site?  Here we go!

Step 1: Search For Your Product on The Connect Home Page

On the Home page, enter a search term with which you can locate your favorite product.  If you’re not signed in yet, do that first.

Step 2: Click To Participate

Once the search engine has produced the output, locate your favorite product and click the Participate action next to it.

Step 3: Submit Product Feedback

After clicking Participate, the following overwhelming screen appears.  The link that you’re looking for is called Submit Product Feedback.

Step 4: The Feedback Center

After clicking Submit Product Feedback, you end up at the Feedback Center.

Looking at the screen above, you’d think that the yellow Submit Feedback button can be used to open the page where you’d need to enter the bug details.  Well, it doesn’t…

You need to search for your issue first, to ensure it doesn’t exist already.

Step 5: The Submit Feedback Button Has Been Found, Eureka!

After you’ve performed the search for your issue, you get to the following page above.  Finally, on that page you can click a real Submit Feedback button that will bring you to the Select Feedback Form page.

Step 6: Select Your Feedback Form Of Choice

On the Select Feedback Form page, select SQL Server Bug Form to file a SQL Server issue, or select SQL Server Suggestion Form to enter a suggestion.

Step 7: Fill Out The Textboxes

Finally, the page we’ve been trying to locate when we started the adventure, quest accomplished!  With this new knowledge, we can now skip directly to step 6 through the following link: Select Feedback Form

Disclaimer: please ensure that you’re not creating duplicate bugs or suggestions, but use whatever search engine your prefer to achieve that goal!

So, here’s my New Year’s present to all of you!  And remember: have fun!

Valentino.

References

Microsoft Connect

Feedback Center

Select Feedback Form

Take a good look at the following screenshot:

That’s right, they are not in the Toolbox anymore but in the SSIS Toolbox instead.  This new toolbox is a bit different from the old one.  Besides the grouping of components that has changed, the most important change is that it will automatically detect any custom components.  You no longer need to right-click, select Choose Items, go fetch a coffee, wait until it cools down a bit, drink it and finally … select your custom component.  No, you’ll have to find another reason to get that coffee shot.  Actually, that’s not entirely true: you still need to right-click and then click Refresh Toolbox and then the custom components will be shown.

Another difference is that it’s split in two parts.  The bottom half of the pane now contains a description of the selected item, including a link that should lead to samples and a link to the Books Online.

Out of curiosity I tried the Find Samples link a couple of times, but for now it doesn’t seem to deliver much content:

Okay, so one thing remains: how do you open the new SSIS Toolbox pane?  According to the Books Online it should be opened automatically when you open an existing project.  Well, apparently not all the time!

The first place I’d look is in the View menu.  But alas, SSIS Toolbox is not one of the menu items.  Not even in the Other Windows submenu.  Why oh why?!

Long story short: do you see those two buttons in the below screenshot?  They’re new!

The first button leads to the Variables pane, the second button will open the SSIS Toolbox.  Good to know isn’t it?!

Further investigation led me to the following: according to the Books Online, the SSIS Toolbox item should actually be located in the View > Other Windows menu.  As that is not the case and I think it’s only logical to have that pane added to the View menu as well, I’ve filed a bug on Microsoft Connect.  Go ahead and vote!

Have fun!

Valentino.

References

SSIS Toolbox

As you can see, this container looks a bit similar to the Sequence Container but the Sequence Container has a different icon.  Also, this container does not have any connectors sticking out.  I’m currently involved in a 2005 > 2008 migration project, which is how I came across this container in the first place, and thus I really needed to find out what this object actually is.  So I started to investigate.  Unfortunately the properties were not very helpful:

No properties, nothing at all, not even an object name.  Then I started to scan the Toolbox pane for the icon.  Guess what?  Right, nothing either!  Is this some kind of custom control??

The logical next step is to perform some internet searches and I found the answer: this is standard SSIS functionality that exists since SQL Server 2005!  WTF! (is what I thought at that moment)

The Grouping Container

In the Control Flow, if you right-click on at least one task, you get the following options:

And when you select Group, you’ll get that group container around the selected objects, allowing you to collapse the items.  Well, okay, I’ll probably still never use it now that I know that it exists.  Which is possibly the reason that I don’t know about it in the first place.

How Not To Use It

Using that grouping functionality you can end up with some weird-looking flows:

What have we learned?  That even with years of SSIS experience, it’s still possible to discover “new” functionality.

And speaking of new functionality, that brings us to SQL Server 2012 (formerly known as Denali).

Grouping In SQL Server 2012

As of SQL Server 2012, not only will you be able to group tasks in the Control Flow.  You can also group components in the Data Flow.  Now that may prove more interesting than its Control Flow counterpart.  Why?  Because in the Control Flow chances are that you’ve already grouped your tasks using some Sequence Containers while you don’t have any containers in the Data Flow.

Here’s what it looks like:

As you can see, the icon is now gone.  But the properties pane is still totally empty with the group control selected.  Oh well…

Have fun!

Valentino.

References

How to: Group Tasks and Containers in a Control Flow

Msg 2601, Level 14, State 1, Line 4
Cannot insert duplicate key row in object ‘dbo.#t’ with unique index ‘PK_Unique’.

That’s right, the “hey, you’re inserting junk, stop that right now!” message.

However, earlier this week I encountered a variation of that error in our ETL logs:

Msg 2601, Level 14, State 1, Line 3
Cannot insert duplicate key row in object ‘dbo.#t’ with unique index ‘PK_Unique’. The duplicate key value is (1, 2).

I’m sure I don’t need to explain how useful that extra sentence can be, right?  So, then I started digging because obviously I want that error to always include the offending values.

Recently our database back-end servers have been replaced with new machines running SQL Server 2008 R2 SP1.  And indeed, as of R2 SP1, this error message has gotten an upgrade!

If you want to check for yourself, the following code snippet can be used to generate the error:

create table #t (PK1 int, PK2 int);
create unique index PK_Unique on #t(PK1, PK2);
insert into #t values (1, 2),(1, 2);

Now, message 2601 is not the only one complaining about duplicate keys.  Another example can be generated using the following query:

create table #t (PK1 int unique, PK2 int unique);
insert into #t values (1, 2),(1, 2);

Executing that snippet on SQL Server 2008 R2 SP1 results in this error:

Msg 2627, Level 14, State 1, Line 1

Violation of UNIQUE KEY constraint ‘UQ__#t________C5776555123EB7A3′. Cannot insert duplicate key in object ‘dbo.#t’. The duplicate key value is (2).

Do you notice the difference?  The message did indeed get an upgrade, but is not able to support a simultaneous violation on multiple columns, while msg 2601 can.  Hopefully they can get that fixed by SQL Server 2012 RTM!

Out of curiosity I decided to get a closer look at other possible variations on this error and dug into the sys.messages table using this query:

select * from sys.messages
where text like ('%duplicate key%');

Results of that query:

So, there’s one more to investigate: 1505.  To get that error generated is not that complicated.  We actually already have all the statements, just need to use them in the right order, like so:

create table #t (PK1 int, PK2 int);
insert into #t values (1, 2),(1, 2);
create unique index PK_Unique on #t(PK1, PK2);

And what do we get?

Msg 1505, Level 16, State 1, Line 1

The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name ‘dbo.#t__________________________________________________________________________________________________________________00000000001D’ and the index name ‘PK_Unique’. The duplicate key value is (1, 2).

Nice, message 1505 also supports multiple columns, that makes two out of three!

Ow, don’t forget to clean up (yes, I even do that for temporary tables).

drop table #t;

Additional Info

Some further digging around on the internet brought me to the following suggestion on Microsoft Connect: Duplicate Key Values by Anton Plotnikov.  Looks like our enhanced statements are the result of that request.

There’s another suggestion as well: Tweak To Duplicate Key Message by Louis Davidson.  Louis suggests to have all offending values added to the message as well, but in a different situation than when dealing with multiple unique columns.  You can create one insert statement that inserts more than one record with offending values, and that’s the situation he’s referring to.  If you feel that’s important, get over to Connect and cast vote!  I do think that the number of values would need to get limited in that case, we might end up with really long error messages otherwise.

One last link, also by Louis Davidson.  He also wrote a blog post about the error message when he found out about it in Denali.

Have fun!

Valentino.

Ever seen this warning message while using the Management Studio?

All I did was open the AdventureWorks.Production.Product table in the designer, moved the ListPrice column above the StandardCost column and hit CTRL+S to save changes.  Instead of changing the table, it tells me:

Saving changes is not permitted.  The changes you have made require the following tabled to be dropped and re-created.  You have either made changes to a table that can’t be re-created or enabled the option Prevent saving changes that require the table to be re-created.

What a message huh?!  It needs to re-create the table, so what, go ahead!  Hang on, even though this is just a Warning, there’s no Continue button!  Ow come on, are you kidding?

Like any good developer that wants to get some work done, you stop reading such a long message after the second line and try to find a solution for your problem.  Too bad, you should have just continued reading the warning message because the solution is actually hidden in there!

If you open up the Options screen from the Tools menu in SSMS, and then drill down to the Designers > Table and Database Designers node, you’ll see an option called Prevent saving changes that require table re-creation.  And by default, that option is checked.  Remove the check from that box and you’ll be able to use the table designer as it was intended.

So, here’s your ADSOTD (Annoying Default Setting Of The Day)!

Small note: be careful when you want to use this feature on a large table.  The script that gets generated in the background to make the changes needs to copy all records from the old table into the new one and that will obviously require some time!

Have fun!

Valentino.

Here are the details:

10 days of SQL Server : 1^st – 10^th September

10 days of SharePoint :11^th – 20^th September

10 days of Silverlight : 21^st – 30^th September

Throughout these dates there will be up to 20% off SQL Server, SharePoint and Silverlight books and 30% off eBooks – for further information visit: http://www.packtpub.com/packt-10-days-of-sql-server-silverlight-sharepoint

In case you have read my previous post, you know that there’s one book that I can surely recommend: Microsoft SQL Server 2008 R2 Master Data Services!

As you’ve probably noticed, the first 10 days have already started, so hurry up, get yourself some reading material to get through the coming Winter (or Summer if you’re living on the Southern Hemisphere)!

Valentino.

I cooperated as one of the Technical Reviewers, which means I’ve read the whole book before it was published.  Obviously, just reading was not all.  I tried out all exercises to ensure that the code did not contain any mistakes and commented on possible errors, or difficult-to-understand paragraphs due to lack of details or sentences that were phrased in a weird way.

And I have to say, I was surprised by the quality of the text before it was even read by a Technical Reviewer.  It’s not easy to write several chapters, building further on the result of the previous one, without making any mistakes.  But the guys managed to do that just fine!

So, if you’re looking for a good book on Master Data Services, have a look at the Microsoft SQL Server 2008 R2 Master Data Services book, published by Packt Publishing.

The book is aimed at everyone new to MDS, no prior knowledge is required.  You will need to know SQL Server though, but as you’re reading my blog I’m sure that shouldn’t be a problem.

This doesn’t mean that it won’t go into detail though, don’t misunderstand me here.  If you’ve read the book from start to finish, you will have thorough knowledge of Master Data Services!

Happy reading,

Valentino.

“Huh, what’s he talking about?”, I hear you thinking.  Read on then.

The Scenario

You’ve got a stored procedure or another SQL statement that needs to get called from the Execute SQL Task in the Control Flow of your package.  So far so good.  One of the parameters that needs to get passed into the statement is of the DateTime type.

How would you do that?

Parameter Mapping – Take 1

Following the KISS principle, let’s say we’ve got the following really complex table in our database:

create table dt ( dtVal datetime );

And in our Execute SQL task we have this extremely complex INSERT statement:

insert into dt values (?)

The statement is expecting one parameter.  The parameter that I want to pass into it is System::StartTime which is of type DateTime as shown in the screenshot below.

“Hang on, how did you get the Variables window to display the system variables?”

Ah, good question, by clicking that Show System Variables button, indicated with the red rectangle.

So you set up the Parameter Mapping as follows, specifying DBTIMESTAMP as Data Type and zero as Parameter Name because it’s the first parameter in the statement:

Then you decide to give it a test run.  But alas, it throws you the following error:

Error: 0xC002F210 at Execute SQL Task 1, Execute SQL Task: Executing the query “insert into dt values (?)” failed with the following error: “Invalid time format”. Possible failure reasons: Problems with the query, “ResultSet” property not set correctly, parameters not set correctly, or connection not established correctly.

So we’ve got a datetime column in the table and we’ve got a DateTime package variable.  But alas, the Execute SQL Task is not happy with passing this value to the query.

Now what?

The SqlStatementSource Expression

Let’s try another method then.  Instead of passing the parameter’s value through the Parameter Mapping page, we’ll set up an expression that constructs the whole INSERT statement, including the parameter’s value.

Have a look at the following expression:

"insert into dt values ('" +

(DT_STR, 4, 1252) DATEPART("yyyy", @[System::StartTime]) + "-" +

(DT_STR, 2, 1252) DATEPART("mm", @[System::StartTime]) + "-" +

(DT_STR, 2, 1252) DATEPART("dd", @[System::StartTime]) + " " +

(DT_STR, 2, 1252) DATEPART("hh", @[System::StartTime]) + ":" +

(DT_STR, 2, 1252) DATEPART("mi", @[System::StartTime]) + ":" +

(DT_STR, 2, 1252) DATEPART("ss", @[System::StartTime]) + "." +

(DT_STR, 3, 1252) DATEPART("ms", @[System::StartTime]) + "')"

It uses the DATEPART function to fetch parts of the System::StartTime variable and feed it into the INSERT statement using a format that works all the time (YYYY-MM-DD HH:MM:SS.MIL).  Here’s what it generated when I clicked the Evaluate Expression button in the Expression Builder:

insert into dt values (’2011-5-31 17:59:37.0′)

So where exactly would you specify that expression?  In the Execute SQL Task editor, open up the Expressions page.  Then click the Expressions item in the Misc list so that the button with the ellipsis appears.  Now click that button, select SqlStatementSource as property and click the Ellipsis button in the Expression field to get to the Expression Builder.

Then you’ll end up with something like this:

Give the package another run.  If everything has been set up as expected, the Execute SQL Task should color green and a select on the table should give one record:

Hang on, does it really have to be this complicated?

Well, maybe not…

Parameter Mapping – Take 2

So let’s give the Parameter Mapping another go.

Set up the Execute SQL Task just like in Take 1 above, with one small difference: select DATE instead of DBTIMESTAMP as Data Type for the parameter.

Now give the package another run.  Look at that, it colors green and there’s an extra record in the table:

DATE doesn’t seem like the most logical type to choose in this scenario, which is why most people won’t even consider it.  But it works!  Actually, “DATE” is not really the best name that could be given to this particular data type.  Here’s the description of DT_DATE (not to be confused with DT_DBDATE!) according to MSDN:

A date structure that consists of year, month, day, hour, minute, seconds, and fractional seconds. The fractional seconds have a fixed scale of 7 digits.

The DT_DATE data type is implemented using an 8-byte floating-point number. Days are represented by whole number increments, starting with 30 December 1899, and midnight as time zero. Hour values are expressed as the absolute value of the fractional part of the number. However, a floating point value cannot represent all real values; therefore, there are limits on the range of dates that can be presented in DT_DATE.

On the other hand, DT_DBTIMESTAMP is represented by a structure that internally has individual fields for year, month, day, hours, minutes, seconds, and milliseconds. This data type has larger limits on ranges of the dates it can present.

What this means is that you have to be careful when using this type.  Even though it works fine today, it may not run fine in a similar scenario that required different date ranges.  But obviously you’ve got that covered by your unit test scenarios!

Conclusion

In this article I have demonstrated how a DateTime package variable can be passed as parameter into the Execute SQL Task in more than one different way.  My method of preference is the one using the DATE type in the Parameter Mapping.

Have fun!

Valentino.

References

SSIS Execute SQL Task

SSIS DatePart function

KISS Principle

SSIS Junkie: Datetime variables don’t always do what you expect

The OP was having an issue getting a record count of all tables in his database.  It was working through the Management Studio, using the undocumented but well-known sp_MSforeachtable stored procedure.

exec sp_MSforeachtable 'select ''?'' TableName, count(*) Cnt from ?';

However, the OP needed the list of record counts inside an Integration Services package.  And this didn’t work out as expected, because sp_MSforeachtable uses a temporary table, something that SSIS does not like.

So I proposed to use a script similar to the following, possibly in a stored procedure.

/*
DESCRIPTION: Returns a list of record counts for each table in the database
AUTHOR:         Valentino Vranken
CREATED:     2011/05/21
VERSION:     1.0
COPIED FROM: http://blog.hoegaerden.be
*/
declare @tempTable table
(
    TableSchema nvarchar(256),
    TableName nvarchar(256),
    Cnt bigint
);

declare @sql nvarchar(4000);
declare @tableSchema nvarchar(256);
declare @tableName nvarchar(256);
declare @columnName sysname;
declare @cnt bigint;

declare tableCursor cursor for
    select TABLE_SCHEMA, TABLE_NAME from INFORMATION_SCHEMA.TABLES
    where TABLE_TYPE = 'BASE TABLE';

open tableCursor;

fetch next from tableCursor into @tableSchema, @tableName;

while @@FETCH_STATUS = 0
begin
    set @sql = 'select @cnt = COUNT(*) from [' + @tableSchema + '].[' + @tableName + ']';

    exec sp_executesql @sql, N'@cnt bigint output', @cnt = @cnt output;

    insert into @tempTable select @tableSchema, @tableName, @cnt;

    fetch next from tableCursor into @tableSchema, @tableName;
end;

close tableCursor;
deallocate tableCursor;

select * from @tempTable;

Yeah, I know, cursors are bad… But hey, at least it works!  Just don’t schedule it for execution every five seconds.

I seem to be repeating myself.  I wrote something similar when I posted the script about finding empty columns many months ago.

The most important part of the script is the following:

select TABLE_SCHEMA, TABLE_NAME from INFORMATION_SCHEMA.TABLES
where TABLE_TYPE = 'BASE TABLE';

This is using one of the information schema views to retrieve a list of tables in the database.  The INFORMATION_SCHEMA.TABLES view contains a list of tables and views, so the where clause is needed to filter out the views.  INFORMATION_SCHEMA.TABLES is just one of many interesting views available in SQL Server, and should always be used in favor of the sys tables.

What does the output look like?  Here’s part of the output when executed on the AdventureWorks2008R2 database:

Have fun!

Valentino.