Counters In SSIS: What Are Your Options?
The other day I needed a counter in my SSIS Control Flow. Using the Foreach Loop container I was looping over a bunch of files and needed to count the number of files that got imported successfully. The counter had to comply with the following requirements:
- Easy to implement
Let’s test some different methods on getting this implemented.
Two of the methods are possible as of SSIS 2005 while the third one is new since SSIS 2012. So yeah, the screenshots are taken using SQL Server 2012 RTM.
Setting The Scene
We’ll start from a new package and create a package variable of type Int32. This variable will keep track of the count.
To be able to performance test the different possibilities, I’m using a For Loop container. This loop is using another package variable called loop, type Int32, and performs the loop 10,000 times. In case you don’t know how to set up such a loop, check out this screenshot:
So my package contains the following two variables:
The cnt variable is the one to actually hold the count.
To test performance I’ll be using the command-line dtexec utility so that the debugging overhead does not mess up the numbers. I’ve also executed each method at least three times to ensure a certain result was not “by accident”.
Using a Script Task
The most logical component that came to mind was a Script Task.
Using a couple lines of code, C# in this case, the counter value can be incremented with one:
int cnt = (int)Dts.Variables["cnt"].Value; Dts.Variables["cnt"].Value = ++cnt;
The above code assumes that the cnt variable has been put in the ReadWriteVariables property:
Here’s what that looks like in the Control Flow:
So how fast does this execute?
About eight seconds, that’s acceptable.
However, what I don’t like about this method is how long it takes to implement. I mean, it takes more than just a couple of seconds, right? And the fact that you actually need to use custom .NET code to perform such a trivial task as adding one to a number. Using .NET code is good for complex situations, when there’s no other option. But the risk of a bug is always larger, imagine I wrote cnt++ instead of ++cnt, what do you think the results would be? (Hint: my laptop would crash before the counter reaches 10,000).
On to another option then!
Using a Execute SQL Task
Instead of resorting to .NET code, increasing a number by one is easy to achieve using a simple T-SQL statement, right? So I thought, let’s try a Execute SQL Task!
Here’s the query:
select ? + 1 as cnt
What does the task look like?
ResultSet has been set to Single row.
The Parameter Mapping page has got the User::cnt variable specified:
And the Result Set page has got the single value from the single row mapped to the User::cnt package variable:
What do you say, easier to implement than the Script method? I do think so!
This method has one limitation though: it needs a Connection Manager connecting to a SQL Server database. However, in most ETL packages you’ll probably have that present already. What I was a bit worried about though is the overhead of connecting to the server, how much will it be?
Let’s find out!
That’s right, using the Execute SQL Task to increment the variable 10,000 times takes about a minute. On my laptop. Connecting to the tempdb on my local instance. When testing this over a network, it even resulted in timings over four minutes. So this solution is really unacceptable in my opinion.
However, we can give it one more try. A Connection Manager has got a property called RetainSameConnection. By default this is set to False which means that our test above has opened and closed ten thousand connections to my local tempdb. Oh my, of course that takes a while!
Setting it to True gives us the following result:
About twenty seconds, which is about one third of the previous result. That surely is better. And what’s perhaps even more interesting is that a similar result is achieved when connecting to a database over the network: from over four minutes down to twenty seconds. So yeah, this would work for me.
Sidenote: for other interesting use of the RetainSameConnection property, check out this post regarding transactions by Matt Masson.
But we’re not stopping just yet. As of SQL Server 2012, we’ve got a third possibility!
Using an Expression Task (SQL2012!)
Instead of resorting to custom .NET code or (ab)using the Execute SQL Task, in SSIS 2012 we’ve got a new task: the Expression Task.
As you can read in the SSIS Toolbox, the Expression Task builds and evaluates SSIS Expressions that set variable values at runtime. That surely sounds exactly like what we’d need, doesn’t it?
So how does that work?
Really easy to set up, we just specify that the User::cnt variable should be set to itself plus one. When put in a For Container, we’ve got a counter!
But how does it perform?
About seven seconds, which is even slightly faster than the Script Task method!
With that we’ve found the method that complies with both requirements: easy to implement and performs fast! Now how am I going to convince my clients to upgrade to 2012, hmm, need to think a bit…
We found out that the new Expression Task is a very useful task for its purpose. In our case we used it to create a counter.
If you’re not on SQL Server 2012 yet, better stick to either Script Task or Execute SQL Task with RetainSameConnection set to True.
- T-SQL Tuesday 42: Life, Change, Don’t Panic!
- Local Install of Books Online 2012
- Hidden Collections in SSRS
- How To Tweet About SQL Server Blog Posts
- Filtering Data Without Changing Dataset [SSRS]
- SSRS Deployment: Generate The Batch Script Through SQL!
- Connecting Shapes In Word
- Automating SSRS Deployment: Download
- SQLUG: Automating SSRS Deployments
- Shall We Join Or Shall We Split Now?
- May 2013 (2)
- March 2013 (3)
- February 2013 (2)
- January 2013 (2)
- December 2012 (2)
- November 2012 (3)
- October 2012 (2)
- August 2012 (2)
- July 2012 (2)
- June 2012 (2)
- May 2012 (2)
- April 2012 (3)
- March 2012 (4)
- February 2012 (4)
- January 2012 (2)
- December 2011 (2)
- November 2011 (2)
- October 2011 (1)
- September 2011 (3)
- August 2011 (2)
- June 2011 (2)
- May 2011 (3)
- April 2011 (3)
- March 2011 (3)
- February 2011 (2)
- January 2011 (5)
- December 2010 (1)
- November 2010 (3)
- October 2010 (3)
- September 2010 (2)
- August 2010 (4)
- July 2010 (2)
- June 2010 (4)
- May 2010 (6)
- April 2010 (3)
- March 2010 (3)
- February 2010 (11)
- January 2010 (9)
- December 2009 (2)
- November 2009 (3)
- October 2009 (3)
- September 2009 (4)
- August 2009 (6)
- July 2009 (2)
- June 2009 (3)
- May 2009 (7)
- April 2009 (3)
- March 2009 (3)
- February 2009 (5)
- January 2009 (4)
- December 2008 (2)
- November 2008 (3)
- October 2008 (1)
- September 2008 (1)
- August 2008 (4)
- July 2008 (3)