So, a couple of weeks ago I decided to post my first bug to Microsoft Connect, an issue which I had discovered in SQL Server 2012 RC0.  Because this process took me more time than I’d expected, I‘m now blogging about my experience.

The hardest part about the whole experience was finding the actual page that needs to be used to enter and submit the content.

Submitting Content To Microsoft Connect

Are you in for a ride on the Connect site?  Here we go!

Step 1: Search For Your Product on The Connect Home Page

On the Home page, enter a search term with which you can locate your favorite product.  If you’re not signed in yet, do that first.

Step 2: Click To Participate

Once the search engine has produced the output, locate your favorite product and click the Participate action next to it.

Step 3: Submit Product Feedback

After clicking Participate, the following overwhelming screen appears.  The link that you’re looking for is called Submit Product Feedback.

Step 4: The Feedback Center

After clicking Submit Product Feedback, you end up at the Feedback Center.

Looking at the screen above, you’d think that the yellow Submit Feedback button can be used to open the page where you’d need to enter the bug details.  Well, it doesn’t…

You need to search for your issue first, to ensure it doesn’t exist already.

Step 5: The Submit Feedback Button Has Been Found, Eureka!

After you’ve performed the search for your issue, you get to the following page above.  Finally, on that page you can click a real Submit Feedback button that will bring you to the Select Feedback Form page.

Step 6: Select Your Feedback Form Of Choice

On the Select Feedback Form page, select SQL Server Bug Form to file a SQL Server issue, or select SQL Server Suggestion Form to enter a suggestion.

Step 7: Fill Out The Textboxes

Finally, the page we’ve been trying to locate when we started the adventure, quest accomplished!  With this new knowledge, we can now skip directly to step 6 through the following link: Select Feedback Form

Disclaimer: please ensure that you’re not creating duplicate bugs or suggestions, but use whatever search engine your prefer to achieve that goal!

So, here’s my New Year’s present to all of you!  And remember: have fun!

Valentino.

References

Microsoft Connect

Feedback Center

Select Feedback Form

The OP was having an issue getting a record count of all tables in his database.  It was working through the Management Studio, using the undocumented but well-known sp_MSforeachtable stored procedure.

exec sp_MSforeachtable 'select ''?'' TableName, count(*) Cnt from ?';

However, the OP needed the list of record counts inside an Integration Services package.  And this didn’t work out as expected, because sp_MSforeachtable uses a temporary table, something that SSIS does not like.

So I proposed to use a script similar to the following, possibly in a stored procedure.

/*
DESCRIPTION: Returns a list of record counts for each table in the database
AUTHOR:         Valentino Vranken
CREATED:     2011/05/21
VERSION:     1.0
COPIED FROM: http://blog.hoegaerden.be
*/
declare @tempTable table
(
    TableSchema nvarchar(256),
    TableName nvarchar(256),
    Cnt bigint
);

declare @sql nvarchar(4000);
declare @tableSchema nvarchar(256);
declare @tableName nvarchar(256);
declare @columnName sysname;
declare @cnt bigint;

declare tableCursor cursor for
    select TABLE_SCHEMA, TABLE_NAME from INFORMATION_SCHEMA.TABLES
    where TABLE_TYPE = 'BASE TABLE';

open tableCursor;

fetch next from tableCursor into @tableSchema, @tableName;

while @@FETCH_STATUS = 0
begin
    set @sql = 'select @cnt = COUNT(*) from [' + @tableSchema + '].[' + @tableName + ']';

    exec sp_executesql @sql, N'@cnt bigint output', @cnt = @cnt output;

    insert into @tempTable select @tableSchema, @tableName, @cnt;

    fetch next from tableCursor into @tableSchema, @tableName;
end;

close tableCursor;
deallocate tableCursor;

select * from @tempTable;

Yeah, I know, cursors are bad… But hey, at least it works!  Just don’t schedule it for execution every five seconds.

I seem to be repeating myself.  I wrote something similar when I posted the script about finding empty columns many months ago.

The most important part of the script is the following:

select TABLE_SCHEMA, TABLE_NAME from INFORMATION_SCHEMA.TABLES
where TABLE_TYPE = 'BASE TABLE';

This is using one of the information schema views to retrieve a list of tables in the database.  The INFORMATION_SCHEMA.TABLES view contains a list of tables and views, so the where clause is needed to filter out the views.  INFORMATION_SCHEMA.TABLES is just one of many interesting views available in SQL Server, and should always be used in favor of the sys tables.

What does the output look like?  Here’s part of the output when executed on the AdventureWorks2008R2 database:

Have fun!

Valentino.

And if those videos didn’t still your hunger, or you just have plenty of time to kill, check out this blog post on Chris Webb’s BI blog.  He has posted a link to the 152 videos of SQL Bits, a three-day SQL Server event in the UK.

Still not satisfied?  Maybe these SQL Server 2008 Microsoft Certified Master (MCM) Readiness Videos can keep you busy for a little while then.

Have fun!

Valentino.

A normal approach would be to rely on the IntelliSense feature of SSMS and start typing something like:

select use

Which gives you a dropdown like:

Cool you think, there are four system functions to return something user-related, one of them would be the Windows user, right?  Well, wrong.

Let’s find out what these functions actually return, and locate a method that does return what we require.

The Different User-Related System Functions

USER

According to the BOL:

Allows a system-supplied value for the database user name of the current user to be inserted into a table when no default value is specified.

So, how many times did you read that?  And do you now know what you’ll get?  Me neither.  As far as I understand it, you’ll get the database user name of the connection context.  Not what we need.

USER_ID

According to BOL:

Returns the identification number for a database user.

The function accepts one argument that accepts a string representing a username.  When no argument is provided, it will return the user ID of the execution context.

But that’s not important because the BOL also mentions that it will be removed in a future version and that you thus shouldn’t use it anymore.  No problem if you need it though, its replacement is called DATABASE_PRINCIPAL_ID.

USER_NAME

According to BOL:

Returns a database user name from a specified identification number.

This is the inverse of USER_ID.  It accepts one argument representing the user ID.  When omitted, the user of the current execution context will be returned.  Which is actually the equivalent of the USER function.

Still no Windows user though, so let’s move on to the next one.

USER_SID

Not documented in the BOL.

It seems to represent the sid column found through following query:

select * from sys.sysusers

And the sys.sysusers view is documented.  Apparently “sid” stands for “security identifier” and it’s a varbinary.  But not only is the view documented, it is also deprecated.  You should use the following view instead:

select * from sys.database_principals

That query returns a list of principals in the current database, and one of the fields is indeed our dear sid.  However, we can now conclude that sid is definitely not what we’re looking for in this little quest, it doesn’t even closely resemble a Windows user name.  So let’s move on again!

CURRENT_USER

Hmm, “current user”, will this finally be the Windows user with which I’m running my queries?

According to BOL:

Returns the name of the current user. This function is equivalent to USER_NAME().

Equivalent to USER_NAME, so still not what we’re looking for.

SESSION_USER

According to BOL:

SESSION_USER returns the user name of the current context in the current database.

You can probably guess by now that again this is not what we’re searching for.  Next.

SYSTEM_USER

According to BOL:

Allows a system-supplied value for the current login to be inserted into a table when no default value is specified.

Ah, finally a description that contains the word “login”!  And indeed, here’s what it returns:

ORDINA\VaVr

That is indeed my Windows user.  Mission accomplished!

Please note: if you’re running this through a window connected via a SQL Server login, you will get that login as opposed to your Windows login.  And if you get ‘sa’ while you’re connected to a production server you should look into making some security changes.

SUSER_NAME and SUSER_SNAME

Two more similar functions exist: SUSER_NAME and SUSER_SNAME.

Both return a login name as well, and both accept a parameter.  When executed without parameter, they return the login name of the current user.  Their only difference is the parameter that they accept.  SUSER_NAME accepts the “login identification number of the user”, which is an int, and SUSER_SNAME accepts the “login security identification number”, a varbinary(85) (remember sid from above?).

Quick Query

The following query shows how to use all the functions, with the default values for any parameters.

select USER as cUSER, USER_ID() as cUSER_ID, USER_NAME() as USER_NAME,
    USER_SID() as cUSER_SID, CURRENT_USER as cCURRENT_USER,
    SESSION_USER as cSESSION_USER, SYSTEM_USER as cSYSTEM_USER,
    SUSER_NAME() as cSUSER_NAME, SUSER_SNAME() as cSUSER_SNAME

Have fun!

One of my favorite statements since many years is the following:

select @@version;

Here’s the output that it produces on my laptop running Windows 7, while connected to my R2 server:

Microsoft SQL Server 2008 R2 (RTM) – 10.50.1600.1 (X64)   Apr  2 2010 15:48:46   Copyright (c) Microsoft Corporation  Enterprise Edition (64-bit) on Windows NT 6.1 <X64> (Build 7600: )

It not only retrieves the version, edition and CPU architecture of SQL Server, on top of that it also shows you the operating system version on which it is running!

The statement is easy to remember and gives you all the details that you need.  Well, that depends on the situation of course.  In one of my previous jobs I was, as developer, last-line support for an application that was used worldwide.  Which means that now and then I had people on the phone that had “some vague issue” with our application.  And even though I speak four languages (more or less), I can assure you that getting details through a phone line can be a tough job.  I remember one case where I had to spend half an hour to get the person on the other side find the Start button.  Yes, that stupid thing on the bottom left, the thing that you need all the time as a regular Windows user.  And no, they we’re not on Windows 3.x anymore!  In those times, it even had the letters S T A R and T all over it!  But alas, it remained impossible to find.  Then I switched to finding the clock.  Guess what: wasn’t in the right bottom corner either!  Further during the conversation I tried all other corners, also tried to explain that the status bar may have been set to automatically hide.  Oh well, at a certain point they’d suddenly found it!  I don’t know where, maybe on another PC, at that point I didn’t care anymore, I just wanted them to open up the Management Studio, er, Query Analyzer more likely.

Anyway, at times like that, you’re glad that easy SQL commands still exist.

The statement above is not the only possibility to get details about the SQL Server version.  Another method is through the SERVERPROPERTY function.  Here’s a sample statement (from the Books Online) to retrieve version information:

SELECT SERVERPROPERTY('ProductVersion') AS ProductVersion,
    SERVERPROPERTY('ProductLevel') AS ProductLevel,
    SERVERPROPERTY('Edition') AS Edition,
    SERVERPROPERTY('EngineEdition') AS EngineEdition;

Try getting that to the other side of the phone line!  And no, email was not an option either…

Here’s the statement’s output:

Not only is the statement more difficult to remember, the output is less readable too.  For instance you need to know that EngineEdition 3 means Enterprise Edition.

Nevertheless, it is a useful function with more possibilities than @@version.  Check out the BOL for all the details and available properties.

Let’s get back on topic now.  I was going to mention a certain page on the Microsoft Support site.  Here it is: How to identify your SQL Server version and edition.

I think this page is a great reference in terms of versions.  If someone wants to know what version of SQL Server he’s running and doesn’t know the version numbers, this is the place to be!  I just whish that Microsoft would keep it more up-to-date.  At this moment it’s not even mentioning SQL Server 2008 R2, while that was released about half a year ago.  No sign of SP2 for SQL Server 2008 either.  I’m going to provide feedback through the textbox at the bottom of the page with this exact statement.  Hopefully that has some effect.

In case you are looking for those latest version numbers, here they are:

Have fun!

Valentino.

How do I get my data in Excel?

With a simple copy/paste.  Since SQL Server 2008, you can just right-click on the results in the Management Studio and select Copy with Headers.  This will put the selected data into memory, including the column headers.

You can even make a selection in the Results pane and only copy that, as shown in the screenshot above.  Here’s what it looks like when pasted into Excel:

How do I make the columns wider?

Excel will not expand the column widths by default, shown in that last screenshot.  You could manually enlarge them one by one, but that’s not convenient when you last filled column is called “DI”.  Instead, I use the following trick.  First I select the whole sheet by clicking on the grey square in the top-left corner:

Then I double-click on the vertical divider in between columns A and B.  This will adjust all column widths so that the data fits nicely.

Does Excel know SELECT DISTINCT?

The other day I needed to filter out all duplicate records out of a really large dataset.  I first tried by activating the regular filter.  This is an Excel feature that I use a lot.  Just clicking the Filter button on the Data ribbon will add the small dropdowns next to each column’s header:

But in those dropdowns I couldn’t find an option which would get me to the required result.

(Note that I added a couple of duplicate records in my dataset.)

After a little search, I found the magic checkbox!  I didn’t know that it could be so easy.

To find it, click the Advanced button in the Data ribbon.

In the Advanced Filter window that appears you see a checkbox called Unique records only.

There’s your SELECT DISTINCT!

And here’s the sheet with the duplicates filtered out:

Do you also see that the filter dropdowns are gone, although I did not remove them?  The Advanced Filter cannot be used in combination with the regular filter.  Once you activate the regular filter by clicking the Filter button, your duplicates will re-appear.

If you want to further filter your data after filtering out all duplicate records, you should select the Copy to another location option in the Advanced Filter window.  Copy it below your data on the same sheet and apply the regular filter on that new set of data.

Copying to another sheet is not supported.  Apparently you can only copy to the active sheet.  Should you try it out anyway, you’ll get this message:

Happy filtering!

Valentino.

And during a schema compare I noticed that the dude can be quite stubborn now and then.

Here’s an example and an explanation of why, as far as I can tell, the dude is behaving that way.

The Scenario

At work we use the data dude’s schema compare to prepare our deployment scripts.  The most fun can be had when preparing a script to deploy the latest finalized version, version x, from the Development (DEV) to the Acceptance (UAT) environment.

In our DEV database there have already been changes for the next version, version x+1, so these changes need to be skipped.

Now imagine the following situation:

• a table called TableA exists
• a new field called NewFieldA was added to TableA
• a view called ViewA exists and was modified – it uses TableA but not NewFieldA

The changes to ViewA belong with version x, while the changes to TableA are part of version x+1.  So the script that I’m going to generate should contain an ALTER VIEW ViewA … but not an ALTER TABLE TableA ….

The Behavior

Using the dude, I open up the dropdown on the line that states Different definition on TableA and I select Skip.  The value gets changed from Update to Skip.  However, when the focus changes to another line in the grid the value changes back to Update!  Furthermore, it’s now grayed out, impossible to change!

The first time that you see this happening you can’t believe your eyes and try it once more, first closing everything and then re-doing your actions.  Just to see if you didn’t select the wrong option somewhere.  But alas, the dude remains stubborn!

Tip: if you have just executed the compare and you find that some update actions cannot be modified, try to do a refresh (right-click on the background of the comparison window and click Refresh).  I noticed that after the refresh you will get the dropdown for these lines and thus will be able to change the Update Action.

The Explanation

It took me a while to realize, but here’s what seems to be happening.  Like with all things stubborn, it helps if you know what the thing in question is actually being stubborn for.

The dude knows that ViewA depends on TableA and it also knows that both ViewA and TableA where changed.  However, the dude is a little short-sighted, it doesn’t know that NewFieldA is not used in ViewA.  It doesn’t know that the change to ViewA was not related to the change on TableA.  So it automatically assumes that you’re making a mistake and prevents you from messing up.  It thinks, “You stupid user, I know it better, I am smart software!  And that’s not all, I AM THE DATA DUDE! And if there’s one thing you don’t want, that’s to mess with the Data Dude!”.

Here’s a screenshot of the dude’s stubbornness:

In this particular case it refused to skip certain new tables, even though there were no dependent objects that needed those tables!  The first two lines show the grayed-out items.  These tables were located under a new schema and I had to put the schema’s Update Action to Skip to get the tables be skipped as well.  The Update Action for the tables was changed automatically when I modified the action for the schema.

But, if you ask me, I don’t see why I shouldn’t be able to Skip a new table even when I am going to create the new schema to which it belongs?  It’s the table that depends on the schema, not the other way around.

The Feature Request

It would have been great if I could manually override the dude’s decision.  As that was not the case, I have to manually change the generated script so that it only contains the changes that I want.  As long as it’s technically feasible, I think manual changes to the Update Action should be allowed.

One More Tip

Take a good look at the following screenshot.

By default, the node that reads Different dependencies was collapsed.  So you think, good, the line says Skip so nothing will happen.  Except, did you notice that minuscule exclamation mark in the bottom-left corner of the icon?  That means that other actions may be happening further down the dependency tree, as illustrated in the screenshot.

So don’t let the dude mislead you into thinking nothing will happen when it says Skip.  Sometimes the actual actions are hidden further down and you need to open up every node with the exclamation mark on the icon to verify if that’s what you want.

Have fun training the dude,

Valentino.

For the examples I will use data from the AdventureWorks2008R2 database, available at CodePlex.

The Data

The AdventureWorks 2008 R2 database contains a view called Sales.vSalesPerson.  This is the data with which I’ll be working in the examples below.  Here’s what it looks like:

I’ve hidden some fields so that all the relevant ones are in view.

The Scenario

Your manager has asked you to create one query, to be executed on the Sales.vSalesPerson table, that returns a list of:

• all employees (FirstName, LastName, JobTitle, CountryRegionName, StateProvinceName, City),
• their sales of last year (SalesLastYear),
• the sum of the sales of last year for their country,
• the average of the sales of last year compared to all employees with the same type of phone (PhoneNumberType)
• the overall average and sum of the sales of last year.

Using Derived Tables

No problem you say, coming right up.  So you start building your query, retrieving all fields as requested.

After quite some typing, here’s what your query looks like:

select S.FirstName, S.LastName, S.JobTitle, S.PhoneNumberType, S.CountryRegionName,
    S.StateProvinceName, S.City, S.SalesLastYear,
    GeographicSales.SalesLastYearGeographic_SUM,
    SalesByPhoneType.SalesLastYearByPhoneNumberType_AVG,
    SalesSUM.SalesLastYear_AVG, SalesSUM.SalesLastYear_SUM
from Sales.vSalesPerson S
--Derived Table 1: the overall aggregates
cross join (
    select SUM(SalesLastYear) SalesLastYear_SUM, AVG(SalesLastYear) SalesLastYear_AVG
    from Sales.vSalesPerson
) SalesSUM
--Derived Table 2: the aggregate on Country level
inner join (
    select CountryRegionName, SUM(SalesLastYear) SalesLastYearGeographic_SUM
    from Sales.vSalesPerson
    group by CountryRegionName
) GeographicSales on GeographicSales.CountryRegionName = S.CountryRegionName
--Derived Table 3: the aggregate on phone type
inner join (
    select PhoneNumberType, AVG(SalesLastYear) SalesLastYearByPhoneNumberType_AVG
    from Sales.vSalesPerson
    group by PhoneNumberType
) SalesByPhoneType on SalesByPhoneType.PhoneNumberType= S.PhoneNumberType;

The main query is retrieving all fields as requested.  Further down there are three derived table queries, each one retrieving aggregates on a different level.

The first derived table is retrieving the overall aggregates.  These are cross-joined with every record in our main query so for each record the totals will be the same, which is what we want.

The second derived table retrieves the aggregates on Country level, including the CountryRegionName.  This is done using the conventional GROUP BY method.  The CountryRegionName is the key on which the derived table is joined to the main table.

The third derived table uses a similar system, this time for the aggregate on phone type.

And here’s the query’s output:

Happy with this result, you go up to the cafeteria to finally have lunch with your colleagues (who left 15 minutes earlier but you wanted to get your query finished first).

Using The OVER Clause

During lunch you explain to your peers what kind of funny request you got from management and told them how you solved it.

Then one of them speaks up and says: “Want to know how you can avoid all that typing?  Use the OVER clause!  I’ll show you when we are back at our desks.”

After lunch, here’s what your colleague helps to produce:

select S.FirstName, S.LastName, S.JobTitle, S.PhoneNumberType, S.CountryRegionName,
    S.StateProvinceName, S.City, S.SalesLastYear,
    SUM(SalesLastYear) OVER (PARTITION BY CountryRegionName)
        SalesLastYearGeographic_SUM,
    AVG(SalesLastYear) OVER (PARTITION BY PhoneNumberType)
        SalesLastYearByPhoneNumberType_AVG,
    SalesSUM.SalesLastYear_AVG, SalesSUM.SalesLastYear_SUM
from Sales.vSalesPerson S
--Derived Table 1: the overall aggregates
cross join (
    select SUM(SalesLastYear) SalesLastYear_SUM, AVG(SalesLastYear) SalesLastYear_AVG
    from Sales.vSalesPerson
) SalesSUM;

As you can see, derived tables 2 and 3 are gone.  They have been replaced with the OVER clause, in combination with PARTITION BY.  What you say with the OVER clause is: “partition the dataset by the fields specified in the PARTITION BY and apply the aggregation on those partitions”.  Another word for this is aggregate window function.

As you like the approach, you ask your co-worker how you can get rid of that cross join.  He doesn’t really know but then another colleague who overheard your conversation says: “On this blog the other day I read that you can use the OVER clause and partition by anything you want.  As long as it’s a constant, it will work!”.

So you give that a try and you end up with the following final query:

select FirstName, LastName, JobTitle, PhoneNumberType, CountryRegionName,
    StateProvinceName, City, SalesLastYear,
    SUM(SalesLastYear) OVER (PARTITION BY CountryRegionName)
        SalesLastYearGeographic_SUM,
    AVG(SalesLastYear) OVER (PARTITION BY PhoneNumberType)
        SalesLastYearByPhoneNumberType_AVG,
    AVG(SalesLastYear) OVER (PARTITION BY 'duh') SalesLastYear_AVG,
    SUM(SalesLastYear) OVER (PARTITION BY 1) SalesLastYear_SUM
from Sales.vSalesPerson;

As illustrated in the example, you can use any constant value to calculate overall aggregates over the whole dataset using the OVER clause.

You happily thank your colleagues and tell them that next time you’ll be able to join them for lunch on time.

About a week later you’re explaining to one of your friends how you’ve gotten to know the OVER clause. After hearing how you use it to aggregate over the whole dataset, he smiles and says: “I know how you can simplify it even more! Don’t partition at all!”.

Taking a closer look it turns out that the PARTITION BY is actually optional:

Ranking Window Functions
< OVER_CLAUSE > :: =     OVER ( [ PARTITION BY value_expression, ... [ n ] ]            <ORDER BY_Clause> )Aggregate Window Functions
< OVER_CLAUSE > :: =     OVER ( [ PARTITION BY value_expression, ... [ n ] ] )

See those square brackets? Means it’s optional.

So here is the real final query:

select FirstName, LastName, JobTitle, PhoneNumberType, CountryRegionName,
    StateProvinceName, City, SalesLastYear,
    SUM(SalesLastYear) OVER (PARTITION BY CountryRegionName)
        SalesLastYearGeographic_SUM,
    AVG(SalesLastYear) OVER (PARTITION BY PhoneNumberType)
        SalesLastYearByPhoneNumberType_AVG,
    AVG(SalesLastYear) OVER () SalesLastYear_AVG,
    SUM(SalesLastYear) OVER () SalesLastYear_SUM
from Sales.vSalesPerson;

Conclusion

When you compare the final query with the first one, tell me, which one would you prefer to maintain?  Do you prefer to have lunch with your peers or to arrive late and miss all the fun?

Have fun!

Valentino.

References

OVER Clause (Transact-SQL)

Join Fundamentals

SELECT (Transact-SQL)

Introduction

Anyone who’s ever used a computer for a significant amount of time has probably come into contact with Excel, the spreadsheet application part of the Microsoft Office suite. Its main purposes are to perform calculations and create charts and pivot tables for analysis.

But people have great imagination and invent new uses for it every day.  I’ve even seen it used as a picture album.  (Sorry dad, but I know you won’t be reading this anyway. )  Ever since he had this specific YACI, or “Yet Another Computer Issue”, because his PC wasn’t powerful enough to open his 45 MB Excel file, uh, “picture collection”, he took some evening classes.  He’s now putting his Photoshopped pictures in PowerPoint…  Anyway, let’s get back on track now.

Another use, and the one that’s the subject of this article, is when Excel has been used as a database.  Come on, you know what I’m talking about, with the first row containing the column headers followed by possibly thousands of data rows.  The following screenshot contains an example, and is also the file that I will be using in this article.  I took all records from the Production.Product table in the AdventureWorks 2008R2 database and dumped them in Excel.

At some point people will realize, either because someone told them or because they lost some data due to inattentiveness, that it wasn’t a really good idea to keep all that data in an Excel sheet.  And they’ll ask you to put it in a real database such as SQL Server.

That’s what I’m going to show you in the next paragraphs: how to import data from Excel into SQL Server.

Using OPENROWSET() To Query Excel Files

There are actually several different ways to achieve this.  In this article I will use the OPENROWSET() function.  This is a T-SQL function that can be used to access any OLE DB data source.  All you need is the right OLE DB driver.  The oldest version which I could confirm that contains this function is SQL Server 7.0, good enough to say that any version supports it.

My sample Excel files are located in C:\temp\.  This folder contains two files: Products.xls and Products.xlsx.  The first file is saved in the old format, Excel 97-2003, while the second file was saved from Excel 2010.  Both files contain the same data.  The sheet containing the list of products is called ProductList.

And here are the queries:

--Excel 2007-2010
SELECT * --INTO #productlist
FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0',
    'Excel 12.0 Xml;HDR=YES;Database=C:\temp\Products.xlsx',
    'SELECT * FROM [ProductList$]');

--Excel 97-2003
SELECT * --INTO #productlist
FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0',
    'Excel 8.0;HDR=YES;Database=C:\temp\Products.xls',
    'select * from [ProductList$]');

These queries are just returning the data from the Excel file into the Results window, when executed using the Management Studio.  To insert the data into a table, uncomment the INTO clause.  When uncommented, the statement retrieves the data from the Excel sheet and puts it into a newly-created local temporary table called #productlist.

Furthermore, the query assumes that the first row contains the header.  If that’s not the case, replace HDR=YES with HDR=NO.

Note: if you get an error message when running the query, look further down in this article.  I’ve covered a couple of them.

With the INTO clause uncommented and the query executed, the temporary table can now be queried just like any other table:

SELECT * FROM #productlist

What Type Is Your Data?

Let’s have a look if this method of using a SELECT INTO in combination with OPENROWSET and a temporary table is smart enough to interpret the correct data types of the data coming in.  Use the following command to describe the metadata of the temporary table:

USE tempdb;
GO
sp_help '#productlist';

Because a temporary table is stored in the tempdb, the sp_help command should be issued against that database.

Here’s the part of the output in which we’re interested:

As you can see, anything that looks like text will be put in a field of type nvarchar(510) and anything that looks like a number (integers, floating-point numbers, datetime values, …) is put into a float(53).  Not a lot of intelligence there.  This is the result when no formatting was put on the cells in Excel.

As an experiment I’ve changed the format of some fields in the Excel file and then retried the SELECT INTO statement.  What did I change?  I identified ProductID as being a number without any decimals, changed StandardCost and ListPrice to a currency with four decimal digits and I changed SellStartDate and SellEndDate to a custom date/time format showing both date and time.

The effect on the table creation was not completely as I would have expected:

ProductID is still being stored into a float field, even though in Excel it’s defined as having no decimals.  And the datetime values are not recognized either.  Okay, I used a custom format there, so maybe it’s due to that.

It’s up to you of course how you use this method of importing the data.  You can put your records into a temporary table to process further, or you can create a table with the expected data types upfront and import the data directly into that one.

Some Possible Issues

Let’s cover some issues related to this method.

Enable ‘AD Hoc Distributed Queries’

The OPENROWSET() function expects that the ‘Ad Hoc Distributed Queries’ option is enabled on the server.  When that’s not the case you’ll see the following message:

Msg 15281, Level 16, State 1, Line 1

SQL Server blocked access to STATEMENT ‘OpenRowset/OpenDatasource’ of component ‘Ad Hoc Distributed Queries’ because this component is turned off as part of the security configuration for this server. A system administrator can enable the use of ‘Ad Hoc Distributed Queries’ by using sp_configure. For more information about enabling ‘Ad Hoc Distributed Queries’, see “Surface Area Configuration” in SQL Server Books Online.

This is one of the advanced options.  To enable it you can use the following command:

sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO

sp_configure 'Ad Hoc Distributed Queries', 1;
GO
RECONFIGURE;
GO

To get a good look at all the different settings, just run the sp_configure procedure without any parameters.

Note: if you’re not the administrator of the server, you should talk to the DBA who’s responsible before attempting this.

The File Needs To Be Closed

When the Excel file is not closed, you’ll end up with the following error:

Msg 7399, Level 16, State 1, Line 1

The OLE DB provider “Microsoft.Jet.OLEDB.4.0″ for linked server “(null)” reported an error. The provider did not give any information about the error.

Msg 7303, Level 16, State 1, Line 1

Cannot initialize the data source object of OLE DB provider “Microsoft.Jet.OLEDB.4.0″ for linked server “(null)”.

So close the file and try the query again.

OLE DB Driver Not Installed

The OPENROWSET() function uses OLE DB, so it needs a driver for your data source, in this case for Excel.  If the right driver is not installed, you’ll see the following error (or similar, depends on the version used).

Msg 7302, Level 16, State 1, Line 1

Cannot create an instance of OLE DB provider “Microsoft.ACE.OLEDB.12.0″ for linked server “(null)”.

To solve the issue, install the right driver and try again.

How can you tell what drivers are installed?  Open up the ODBC Data Source Administrator window (Start > Run > type ODBCAD32.EXE and enter) and have a look in the Drivers tab.  The following screenshot (taken on a Dutch Windows XP) shows both the JET 4.0 driver for Excel 97-2003 and the fairly-new ACE driver for Excel 2007.

The drivers can be downloaded from the following pages on the Microsoft site:

Excel 97-2003 Jet 4.0 driver

Excel 2007 ACE driver – 12.00.6423.1000

Excel 2010 ACE driver (beta) – 14.00.4732.1000

Sidenote: the Excel 2010 driver is not supported on Windows XP, but I was able to query the 2010 Excel sheet using the 2007 driver.  I guess that this is the result of the Office Open XML standard which was introduced in Office 2007.

Driver backward-compatibility

The ACE drivers are backwards-compatible.  So the following queries are working perfectly:

--old Excel with new ACE driver - working query 1
SELECT * --INTO #productlist
FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0',
    'Excel 8.0;HDR=YES;Database=C:\temp\Products.xls',
    'SELECT * FROM [ProductList$]');

--old Excel with new ACE driver - working query 2
SELECT * --INTO #productlist
FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0',
    'Excel 12.0;HDR=YES;Database=C:\temp\Products.xls',
    'SELECT * FROM [ProductList$]');

In other words, you won’t be needing that first link for the Jet driver.  For the full story have a look at this blog post by Adam Saxton of the CSS SQL Server Escalation Services team.

The 64-bit Story

So, what if you’re running a 64-bit OS?  I’ll start by saying that I had quite some issues getting OPENROWSET to work, but finally I managed it.  Following is a list of my attempts, each time with the resulting message.  And finally I’ll show you how I got it to work.  The problem was something really unexpected…

ACE 14 64-bit through SSMS

My main laptop is running Windows 7 64-bit, Office 2010 64-bit and SQL Server 2008 R2 64-bit.  So I installed the 64-bit version of the ACE 14 driver, which happens to be the first OLE DB driver for Excel that ships in 64-bit.  But when I execute my query I’m getting the following message:

Msg 7403, Level 16, State 1, Line 1

The OLE DB provider “Microsoft.ACE.OLEDB.14.0″ has not been registered.

Is this because SSMS ships only in 32-bit?  Maybe, but I’m not able to install the 32-bit driver.  It doesn’t allow me to because I’ve got Office in 64-bit installed.  The installer throws me the following error:

ACE 12 32-bit on a 64-bit machine

When I check the installed drivers using the 32-bit version of the ODBC Data Source Administrator (located in C:\Windows\SysWOW64), I notice that the ACE 12 driver is installed.  However, trying to use that one from the Management Studio gives me this:

Msg 7399, Level 16, State 1, Line 1

The OLE DB provider “Microsoft.ACE.OLEDB.12.0″ for linked server “(null)” reported an error. The provider did not give any information about the error.

Msg 7330, Level 16, State 2, Line 1

Cannot fetch a row from OLE DB provider “Microsoft.ACE.OLEDB.12.0″ for linked server “(null)”.

The Results pane shows all the columns with the right column names, retrieved from Excel.  But the driver seems to have a problem retrieving the actual data.

This issue with error 7330 is mentioned in the following thread on the SQL Server MSDN forum, but unfortunately the proposed solution didn’t solve the problem in my case.

64-bit SQLCMD using ACE 14 driver

I also tried using the 64-bit version of sqlcmd.exe, but strangely enough that throws the same error.

I actually expected this last method to work, after all, everything is now running in 64-bit.  But alas, it didn’t…

One more go…

After some more trial and error, I have actually found a way to get the query to work.  I don’t have a logical explanation on why it’s behaving the way it is, but, well, it is working…

This query is running fine:

SELECT * --INTO #productlist
FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0',
    'Excel 12.0 Xml;HDR=YES;Database=C:\temp\Products.xlsx',
    'SELECT * FROM [ProductList$]');

But this one isn’t:

--Excel 2007-2010
SELECT * --INTO #productlist
FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0',
    'Excel 12.0 Xml;HDR=YES;Database=C:\temp\Products.xlsx',
    'SELECT * FROM [ProductList$]');

It’s exactly the same query, only difference is the comment line at the start.  And even weirder, if I add a space after the double-dash, the query works fine as well!

Then I decided to remove the commented INTO clause.  This made the weird behavior disappear.  So for some reason SQL Server doesn’t like the OPENROWSET function combined with comments inside the query.  The strange behavior also disappears when a space is added between the double-dash and the INTO keyword.

Uh, computers can be so much fun, right?

If anyone has got an explanation on this strange behavior: please do post a comment!  For now my conclusion is: don’t use comments when creating an OPENROWSET query.

IMPORTANT UPDATE (April 11, 2010): it seems that the current installer for the ACE 14 driver contains a bug and registers it as being “Microsoft.ACE.OLEDB.12.0” instead of “Microsoft.ACE.OLEDB.14.0” .  This explains some of the issues shown above.  Some evidence on the issue:

Microsoft Connect: Access Database Engine 2010 installation issue to use with ADO access technology to access data from Jet database (.mdb files)

The ‘Microsoft.ACE.OLEDB.14.0′ provider is not registered ….. (see last comment)

Excel Services, ODC and Microsoft.ACE.OLEDB.14.0

Conclusion

The above has shown that OPENROWSET() can be a useful function, given the right circumstances.  But in the wrong setting it can be quite cumbersome to get to work.

I would recommend this method only for one-off quick imports, such as when you as a developer are given a bunch of data in a spreadsheet and need to get it into the database, one way or another.  I would not use it for an automated import process.  For that we’ve got a more interesting alternative which I’ll cover in an upcoming article.

Have fun!

Valentino.

References

BOL 2008: Special Table Types (incl. temporary tables)

BOL 2008: OPENROWSET() function

BOL 2008: the INTO clause

CSS SQL Server Engineers: How to get a x64 version of Jet?

If you’re still on 2005 and thus more interested in an SP4 for that version, it has also been posted at Connect.

How did I find out?  Through the following blog posts at SSQA.NET:

SQL Server 2008 SP2

SQL Server 2005 SP4

Update (27 Feb 2010):

SQL Server 2008 SP2 is scheduled for Q3 2010.

SQL Server 2005 SP4 is scheduled for Q4 2010.

See this post on the Microsoft SQL Server Release Services blog for details.