Finally time to retire SQL Server 2005

Introduction

More than ten years ago, on November 5, 2005, Microsoft released SQL Server 2005 to manufacturing.  Over that span of time, it has become one of the most venerable -- and solid -- releases in the product line.  All good things must come to an end though and the same is true of SQL 2005. It has been in extended support since April of 2011 but even that will end on April 12, 2016.  see Extended support for SQL Server 2005 ends on Aptil 12, 2016  for more information.

This article highlights some of the innovations it brought us but more importantly, shows the developments since November 2005 and SQL.Next: SQL Server 2016.  

Highlights of SQL Server 2005

For nostalgia's sake, you might want to review the details at What's New in SQL Server 2005, but here are a few notable enhancements brought to us in that release:

• Service Broker
• CLR integration
• The XML data type
• Ranking functions
• Common Table Expressions 
• Pivot and Unpivot
• Database schemas
• Partitioning
• New architecture for Integration Services and dtsx packages
• Reporting Services as a new feature
• Multiple enhancements for Analysis Services
• Replication enhancements including transactional, peer to peer and merge replication and support for  heterogeneous environments

Since 2005 ...

SQL Server releases 2008, 2008 R2, 2012 and 2014 greatly extended the power of Microsoft's enterprise-grade RDBMS and SQL Server 2016, due out in just a few months, promises to do the same.  It's worth reviewing all the "What's new" pages which you can find using these links:

• SQL Server 2008
• SQL Server 2008 R2
• SQL Server 2012
• SQL Server 2014
• SQL Server 2016

Just to name a few of the multitude of enhancements since 2005:

• Slipstream installations (2008)
• Filtered indexes (2008)
• PowerPivot (2008 R2)
• Master Data Services (2008 R2)
• Data Tier Applications (2008 R2)
• File tables (2012)
• Semantic search and enhanced full-text search (2012)
• Sequence objects (2012)
• OVER clause and analytic functions (2012)
• Columnstore indexes (2012)
• Always On (2012)
• Data Quality Services (2012)
• Memory optimized tables and compiled stored procedures (2014)
• Data files in Azure (2014)
• Clustered columnstore indexes (2014)
• Stretch database (2016)
• R integration (2016)
• JSON support (2016)
• Query store (2016)
• Temporal tables (2016)

Of course this is just a list of highlights that I happen to like!  Please follow the links to get the full history of improvements.

I need to upgrade! But...

Upgrading from SQL Server 2005 to SQL Server 2014 is not an exercise for the faint of heart.  To begin with, you cannot upgrade directly from RTM 2005 to 2014.  You must have installed Service Pack 4 of SQL Server 2005 before you begin.  Also, if you want to upgrade to version 2016, you will be facing a two-part upgrade, since 2016 no longer supports any version of 2005 as a base for upgrading.  

In addition, before you plan your SQL Server migration, you may need to plan a Windows Server upgrade.  Many SQL Server 2005 installations are still running on Windows Server 2003, but the minimum version of Windows Server supported for SQL Server 2014 is Windows Server 2008 R2 SP1,  For SQL Server 2016 the minimum is Windows Server 2012 R2.  Note that Windows Server 2003 extended support ended on July 14, 2015, so if you are still running that version and something breaks, you're on your own unless you opt for pay-as-you-go support.

For installations using mirroring or replication, special care must be taken to ensure minimum downtime and avoiding running "exposed" (that is, with only one active and no passive instances). A good discussion of this topic can be found here: Minimize Downtime for Mirrored Databases When Upgrading Server Instances from a 2016 perspective, but the same principles apply for other versions.

Should I stay or should I go now?

When you begin to consider upgrading a SQL Server 2005 instance to SQL Server 2014/16 -- especially when you may also need to upgrade Windows Server 2003 as well -- you may wonder if it is worth it.  After all, it will cost you time and resources, not to mention possible changes in licensing costs.  Considering the approaching End of Extended support for the product (and the fact that all support for Windows Server 2003 has ended), it's probably a good idea to plan for the upgrade, build a realistic project plan with appropriate budgetary and resource support, get your team trained for the upgrade(s) and the new world of SQL Server 2014/16 and set specific timelines.

On the other hand, if your SQL Server 2005 instance is not supporting business critical processes and you can afford to lose it entirely, you may want to stand pat.  But then, why are you still running that system at all?

Double Abstraction + Principle of Least Priviledge = Easy-to-manage Security Part IV

Introduction

In the first three parts of this series, we looked at a method to set up easily-manageable security using two principles:

1. Double abstraction
• Put all users into Windows groups
• Put all permissions into custom database (or server, for SQL Server 2012 and up) roles
• Add the groups to the custom roles
2. Principle of least privilege
• Only grant the minimum privileges needed to get the job done

If you've followed along and implemented or at least thought about implementing this strategy, you may be wondering how to enforce it.  After all, if the option still exists for some DBA to add an individual login to your instance or individual users to a database or grant specific permissions to specific users that are not custom database roles, you want to know if it's happening.

SQL Server Auditing to the rescue!

SQL Server Auditing

Quoting from BOL: SQL Server Audit (Database Engine)

Auditing an instance of the SQL Server Database Engine or an individual database involves tracking and logging events that occur on the Database Engine. SQL Server audit lets you create server audits, which can contain server audit specifications for server level events, and database audit specifications for database level events. Audited events can be written to the event logs or to audit files.

Setting up Auditing is easy.  Let's set up an audit to catch anyone violating the DA-PLP  regime we've established. In SSMS you open the Security/Audits sub tree.  There you create a server audit for this purpose, a server audit specification hooked up to the server audit and enable them both.  Graphically:

In T-SQL

USE [master] GO CREATE SERVER AUDIT [Audit-20150828-112959] TO FILE ( FILEPATH = N'C:\temp' ,MAXSIZE = 0 MB ,MAX_ROLLOVER_FILES = 2147483647 ,RESERVE_DISK_SPACE = OFF ) WITH ( QUEUE_DELAY = 1000 ,ON_FAILURE = CONTINUE ) GO

Now the Specification:

Or, in T-SQL:

USE [master] GO CREATE SERVER AUDIT SPECIFICATION [ServerAuditSpecification-20150828-113306] FOR SERVER AUDIT [Audit-20150828-112959] ADD (SERVER_ROLE_MEMBER_CHANGE_GROUP), ADD (DATABASE_ROLE_MEMBER_CHANGE_GROUP) GO

Now, simply enable the audit and audit specification (right-click on them, select Enable) and you're ready to catch violaters!  Note, that, if you chose to go with a file as I did, you can use the system function sys.fn_get_audit_file to view it in tabular format.  There you will see anyone (with their login id) who changed anything in these areas and you can easily sort out legitimate changes from rule breakers.

Summary

With this article, we wrap up this series.  We've discussed two important principles: Double Abstraction and the Principle of Least Privilege that together make our lives easier for managing security in SQL Server. 

Double Abstraction + Principle of Least Priviledge = Easy-to-manage Security Part III

Introduction

In the previous two posts, we discussed the concept of double-abstraction security for SQL Server.  The motivation is that we do not want to be administering permissions at an individual user to individual object basis.  This does not scale well and if you're anything like me, you'll make mistakes with that approach.

The double-abstraction approach makes it much easier.  

1. Put users into groups that reflect their business needs
2. Create database roles to hold the permissions to fit those needs
3. Add groups to roles

In this section, we're going to put all three things together.  We'll be using the same database as in parts I and II:

Double Abstraction Part I
Double Abstraction Part II

Set up users and groups

Before we get to SQL, we need to set up Windows users and groups (Step 1 above).  In Computer Management, we add the user like this:

To keep things simple, we checked both "User cannot change password" and "Password never expires" (not recommended for actual use!).  Next, we'll create a group and add the user to the group:

Now that we have a user and group to work with, let's add a new login to SQL Server for the group:

USE [master] GO CREATE LOGIN [JERRY-PC\SecureGroupReadOnly] FROM WINDOWS WITH DEFAULT_DATABASE=[SecureDB] GO

Next, we need to add the group to the database as a user:

USE [SecureDB] GO CREATE USER [SecureGroupReadOnly] FOR LOGIN [JERRY-PC\SecureGroupReadOnly] WITH DEFAULT_SCHEMA=[SecureSchema] GO

We need a new database role for this read-only group to use. We'll give it only SELECT and EXECUTE permissions in the SecureSchema:

-- Create the new role USE [SecureDB] GO CREATE ROLE [SecureRoleReadOnly] AUTHORIZATION [SecureRole] GO USE [SecureDB] GO -- Add the restricted group to the role ALTER ROLE [SecureRoleReadOnly] ADD MEMBER [SecureGroupReadOnly] GO USE [SecureDB] GO -- Grant EXECUTE and SELECT permissions GRANT EXECUTE ON SCHEMA::[SecureSchema] TO [SecureRoleReadOnly] GO USE [SecureDB] GO GRANT SELECT ON SCHEMA::[SecureSchema] TO [SecureRoleReadOnly] GO

That's it!  Now, all we have to do to manage access is to add/remove users from the Windows (or Active Directory) group!

Double Abstraction + Principle of Least Priviledge = Easy-to-manage Security Part II

Intoduction

In Part I of this series, we introduced the idea of Double Abstraction as a way to manage permissions with ease and consistency.  The basic concepts are:

1. Use Server and Database custom roles to hold permission grants
2. Add Windows groups to the roles
3. Add users to the Windows Groups

We combine this with the principle of least privilege.  That is, we ensure that only the permissions actually required to perform the duties of the role (in a business sense) are granted to the role.

In this post, we will work through an example to demonstrate some of these ideas.  We'll create a new database, a database schema, a database role, and a SQL user to add to the role.  In Part III, we'll extend the ideas to use Windows groups for management.

So, let's begin!  

1. Create a database:

-- Create a database to demonstrate security principles USE master; GO -- Delete database if it already exists IF db_id('SecureDB') IS NOT NULL BEGIN ALTER DATABASE SecureDB SET SINGLE_USER WITH ROLLBACK IMMEDIATE; DROP DATABASE SecureDB; END CREATE DATABASE SecureDB;

2. Switch to the context of the new database and create a database role:

-- Change the database context to the new database USE SecureDB; GO -- Create a role in the new databse CREATE ROLE SecureRole; GO -- View the attributes of the new role SELECT * FROM sys.database_principals WHERE name = 'SecureRole'; GO

3. Create a database schema and a user with access to it:

-- Create a schema to hold secure objects owned by the new role CREATE SCHEMA SecureSchema AUTHORIZATION SecureRole; GO -- View attributes of new schema. Note that the principal_id matches the one in sys.database_principals SELECT * FROM sys.schemas WHERE name = 'SecureSchema' GO -- Create a database user without a matching SQL Server Login. CREATE USER SecureUser WITHOUT LOGIN WITH DEFAULT_SCHEMA = SecureSchame; GO -- View attributes of new user. Note default schema. SELECT * FROM sys.database_principals where name = 'SecureUser' -- Grant new user permission to create tables in the database GRANT CREATE TABLE TO SecureUser; GO

4. The new schema is owned by the new role and the user has database permissions to create tables. Let's try that:

EXECUTE AS user = 'SecureUser'; CREATE TABLE SecureSchema.SecureTable (SecureId INT); GO -- Switch the execution context back to the caller of the last EXECUTE AS statement. REVERT;

5. Whoops! That didn't work! If you are following along on your own SQL Server instance, you should have seen an error message like this:

Msg 262, Level 14, State 1, Line 42

CREATE TABLE permission denied in database 'SecureDB'.

The thing is, the user needs ALTER TABLE permission in addition to CREATE TABLE. The database schema is owned by the custom role, so let's add the user to that role and try again:

ALTER ROLE SecureRole ADD member SecureUser; -- ALTER ROLE SecureRole DROP member SecureUser GO -- Set the execution context to the newly-created user and try to create a table EXECUTE AS user = 'SecureUser'; SELECT SUSER_NAME() CREATE TABLE SecureSchema.SecureTable (SecureId INT); -- Switch the execution context back to the caller of the last EXECUTE AS statement. REVERT; SELECT SUSER_NAME();

\
6. It worked! The table creation worked. Let's see if we can access it:

EXECUTE AS user = 'SecureUser'; SELECT * from SecureSchema.SecureTable; REVERT;

Success! We can access the table. Actually we can do anything to it. Since the new user is a member of the custom role that owns the schema, we have carte blanche.

Summary

In this section, we have shown how to use a database role to control who can do what.  The role we created is the owner of the schema, so members of the role can do anything, provided other requirements are met.

Next time, we'll look at creating a restricted role with very limited access, applying the principle of least privilege.

Double Abstraction + Principle of Least Priviledge = Easy-to-manage Security Part I

Introduction

Securing a SQL Server instance is critical to every business.  Whether to comply with regulatory requirements, pass external audits with flying colors or to ensure proper restrictions on access to sensitive or financial data, it is imperative that database resources are only accessed by those with the need to do so and in a manner matching requirements.  This is the first in a series of articles on managing Security in SQL Server.

Challenges

Your database is accessed by a wide variety of people with different needs.  You need to be able to secure database assets in a  manageable way.  All too often we see implementations where every user has serveradmin rights and/or database owner (dbo) rights on one or more databases in some SQL Server instance.  This is a recipe for disaster!  Even the most conscientious user will make mistakes and the careless, disgruntled and even criminally-oriented users can wreak havoc, destroy valuable data or even siphon it off to sell to the highest bidder.

We know that SQL Server has the tools to GRANT or DENY access to its objects but all too often get frustrated managing user-by-user.  This is where the double-abstraction approach shines.

Double Abstraction

The first part of Double Abstraction (DA for short) can be found by reversing the letters in the acronym.  AD, or Active Directory allows us to create groups so that birds of a feather can flock together -- that is, so that users with similar requirements can be managed as a unit by placing them into a suitable group.  So, the first step, is to identify the sorts of actions different groups might need.  A reporting team needs to be able to read some tables from one or more databases; an ETL job needs to be able to load tables; a web application needs to be able to query and possibly update certain data; auditors need to be able to monitor the security of the database; administrators at various levels of responsibility need to manage the server, create databases, backup/restore and so on.  Each of these functions (and probably others, depending on your requirements) deserves its own group.

The second part of DA exploits database roles and server roles.  SQL Server comes with many out of the box, but for our purposes it makes sense to define custom database roles and (in SQL Server 2012) custom server roles.  Each role would be given just the privileges corresponding to the actual needs of the various groups.  For example, access to certain schemas, execute access on some stored procedures, the ability to create databases, tables and other objects, the ability to view all of these privileges and so on.  We set up these roles using the Principle of Least Privilege: each role has just the privileges actually required and no more.

To bring these together -- AD groups and DB/Server roles -- we use the ALTER ROLE ADD MEMBER command (or sp_addrolemember system stored procedure).  Say that we have created an AD group Corp\SalesReports and a database role in the Sales database called SalesReaders.  Then we can:

1. Add domain users (and other groups) to the Corp\SalesReports group
2. Add the AD group to the SalesReaders role like this:

ALTER ROLE SalesReaders Add Member [Corp\SalesReports]

Next time, we'll dig deeper and add more examples.

Summary

Managing security in SQL Server is made easier with the principle of Double Abstraction: Add users to groups; create roles in the database to cover the functions; add the groups to roles.

Alternative to cursor-based code for maintenance operations

Introduction

Many maintenance operations in SQL Server require us to iterate over some list and perform one or more operations against each item in the list. For example, perhaps you want to backup a number of databases or rebuild a number of indexes. Since the basic BACKUP and ALTER INDEX commands cannot take variables for database or index names, we often see a pattern like this:

DECLARE mycursor as CURSOR FOR SELECT ... OPEN mycursor DECLARE @var1 ... FETCH NEXT FROM mycursor into @var1 ... WHILE @@FETCH_STATUS = 0 BEGIN -- do something with @var1 FETCH NEXT mycursor into @var1 END CLOSE mycursor DEALLOCATE mycursor

In Real Life

For a real-life example, consider ETL (Extract, Transform and Load).  A typical sequence runs like this:

1. Extract rows from some source
2. Change/Add (transform) the data
3. Load the rows into some target table

Often, prior to step 3, it is helpful to disable the non-clustered indexes (NCIs), then re-build them when the load is done.  (As a side note, if your table has a clustered index, and you ensure proper ordering of your data to be loaded, you want the clustered index.  You can't disable it, since that effectively disables access to the table (since the clustered index is the table)).

Using a Cursor

Here is a script that will disable the NCIs for a table in the AdventureWorks database:

DECLARE cur CURSOR READ_ONLY FAST_FORWARD FOR SELECT i.NAME ,o.NAME FROM sys.indexes i INNER JOIN sys.objects o ON i.object_id = o.object_id INNER JOIN sys.schemas s ON s.schema_id = o.schema_id WHERE i.type_desc = 'NONCLUSTERED' AND o.type = 'U' AND o.NAME = 'EmployeeDepartmentHistory' AND s.NAME = 'HumanResources'; OPEN cur; DECLARE @ix_name SYSNAME, @db_name SYSNAME; FETCH NEXT FROM cur INTO @ix_name, @db_name; WHILE @@FETCH_STATUS = 0 BEGIN PRINT 'ALTER INDEX ' + @ix_name + ' ON ' + @db_name + ' DISABLE;' + CHAR(13) + CHAR(10); FETCH NEXT FROM cur INTO @ix_name, @db_name; END; CLOSE cur; DEALLOCATE cur;

This does the job quite handily.  We can even use similar logic to rebuild the indexes when the load is done (or simply REBUILD ALL).  What I'd like to show is a method to do the same without using a cursor.

Using a set-based operation

The basic idea is to build up the ALTER INDEX commands as a single NVARCHAR string, then use sp_executesql to run it.  The code looks like this:

DECLARE @sql AS NVARCHAR(MAX) = ( SELECT 'ALTER INDEX ' + i.NAME + ' ON ' + o.NAME + ' DISABLE;' + CHAR(13) + CHAR(10) FROM sys.indexes i INNER JOIN sys.objects o ON i.object_id = o.object_id INNER JOIN sys.schemas s ON s.schema_id = o.schema_id WHERE i.type_desc = 'NONCLUSTERED' AND o.type = 'U' AND o.NAME = 'EmployeeDepartmentHistory' AND s.NAME = 'HumanResources' FOR XML PATH(''), TYPE ).value('.', 'NVARCHAR(MAX)'); PRINT @sql;

As you can see, there is less code to write and debug,  I also find it less visually distracting.  The net results are exactly the same.  The performance is also the same, except for the front-end job of building the query string.  As is usually the case, the set-based code from the second example out-performs the first.  However, since we are doing this for maintenance-type jobs, there are not millions of rows (perhaps only hundreds or thousands) so the run time of the front-end code is overwhelmed by the run time of the executed code.

Summary

For many small jobs where a cursor seems a natural way to go, there is an alternative using a set-based operation to construct a list of commands and running the list in one execution.  Run times are essentially the same but the code can be more compact and done in just two statements (build the list, run the list) instead of the several statements required by a cursor-based approach.

Set operations in SQL Server - Part 2: ANY() and ALL()

In the previous post Set operations in SQL Server - Part 1, we looked at the basic set options Union, Intersection, Complement (or subtraction) and how they can be used and combined.  We also looked at the problem of determining table equality in a relational database -- especially when duplicate rows are permitted.  In this installment, we will look at various ways of testing set membership.

Set membership is a property of an item that may or may not be a member of a given set.  In database terms, membership implies that there exists some row in a table that satisfies the membership test.  This usually means that one or more columns contain data that satisfy some constraint such as equality or inequality.  Testing for set membership is an essential operation in processing data in a relational database, such as SQL Server.

Starting easy

Let's start off with a simple example.  Suppose you have a table that contains information on bicycles.  Perhaps it looks something like this:

CREATE TABLE Bicycle ( BicycleId INT , ModelName VARCHAR(50) , Manufacturer VARCHAR(50) , Cost DECIMAL(6, 2) ) -- Add some data INSERT INTO Bicycle (BicycleId, ModelName, Manufacturer, Cost) VALUES (1, '700', 'Trek', 1000.01), (2, '500', 'Giant', 600.00)

Now, suppose this table was populated with various bike data and you want to see if there are any bicycles that cost over $1,000.00.  A typical query might look like:

IF EXISTS(SELECT * FROM Bicycle WHERE Cost > 1000.00) PRINT 'There is at least one bicycle costing over $1,000.00'

This is simple test for membership that prints a message if the constraint is satisfied. Note that it uses the function EXISTS(). There are actually four functions available to help us with membership testing like this: EXISTS(), IN(), ANY() (a.k.a SOME()) and ALL(). You're probably familiar with EXISTS() and IN() and may have used them before. Perhaps you are curious about the ANY() and ALL() functions and what they can do for you.

Using ANY()

Let's rewrite the sample query using the ANY() function instead:

IF ANY (SELECT Cost FROM Bicycle) > 1000.00 PRINT 'There is at least one bicycle costing over $1,000.00'

This query returns the same result as the first. It is actually a more direct translation of the requirement and is shorter to boot! Before we go further though, I'm going to rewrite this query this way:

IF 1000 < ANY (SELECT Cost FROM Bicycle) PRINT 'There is at least one bicycle costing over $1,000.00'

Why did I do that? Well, sometimes the query inside the call to the ALL() function can be long, making it harder to spot the condition being tested.

Using ALL()

Now that we've seen how ANY() operates, what about ALL()?  It turns out that this is fairly intuitive. The function performs the desired comparison against all rows in the query and only returns True if the comparison is True for every member examined. Sticking with out bicycle example:

IF 1000 < ALL (SELECT Cost FROM Bicycle) PRINT 'All bicycles cost more than $1,000.00'

If you try this against the sample data inserted above, you should receive no message, since there is at least one bicycle (the Giant 500) that costs less than $1,000.00.

Negating ANY() and ALL()

SQL allows for these operators when using ANY() and ALL():

    { = | <> | != | > | >= | !> | < | <= | !< } 

 

However the first time you use the negative forms, you may be surprised at the results.  For example, if you run this query:

IF 1000 !< ANY (SELECT Cost FROM Bicycle) PRINT 'No bicycle costs less than $1,000.00'

you might be surprised to see that the message is indeed printed! If so, that is probably because you are reading the query in the ordinary English meaning of the command, "If there is no bicycle that costs less than $1,000.00, print the message." However, SQL sees it differently. You see,

    <condition> ANY()

is really short hand for:

    value <condition> first item OR value <condition> second item, etc.

With our sample data, negating that:

    1000.00 !< ANY(...)

is executed as:

    1000.00 !< 1000.01 OR 1000.00 !< 600.00

Clearly. 1000.00 > 600.00 so the expression evaluates to True and the message is printed.  You might have expected behavior similar to:

IF NOT EXISTS(SELECT * FROM Bicycle WHERE Cost < 1000.00) PRINT 'No bicycle costs less than $1,000.00'

which would result in no message, but NOT EXISTS() is not the same as !< ANY() as shown. In fact, this is a case where using the alias SOME() can make the intent clearer.  That is:

IF 1000 !< SOME (SELECT Cost FROM Bicycle) PRINT 'No bicycle costs less than $1,000.00'

more clearly indicates what we're asking for. What about negating the ALL() function? Well, similar to ANY(), ALL() is evaluated like this:

    value <condition> first item AND value <condition> second item etc. So, this query:

IF 1000.00 !< ALL (SELECT Cost FROM Bicycle) PRINT 'No bicycle costs less than $1,000.00'

is executed as:

    1000.00 !< 1000.01 AND 1000.00 !< 600.00

which evaluates to False, so the message is not printed.

Note:  We can rewrite the !< ANY() query (that didn't work) to achieve the desired result in this way:

IF NOT 1000 < ALL (SELECT Cost FROM Bicycle) PRINT 'No bicycle costs less than $1,000.00'

which, when you think about it, makes perfect sense!

Summary

ANY() and ALL() are valuable functions in SQL Server that can be used to write succint queries that reflect the business logic.  However, we need to be careful when using them with negative comparisons, since their execution differs from the normal, English reading.