Wednesday, June 22, 2011

Enable non sysadmins to run a profiler trace for a set period of time and view the results of the trace

My customer is a large corporate with a large Microsoft Development team. This team develops and suports numerous in house applications. These developers quite often need to troubleshoot problems on these apps and them having the ability to run a profiler trace greatly improves their ability to effectively troubleshoot problems. Obviously sysadmin rights on production servers for Developers is not allowed and not a good idea. Even the developers agree. With SQL 2005 and 2008 assigning sufficient permissions to achieve the above is not too difficult.

I've encapsulated everything the developers need into one stored procedure that takes 1 input paramater, run duration for the trace, with a default of 5 minutes.

The developers AD group is given login rights to a production SQL Server that they need to run the trace on. Then the ALTER TRACE Server permission is given to this group. The enables them to create a profiler trace (using sp_trace_create) and start (sp_trace_setstatus) the trace. The trace that's created by the stored procedure is set to capture RPC:Completed, SQL:BatchCompleted events as well as the CPU usage, Duration, Reads and Writes and some other information. The trace writes to a .trc file to the root of the C drive. The run duration input parameter for the sp specifies how long the trace runs for. Once the trace run time is done the trace is stopped and closed. The trace file is then imported into a table in msdb called PerfTrace. This is done using the fn_trace_gettable system function. Finally the PerfTrace table is queried and the results displayed to the user. The code is reasonably self explanatory.


USE [MSDB]
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[PerfTrace](
[TextData] [ntext] NULL,
[BinaryData] [image] NULL,
[DatabaseID] [int] NULL,
[TransactionID] [bigint] NULL,
[LineNumber] [int] NULL,
[NTUserName] [nvarchar](256) NULL,
[NTDomainName] [nvarchar](256) NULL,
[HostName] [nvarchar](256) NULL,
[ClientProcessID] [int] NULL,
[ApplicationName] [nvarchar](256) NULL,
[LoginName] [nvarchar](256) NULL,
[SPID] [int] NULL,
[Duration] [bigint] NULL,
[StartTime] [datetime] NULL,
[EndTime] [datetime] NULL,
[Reads] [bigint] NULL,
[Writes] [bigint] NULL,
[CPU] [int] NULL,
[Permissions] [bigint] NULL,
[Severity] [int] NULL,
[EventSubClass] [int] NULL,
[ObjectID] [int] NULL,
[Success] [int] NULL,
[IndexID] [int] NULL,
[IntegerData] [int] NULL,
[ServerName] [nvarchar](256) NULL,
[EventClass] [int] NULL,
[ObjectType] [int] NULL,
[NestLevel] [int] NULL,
[State] [int] NULL,
[Error] [int] NULL,
[Mode] [int] NULL,
[Handle] [int] NULL,
[ObjectName] [nvarchar](256) NULL,
[DatabaseName] [nvarchar](256) NULL,
[FileName] [nvarchar](256) NULL,
[OwnerName] [nvarchar](256) NULL,
[RoleName] [nvarchar](256) NULL,
[TargetUserName] [nvarchar](256) NULL,
[DBUserName] [nvarchar](256) NULL,
[LoginSid] [image] NULL,
[TargetLoginName] [nvarchar](256) NULL,
[TargetLoginSid] [image] NULL,
[ColumnPermissions] [int] NULL,
[LinkedServerName] [nvarchar](256) NULL,
[ProviderName] [nvarchar](256) NULL,
[MethodName] [nvarchar](256) NULL,
[RowCounts] [bigint] NULL,
[RequestID] [int] NULL,
[XactSequence] [bigint] NULL,
[EventSequence] [bigint] NULL,
[BigintData1] [bigint] NULL,
[BigintData2] [bigint] NULL,
[GUID] [uniqueidentifier] NULL,
[IntegerData2] [int] NULL,
[ObjectID2] [bigint] NULL,
[Type] [int] NULL,
[OwnerID] [int] NULL,
[ParentName] [nvarchar](256) NULL,
[IsSystem] [int] NULL,
[Offset] [int] NULL,
[SourceDatabaseID] [int] NULL,
[SqlHandle] [image] NULL,
[SessionLoginName] [nvarchar](256) NULL,
[PlanHandle] [image] NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

GO

USE [MSDB]
GO

CREATE PROC usp_startTrace5
@Runtime VARCHAR(10) = '00:05:00'
AS
BEGIN

DECLARE @RC int, @TraceID int, @on bit, @maxfilesize BIGINT, @stoptime DATETIME, @FileName nvarchar(256), @Dt VARCHAR(20), @VAL BIGINT
SET @Val = 1000
set @on = 1
SET @maxfilesize = 100
SET @Dt = CONVERT(VARCHAR(20), GETDATE(), 105)+'_'+CONVERT(VARCHAR(10), DATEPART(Hh, GETDATE()))+CONVERT(VARCHAR(10), DATEPART(mi, GETDATE()))
--SET @stoptime = DATEADD(mi, 1, GETDATE())
SET @FileName = 'C:\PerfTrace5_'+CAST(SERVERPROPERTY('MachineName') AS NVARCHAR(50))+'_'+@Dt

EXEC @rc = sp_trace_create @TraceID output, 0, @FileName, @maxfilesize, NULL
-- Set the events and data columns you need to capture.
EXEC sp_trace_setevent @TraceID, 10, 1, @On --1 = Text Data, RPC:Completed
EXEC sp_trace_setevent @TraceID, 12, 1, @On --1 = Text Data, SQL:BatchCompleted
EXEC sp_trace_setevent @TraceID, 10, 3, @On --3 = DatabaseID
EXEC sp_trace_setevent @TraceID, 12, 3, @On --3 = DatabaseID
EXEC sp_trace_setevent @TraceID, 10, 12, @On -- 12 = SPID
EXEC sp_trace_setevent @TraceID, 12, 12, @On -- 12 = SPID
EXEC sp_trace_setevent @TraceID, 10, 13, @On -- 13 = Duration
EXEC sp_trace_setevent @TraceID, 12, 13, @On -- 13 = Duration
EXEC sp_trace_setevent @TraceID, 10, 14, @On -- 14 = StartTime
EXEC sp_trace_setevent @TraceID, 12, 14, @On -- 14 = StartTime
EXEC sp_trace_setevent @TraceID, 10, 16, @On -- 16 = Reads
EXEC sp_trace_setevent @TraceID, 12, 16, @On -- 16 = Reads
EXEC sp_trace_setevent @TraceID, 10, 17, @On -- 17 = Writes
EXEC sp_trace_setevent @TraceID, 12, 17, @On -- 17 = Writes
EXEC sp_trace_setevent @TraceID, 10, 18, @On -- 18 = CPU
EXEC sp_trace_setevent @TraceID, 12, 18, @On -- 18 = CPU
EXEC sp_trace_setevent @TraceID, 10, 26, @On -- 26 = ServerName
EXEC sp_trace_setevent @TraceID, 12, 26, @On -- 26 = ServerName

-- Set any filter by using sp_trace_setfilter.
--EXEC sp_trace_setfilter @TraceID, 13, 0, 2, @Val This would filter out any queries that run for shorter than 1 second

-- Start the trace.
EXEC sp_trace_setstatus @TraceID, 1;

WAITFOR DELAY @Runtime

EXEC sp_trace_setstatus @TraceID, 0;
EXEC sp_trace_setstatus @TraceID, 2;

WAITFOR DELAY '00:00:05'

TRUNCATE TABLE msdb.dbo.PerfTrace;
declare @FN VARCHAR(100)
SET @FN = @FileName+'.trc'
--select @FN
INSERT PerfTrace Select * FROM ::fn_trace_gettable(@FN, default);

SELECT TextData, db.name as DatabaseName, Duration, Starttime, Reads, Writes, CPU, SPID
FROM [MSDB].[dbo].[PerfTrace] p INNER JOIN master.sys.databases db
on p.DatabaseID = db.database_ID
WHERE DatabaseID > 4

END

USE MASTER
GO

GRANT ALTER TRACE TO [Domain\group]

USE MSDB
GO

GRANT EXEC ON usp_startTrace5 TO [Domain\group]
GRANT INSERT, SELECT, DELETE ON PerfTrace TO [Domain\group]


Friday, June 3, 2011

Post SQL Installation Configuration Script

After installing SQL Server and applying the latest or required Service pack, there are always a number of tasks that may be completed in a particular environment. These can be quite repetative, especially in large environments with many servers being deployed.

Some of these tasks may include configuring SQL Mail, changing the default configuration for the Model database, setting up operators, etc.

Below is a list of these tasks:
Set up default maintenance plan
Configure SQL Mail
Change recovery model of Model database to simple
Change default file growth settings in Model
Set up SQL Agent notifications and DDSQLTeam operator
Change job history
Set up error logs to cycle and keep 31 logs

Most of these can be accomplished using T-SQL or powershell. The only one I didn't take the time to script was the default maintenance plan.

The rest of the configuration options are in the script below:


-- Create a Database Mail profile
EXECUTE msdb.dbo.sysmail_add_profile_sp
@profile_name = 'DEVS SQL Mail',
@description = 'DEVS SQL Mail' ;

-- Create a Database Mail account
EXECUTE msdb.dbo.sysmail_add_account_sp
@account_name = 'DEVS',
@description = 'DEVS',
@email_address = 'DEVS@za.com',
--@replyto_address = 'john.doe@domain_name.com',
@display_name = 'DEVS SQL Mail',
@mailserver_name = '10.10.10.1' ;

-- Add the account to the profile
EXECUTE msdb.dbo.sysmail_add_profileaccount_sp
@profile_name = 'DEVS SQL Mail',
@account_name = 'DEVS',
@sequence_number =1 ;

-- Grant access to the profile to the DBMailUsers role
EXECUTE msdb.dbo.sysmail_add_principalprofile_sp
@profile_name = 'DEVS SQL Mail',
@principal_id = 0,
@is_default = 1 ;

SELECT * FROM msdb.dbo.sysmail_profile
SELECT * FROM msdb.dbo.sysmail_account


-- Alter Model db recovery setting
ALTER DATABASE model
SET RECOVERY SIMPLE
GO

-- Alter Model file growth settings
ALTER DATABASE model
MODIFY FILE
(
Name = modeldev,
Size = 100,
FILEGROWTH = 100
)
GO

ALTER DATABASE model
MODIFY FILE
(
Name = modellog,
Size = 20,
FILEGROWTH = 20
)

-- Add SQL Team operator

USE [msdb]
GO

EXEC msdb.dbo.sp_add_operator @name=N'SQLTeam',
@enabled=1,
@weekday_pager_start_time=90000,
@weekday_pager_end_time=180000,
@saturday_pager_start_time=90000,
@saturday_pager_end_time=180000,
@sunday_pager_start_time=90000,
@sunday_pager_end_time=180000,
@pager_days=0,
@email_address=N'SQLTeam@gcn.local',
@category_name=N'[Uncategorized]'
GO

-- Change Job history
EXEC master..xp_regwrite @rootkey='HKEY_LOCAL_MACHINE', @key='SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.DEVS\SQLServerAgent', @value_name='JobHistoryMaxRows', @type='REG_DWORD', @value=100000
EXEC master..xp_regwrite @rootkey='HKEY_LOCAL_MACHINE', @key='SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.DEVS\SQLServerAgent', @value_name='JobHistoryMaxRowsPerJob', @type='REG_DWORD', @value=1000

-- Change number of error logs
EXEC master..xp_regwrite @rootkey='HKEY_LOCAL_MACHINE', @key='SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.DEVS\MSSQLServer', @value_name='NumErrorLogs', @type='REG_DWORD', @value=31

-- Add Job to cycle error logs
USE [msdb]
GO

BEGIN TRANSACTION
DECLARE @ReturnCode INT
SELECT @ReturnCode = 0
/****** Object: JobCategory [[Uncategorized (Local)]]] Script Date: 06/01/2011 13:30:22 ******/
IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'[Uncategorized (Local)]' AND category_class=1)
BEGIN
EXEC @ReturnCode = msdb.dbo.sp_add_category @class=N'JOB', @type=N'LOCAL', @name=N'[Uncategorized (Local)]'
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback

END

DECLARE @jobId BINARY(16)
EXEC @ReturnCode = msdb.dbo.sp_add_job @job_name=N'Cycle Error Logs',
@enabled=1,
@notify_level_eventlog=0,
@notify_level_email=0,
@notify_level_netsend=0,
@notify_level_page=0,
@delete_level=0,
@description=N'No description available.',
@category_name=N'[Uncategorized (Local)]',
@owner_login_name=N'sa', @job_id = @jobId OUTPUT
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
/****** Object: Step [Cycle Log] Script Date: 06/01/2011 13:30:24 ******/
EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id=@jobId, @step_name=N'Cycle Log',
@step_id=1,
@cmdexec_success_code=0,
@on_success_action=1,
@on_success_step_id=0,
@on_fail_action=2,
@on_fail_step_id=0,
@retry_attempts=0,
@retry_interval=0,
@os_run_priority=0, @subsystem=N'TSQL',
@command=N'exec sp_cycle_errorlog',
@database_name=N'master',
@flags=0
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_update_job @job_id = @jobId, @start_step_id = 1
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_add_jobschedule @job_id=@jobId, @name=N'Daily',
@enabled=1,
@freq_type=4,
@freq_interval=1,
@freq_subday_type=1,
@freq_subday_interval=0,
@freq_relative_interval=0,
@freq_recurrence_factor=0,
@active_start_date=20090420,
@active_end_date=99991231,
@active_start_time=0,
@active_end_time=235959
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_add_jobserver @job_id = @jobId, @server_name = N'(local)'
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
COMMIT TRANSACTION
GOTO EndSave
QuitWithRollback:
IF (@@TRANCOUNT > 0) ROLLBACK TRANSACTION
EndSave:

GO



--Enable SQL Agent Mail
USE [msdb]
GO
EXEC msdb.dbo.sp_set_sqlagent_properties @email_save_in_sent_folder=1
GO
EXEC master.dbo.xp_instance_regwrite N'HKEY_LOCAL_MACHINE', N'SOFTWARE\Microsoft\MSSQLServer\SQLServerAgent', N'UseDatabaseMail', N'REG_DWORD', 1
GO
EXEC master.dbo.xp_instance_regwrite N'HKEY_LOCAL_MACHINE', N'SOFTWARE\Microsoft\MSSQLServer\SQLServerAgent', N'DatabaseMailProfile', N'REG_SZ', N'DEVS SQL Mail'
GO


Just a bit about each option.
Configure SQL Mail is straightforward. Once you've done that you need to enable SQL Agent to use the SQL Mail profile you've created.

Changing the Model database growth and recovery model settings is quite important in a large busy environment. The default growth out of the box for the data file is 1MB, starting at 1MB. This is NOT ideal and if a database is created without changing this setting, the growth will cause severe fragmentation of the disk and of the mdf. In my particular environment there are 1600 databases. The policy is to have all db's use the SIMPLE recovery model. This means that if a database is created and the default setting is not changed, the recovery will be FULL. This new database log file will grow very big and will not be backed up.

The SQL Agent Operator is also quite obvious. Each SQL Server Agent has jobs that will inevitably fail sometime or another. Notifications need to be sent to a group of DBA's. This is a standard distribution list.

Changing the Job history is not so obvious. The default setting here is 1000 entries in the job history table with a max of 100 per job. This will be overrun very quickly on a busy server with many jobs that run frequently. If you want to see a jobs history, it may have been overwritten. Hence this change. Its two registry keys that are changed using xp_regwrite.

Changing the number of error logs to keep is quite handy when troubleshooting. I like to keep each days worth of SQL error logs in 1 log file. I then keep 31 error logs, so I have a months worth of error logs. I set up a job that cycles the error log at midnight. Neat, tidy, efficient. There is nothing worse than trying to open a huge SQL error log to try and find an error that may have happened 2 weeks ago.

Saturday, May 29, 2010

Automatically restore a list of databases

I recently had a situation at my client where we were troubleshooting a SQL server hanging and one candidate was a faulty RAID array or SCSI controller. One option to explore was to restore all the databases to another server and run an Index rebuild and Update statistics jobs against these restored databases. This was because these jobs caused the server to hang. So we had 20 odd databases to restore. The same requirement, that of restoring a list of databases to another server was also required by another DBA team at another client, so my quest for automation would be used by my team members at another client.

The databases in both scenario's are backed up by a maintenance plan. As such there is a backup folder with a subfolder for each database, with the sub folder name being the name of the database being restored. This is quite a common scenario. So I could copy the whole backup folder and then iterate through the sub folders and restore each database contained in each backup file. 2 minor challenges, the restore location may be, and in both cases, is different, the 2nd challenge is that each database may have more than one mdf and ldf. My script caters for both of these scenarios.

The strategy is to write a function (encapsulated single unit of work) that restores one database from one backup file. It takes the backupfile name and the database name as input parameters. I've hard coded the restore location in the function, but this could be passed in as a 3rd parameter.

Once I have this function working, I can iterate through the folder structure and for each sub directory, grab the folder name, which would be the database name, and restore the database in the backup file in that sub folder.

Herewith the code:




FUNCTION Restore-Db ($BackupFile, $DbName)
{
$RestP = "K:\MSSQL\MSSQL10.MSSQLSERVER\MSSQL\DATA"
$moveStr = ""
$q = "RESTORE FILELISTONLY FROM DISK = "+ $BackupFile
$b = Invoke-Sqlcmd -Query $q
$ln = $b | select logicalName, PhysicalName
foreach ($lnn in $ln) {$e = $lnn.PhysicalName.LastIndexOf("\")+1;$l =$lnn.PhysicalName.length; $ll = $l-$e; $phys = $lnn.PhysicalName.Substring($e,$ll); $moveStr = $moveStr+"MOVE '"+ $lnn.LogicalName + "' TO '"+$RestP+"\"+$phys+"', "}
$RestStr = "Restore Database "+$DbName+" FROM DISK = "+$BackupFile+" WITH "+$moveStr+ " STATS = 5"
Invoke-Sqlcmd -Query $RestStr -querytimeout 3600
}

$BackupsFiles = "F:\"
$dirs = dir $BackupsFiles | Where {$_.psIsContainer -eq $true}
$dbs = $dirs | select name, fullname

foreach ($dbx in $dbs) {$BackupFile = gci -path $dbx.fullname -include *.bak -recurse; $DbName = $dbx.name; write-host "Restoring database $DbName from file $BackupFile";$BackupFile = "'"+$BackupFile+"'"; Restore-Db $BackupFile $DbName}




One point to take note of, the code does not cater for cases where there is more than one backup file per sub folder. If this is the case then you'll have to edit the function accordingly. Another point is I've put in a -querytimeout value of 3600, or 1 hour. If a very big database restore takes longer than this, adjust this value accordingly.

Its a beautiful thing when you can fire off a script and watch a list of 30 databases being restored, automatically, while you do other stuff. :-)

Monday, May 10, 2010

Check that all databases on all servers have been backed up

At the client that I work at, there are over 1500 databases on more than 200 SQL servers, ranging from SQL 2000 to 2008. For me to have a good nights sleep, I want to know that each and every database is being and has been backed up. I could check each databases properties, I could go through 200 emails, and other such tedious methods?! Heck no, read the title of this Blog!!!

Prior to SQL 2008 and the Powershell provider it ships with (sqlps.exe) you could either write your own cmdlets or using SQLPSX from Codeplex, thanks Chad. The beauty with Powershell and SMO and .Net is that a database is an object and the lastbackupdate is a property of that object. Grab the databases collection of a SQL server and for each database get the last backup date. Using T-SQL you'd join a few system tables and write some code. Thats for 1 server, for multiple servers you could use any one of a number of methods to get this information. All this is quite a bit more tedious than the PS approach.

The high level PS approach is I have a list of SQL servers, for each server go to the databases collection and return the database name and the last backup date. That's as simple as it gets and the code is not much more complicated. Well sort of.

To automate the above I can use the SQL Agent on my DBA Management server that has SQL 2008 client tools installed. I can create a SQL Agent Job that runs a Powershell step. How cool is that?!! So in the command window I type vanilla PS code. Whoop!! Thing is, and this is what to me a while to figure out, the PS shell that SQL agent runs is the vanilla shell, not the sqlps shell with the SQL snapins. So to get around this I created a profile file that applies to the native powershell shell that is used by all users.

I create the file and add the two entries using powershell:



ni C:\Windows\system32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1 -type "file"
ac -path C:\Windows\system32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1 -value "Add-PSSnapin SqlServerCmdletSnapin100"
ac -path C:\Windows\system32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1 -value "Add-PSSnapin SqlServerProviderSnapin100"



The great thing now is that, besides from SQL Agent, I can browse SQL like a directory. What I do from the SQL Agent Job is just that. The code:



$svl = 'SQLsvr1', 'SQL-Dev01', 'SQL2k-03'
foreach ($s in $svl) {gci sqlserver:sql\$s\default\databases | select parent, name, lastbackupdate| where {$_.lastbackupdate -lt $(Get-Date).AddDays(-7)}}



Thats it! The above queries my server list ($svl) and gets the properties (gci or get-childitme) of each database in the databases collection for each server. The specific property is the LastBackupDate and where its older than 7 days ago.

Work smart, sleep easy. ;-)

Monday, March 8, 2010

Viewing SQL Agent Job History and Duration visually using floating bars

Viewing SQL Agent Job History and Duration visually using floating bars. The title only tells half the story. What this article will describe is this and then applying this view across a number of inter-related servers. The gem though, is the last bit, reverse engineering the Job schedule based on the new found view that was previously not visible. So you may see that there is a peak period between 4am and 7am where a number of jobs are running and some jobs in this period may be moved earlier or later. I can click on the job and change the schedule, all from reporting services. ;-).

The intro kinda gives you a good idea of the contents of this blog. Basically, when viewing performance and what happens when on which server an important place to look is at the SQL Agent and to see what jobs are running, what they do, how long they run for and if they could be contributing to performance issues. This is particularly relevant when multiple jobs doing different tasks are run on a number of servers.

So first we need a T-SQL Query that shows us the job history and the job duration. This is found in the msdb database in the sysjobhistory table and the sysjobs table. I could use the sp_help_jobhistory system stored procedure to return most, but not all the information that I need. The query I've written returns exactly the information that I need to get the required view in Excel. Prior to the actual query I need 2 user defined functions, these are to format the run_duration and run_date columns in the sysjobhistory tables into a DATETIME Value.

Herewith the two ufn's:



USE MSDB
GO

CREATE function ufn_Run_time (@Run_Date INT, @Run_time INT)
RETURNS VARCHAR(30)
AS
BEGIN
DECLARE @TL INT
DECLARE @T VARCHAR(30)
DECLARE @RTC VARCHAR(10)
DECLARE @DS VARCHAR(15)

SET @TL = LEN(@Run_time)
SET @RTC = CAST(@Run_time AS VARCHAR(10))

SET @DS = CAST(@Run_Date AS VARCHAR(15))
SET @DS = SUBSTRING(@DS, 1, 4)+'/'+SUBSTRING(@DS, 5, 2)+'/'+SUBSTRING(@DS, 7, 2)

IF (@TL = 5)
BEGIN
SET @T = '0'+SUBSTRING(@RTC,1,1)+':'+SUBSTRING(@RTC,2,2)+':'+SUBSTRING(@RTC,4,2)
END
ELSE
IF (@TL = 1)
BEGIN
SET @T = '00:00:0'+@RTC
END
ELSE
IF (@TL = 2)
BEGIN
SET @T = '00:00:'+@RTC
END
ELSE
IF (@TL = 3)
BEGIN
SET @T = '00:0'+SUBSTRING(@RTC,1,1)+':'+SUBSTRING(@RTC,2,2)
END
ELSE
IF (@TL = 4)
BEGIN
SET @T = '00:'+SUBSTRING(@RTC,1,2)+':'+SUBSTRING(@RTC,3,2)
END
ELSE
SET @T = SUBSTRING(@RTC,1,2)+':'+SUBSTRING(@RTC,3,2)+':'+SUBSTRING(@RTC,5,2)

SET @T = @DS + ' ' + @T
RETURN(@T)
END


--select (dbo.ufn_Run_time(20090815, 1234))





USE MSDB
GO

CREATE function ufn_Run_Duration (@Run_time INT)
RETURNS VARCHAR(30)
AS
BEGIN
DECLARE @TL INT
DECLARE @T VARCHAR(30)
DECLARE @RTC VARCHAR(10)
DECLARE @DS VARCHAR(30)
SET @TL = LEN(@Run_time)

SET @RTC = CAST(@Run_time AS VARCHAR(10))


SET @DS = '19000101'
SET @DS = SUBSTRING(@DS, 1, 4)+'/'+SUBSTRING(@DS, 5, 2)+'/'+SUBSTRING(@DS, 7, 2)

IF (@TL = 5)
BEGIN
SET @T = '0'+SUBSTRING(@RTC,1,1)+':'+SUBSTRING(@RTC,2,2)+':'+SUBSTRING(@RTC,4,2)
END
ELSE
IF (@TL = 1)
BEGIN
SET @T = '00:00:0'+@RTC
END
ELSE
IF (@TL = 2)
BEGIN
SET @T = '00:00:'+@RTC
END
ELSE
IF (@TL = 3)
BEGIN
SET @T = '00:0'+SUBSTRING(@RTC,1,1)+':'+SUBSTRING(@RTC,2,2)
END
ELSE
IF (@TL = 4)
BEGIN
SET @T = '00:'+SUBSTRING(@RTC,1,2)+':'+SUBSTRING(@RTC,3,2)
END
ELSE
SET @T = SUBSTRING(@RTC,1,2)+':'+SUBSTRING(@RTC,3,2)+':'+SUBSTRING(@RTC,5,2)

--SET @T = @DS + ' ' + @T
RETURN(@T)
END


--select dbo.ufn_Run_Duration(1234)



Right, so now for the T-SQL query:


select jh.Server + ' - ' + j.Name AS ServerJob,
CONVERT(VARCHAR(30),CAST(dbo.ufn_Run_time(Run_Date, Run_Time) AS DATETIME), 120) AS StartTime,
CONVERT(VARCHAR(30), CAST(dbo.ufn_Run_time(Run_Date, Run_Time) AS DATETIME) + CAST(dbo.ufn_Run_Duration(Run_Duration) AS DATETIME), 120) AS EndTime
from sysjobhistory jh inner join sysjobs j
on jh.job_id = j.job_id
where run_date = '20100308' AND Name NOT LIKE '%Biz%' AND Name NOT LIKE 'ASP%' AND Name NOT LIKE '%Mail%'
AND Name NOT LIKE 'Update DbIo' AND Name NOT LIKE '%distribution' AND Name NOT LIKE '%Capture DB%'
AND Name NOT LIKE '%Replication%' AND Name NOT LIKE '%Mobile - Process%' AND NAME NOT LIKE 'MSCRM_CONFIG.HardDelete'
and step_name = '(Job outcome)'
--AND Run_Duration > 30



This shows me the Server and Job name and the StartTime and EndTime of the Job. This is what I require in Excel to show floating bars, across a time scale. And in SQL 2008 Reporting Services, which has Dundas chart controls built in, you can do it to! :-). So the above query is the Dataset query.

I then create a range chart with the floating bars:




The Series data is configured as the two date fields, see below image:



I drop the Server - Job field in as the category field:



If everything is set up correctly, when you preview the report you should have a view of something you could never visualize so succinctly before, all your SQL Agent jobs and their durations across a time scale. Son, its a beautiful thing.... :-).




Now for the final bit, I'm gonna write some pseudo code. ;-). Basically I want to be able to click on a Job bar and be able to change its start time. So I'll need a second report that does the work of changing the schedule. A report that changes code?! Yip, it is possible. :-). This 2nd report takes the job name as an input parameter. On the 1st report I set a click action to navigate to the second report and populates the job name parameter. The only other input needed is the new start time, in the format of hhmmss, i.e. 031500 for a start time of 03:15am. I'll post the stored procedure code below to actually affect the schedule change. The report bit you'll have to figure out. If you don't come right, I can email you the rdl. ;-).

Code to change job schedule from Reporting Services:

ALTER PROC usp_Change_JobStartTime @JobName VARCHAR(255), @NewTime VARCHAR(50)
AS
BEGIN
DECLARE @ScheduleID INT
DECLARE @ReturnString VARCHAR(500)
DECLARE @ReturnTime VARCHAR(50)
--SET @JobName = 'Toets32'
--SET @NewTime = '122920'

SET @ScheduleID = (SELECT s.schedule_id FROM sysjobschedules s INNER JOIN sysjobs j
on s.job_id = j.job_id WHERE j.Name = @JobName)

exec sp_update_schedule @schedule_id = @ScheduleID, @active_start_time = @NewTime

SET @ReturnTime = (select active_start_time from sysschedules WHERE schedule_id = @ScheduleID)
SET @ReturnString = 'Job: ' + @JobName + ' Start time has been changed to ' + (dbo.ufn_Run_Duration(@ReturnTime))
SELECT @ReturnString
END

usp_Change_JobStartTime 'Toets32', '124935'

Thursday, January 14, 2010

Rarring files, Maintenance Plans and powershell

Hi Peeps. Been a while. Happy 2010 and all that.

Right, automation, powershell and maintenance plans. All meant to make our lives easier. Throw in some WinRar (licensed of course ;-)) and maybe some multi server administration (SQL) and you have a seriously powerful mix of stuffs!

Ok, so, first the problem statement: I want to be able to backup up all my SQL databases and then subsequent to that, compress all these backups and remove rar files older than say, 2 days.

Now the solution overview: Create a maintenance plan in SQL Management Studio (SSMS). The 1st task would be a Backup database task to back up all your databases. Then the 2nd maint plan task would be a T-SQL task. This would code would start a SQL Agent job that would run a Powershell script. This PS script would trawl through the SQL backup directory and rar all the backup files. The 3rd maint plan task would also kick off a SQL Agent Job to remove rar files older than say 2 days. Groovy.

This all seems simple enough. The funky bit in the code (below) is to tell Powershell to wait for just enough time for Winrar to run to compress the current file. Thats because winrar is running outside of the PS process. I've divised a very simple strategy here. Take the size of the current file being rarred, divide it by a number that will return an integer value that equates to a wait time just long enough for the rar process to complete. This number will depend on your server hardware, so you may want to tweak it a bit. It works, beeee-yute-ifully!

Here is the PS script:

$WinRAR = 'C:\Program Files\WinRAR\Winrar.exe'
$path = 'D:\MSSQL\MSSQL\BACKUP\'
sl $path
$FullBackup = gci $path -recurse -include *.bak
foreach ($File in $FullBackup) {$FileName = $File.Fullname
if ($FileName -ne $Null) {#echo "Compressing $FileName
&$WinRar a -v50000 m "$FileName.rar" $FileName; $ss = $File.Length/2500000; write-host "Rarring $filename with wait time of $ss seconds"; start-sleep $ss}
}



The above basically gets (Get-ChildItem gci) all the backup files (*.bak) in a specified path ($path). Preceeding this it creates a Winrar object ($Winrar). Now we create a foreach loop to rar each file in our $FullBackup list of files. The funky bit is to take the File size ($File.Length) and divide it by 2500000. This gives us our wait time (start-sleep $ss).

Putting it all together:

I save the above PS script on the SQL Server. I then create a SQL Agent Job that runs this script. I don't set a schedule because it will be called by the Maintenance Plan Job. I then create my Maintenance Plan with the 1st step being my backup database task and my 2nd step being a custom T-SQL Task that executes the below T-SQL code to start a SQL Agent Job. T-SQL Code:


exec msdb..sp_start_job 'Rarfiles'


The last step is to create a 2nd SQL Agent Job that calls a PS script that deletes rar files in the SQL Backup directory that are older than, say, 2 days. This is a one liner:


gci '\\devss165\D$\MSSQL\MSSQL.1\MSSQL\Backup\' -recurse -include *.bak | where {($_.CreationTime -le $(Get-Date).AddDays(-2))} | ri -force


This 2nd SQL Agent will be called from the 3rd task in the Maintenance plan, also a custom T-SQL task, that would run a sp_start_job as well.

Now if you roll this out using a multi server maintenance plan to all your SQL servers you could just save your corporate a heap of disk space and money. :-).

Friday, June 12, 2009

Gathering Boot Events and Calculating and Reporting Uptime

Uptime! Part of the Trinity, along with Single Sign On and ..... Ok so its a big topic. Something some clever people write theses about and do PhD's about.

In our case we have a reasonably simple requirement. We need to have 99% uptime for servers. Uptime is between support hours, during the week, excluding public holidays. Great.

Lets first start with what data we will use to calculate uptime. We are using the Windows event logs, in particular the System event log and EventID's 6006, 6008 and 6009. So 1st off I go and grab these events from a servers event log. I use Log Parser. The reason I use Log Parser is I can filter on EventID easily. Herewith the PS code, to get the last 45 days worth of boot events:


function Get-BootEvents ($svr)
{
$evt = 'D:\Data\BootEvents\'+$svr+'_BootEvents.csv'
$myQuery = new-object -com MSUtil.LogQuery
$objInputFormat = New-Object -com MSUtil.LogQuery.EventLogInputFormat
$objOutputFormat = New-Object -com MSUtil.LogQuery.CSVOutputFormat
$strQuery = "Select ComputerName, TimeGenerated, EventID, EventTypeName, Message INTO "+$evt+" FROM \\"+$svr+"\System WHERE TimeWritten >= SUB( TO_LOCALTIME(SYSTEM_TIMESTAMP()), TIMESTAMP('0000-02-15', 'yyyy-MM-dd')) AND (EventID = '6006' OR EventID = '6008' OR EventID = '6009') ORDER BY TimeGenerated"
$myQuery.ExecuteBatch($strQuery, $objInputFormat, $objOutputFormat)
}


I run this for each server in a list.

$sl = gc 'D:\Powershell\servers.txt'
foreach ($svr in $sl) {write-host "Getting boot events for $svr" -foregroundcolor "Green"; Get-BootEvents $svr}


Then I copy these csv files to my DBA SQL Server and Bulk Insert them into a BootEvents Table. BootEvents Table code:


CREATE TABLE [dbo].[BootEvents](
[ComputerName] [varchar](128) NULL,
[TimeGenerated] [datetime] NULL,
[EventID] [int] NULL,
[EventTypeName] [varchar](100) NULL,
[Message] [varchar](500) NULL
)



The bulk insert function is below:



function BulkInsert-BootEvents ($fl)
{
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=DBASQL1;Database=master;Integrated Security=True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
#$SqlCmd.CommandText = "use DBA; TRUNCATE TABLE BootEvents;"
#$SqlCmd.Connection = $SqlConnection
#$SqlConnection.Open()
#$sqlCmd.ExecuteNonQuery()
#$SqlConnection.Close()
$SqlCmd.CommandText = "BULK INSERT DBA..BootEvents FROM '"+$fl+"' WITH (FIELDTERMINATOR = ',', FIRSTROW = 2, ROWTERMINATOR = '\n')"
$SqlCmd.Connection = $SqlConnection
$SqlConnection.Open()
$sqlCmd.ExecuteNonQuery()
$SqlConnection.Close()
}



I run the following to Bulk Insert each csv into my BootEvents Table:

cpi d:\data\bootevents\*bootevents.csv \\DBASQL1\d$
$l = gci '\\DBASQL1\d$\*' -include *bootevents.csv | select fullname
foreach ($fl in $l) {BulkInsert-BootEvents $fl.fullname}



Now I have all the boot events for all my servers in a SQL Table. The 1st thing I do is create some T-SQL functions to work out the number of work seconds of downtime between boot events or for a hang boot event, and also if a particular date is a public holiday. I also have a function to convert seconds to a string of hours, minutes and seconds, which is more readable.

Convert Seconds to string

--drop function ms

CREATE function ufn_ConvertSec
(@s bigint)
RETURNS VARCHAR(50)
as
BEGIN
DECLARE @st VARCHAR(50)
DECLARE @m bigint
DECLARE @h int
DECLARE @d int

BEGIN
SET @h = @s/3600
SET @s = @s - (@h*3600)
SET @m = @s/60
SET @s = @s - (@m*60)
SET @st = CAST(@h AS VARCHAR(10))+' hour(s); '+ CAST(@m AS VARCHAR(15))+' minute(s); '+ CAST(@s AS VARCHAR(5))+' second(s)'
END
RETURN @st
END

--select dbo.ufn_ConvertSec(29098)



Check if Date is Public Holiday


CREATE FUNCTION ufn_IsHoliday (@Dt DATETIME)
RETURNS INT

AS

BEGIN
DECLARE @IsHol INT
DECLARE @PublicHolidays TABLE
(
DATE DATETIME,
HolidayName VARCHAR(50)
)
INSERT @PublicHolidays
VALUES('2009/04/10', 'Good Friday')
INSERT @PublicHolidays
VALUES('2009/04/13', 'Easter Monday')
INSERT @PublicHolidays
VALUES('2009/04/22', 'Voting Day')
INSERT @PublicHolidays
VALUES('2009/04/27', 'Freedom Day')
INSERT @PublicHolidays
VALUES('2009/05/01', 'Workers Day')
INSERT @PublicHolidays
VALUES('2009/06/16', 'Youth Day')
INSERT @PublicHolidays
VALUES('2009/08/10', 'Womens Day')
INSERT @PublicHolidays
VALUES('2009/09/24', 'Heritage Day')
INSERT @PublicHolidays
VALUES('2009/12/16', 'Day of Reconciliation')
INSERT @PublicHolidays
VALUES('2009/12/25', 'Christmas Day')

--DECLARE @Dt DATETIME
--SET @Dt = '2009/06/17 07:17:16'
--SELECT @Dt
SET @DT = SUBSTRING(CONVERT(VARCHAR(30), @Dt, 111), 1, 10) + ' 00:00:00.000'
--SELECT @Dt

IF (SELECT COUNT(*) FROM @PublicHolidays WHERE DATE = @DT) > 0
SET @IsHol = 1
ELSE
SET @IsHol = 0
RETURN(@IsHol)
END

select dbo.ufn_IsHoliday('2009/06/16 07:17:16')



Calculate Work Seconds between two dates

CREATE FUNCTION ufn_WorkSec (@Down datetime, @Up DATETIME)
RETURNS int
AS
BEGIN

DECLARE @WorkSec INT
SET @WorkSec = 0

WHILE @Down <= @Up
BEGIN
IF dbo.ufn_IsHoliday(@Down)=1
BEGIN
GOTO TICK
END
ELSE
IF (DATEPART(weekday, @Down) < 6 AND (DATEPART(hour, @Down) BETWEEN 7 AND 17))
BEGIN
WHILE (DATEPART(hour, @Down) < 18 AND @Down <= @Up)
BEGIN
SET @WorkSec = @WorkSec + 1
SET @Down = DATEADD(second, 1, @Down)
END
END

TICK:
SET @Down = DATEADD(day, 1, SUBSTRING(CONVERT(VARCHAR(30), @Down, 111), 1, 10) + ' 06:59:59.000')
SET @Down = DATEADD(second, 1, @Down)
END

RETURN(@WorkSec)
END



Now I'm ready to calculate the uptime percentage and insert the summarized data into a summary table called UptimeSummary. Code for table:


CREATE TABLE UptimeSummary
(
ComputerName VARCHAR(128),
Downtime VARCHAR(100),
UptimeSince DATETIME,
UptimePercent Decimal(18,4)
)


Now comes the T-SQL magic to use all the above code and churn out the uptime percentage:


declare @bootevents TABLE
(ID INT IDENTITY(1,1),
ComputerName VARCHAR(128),
TimeGenerated DATETIME,
EventID INT,
EventTypeName VARCHAR(100),
Message VARCHAR(500)
)
DECLARE @UptimeSummary TABLE
(
ComputerName VARCHAR(128),
DowntimeSec INT,
WorkSec INT,
UptimePercent Decimal(18,4)
)

INSERT @bootevents
select * from bootevents

DECLARE @StartDate DATETIME
DECLARE @EndDate DATETIME
DECLARE @WorkSec INT
DECLARE @Downtime INT
DECLARE @Up DATETIME
DECLARE @Down DATETIME
DECLARE @EventID INT

SET @StartDate = GETDATE()-45
SET @EndDate = GETDATE()
SET DATEFIRST 1
SET @WorkSec = (SELECT dbo.ufn_WorkSec(@StartDate, @EndDate))
DECLARE @Server VARCHAR(128)
DECLARE ServerC CURSOR
FOR SELECT DISTINCT ComputerName FROM @BootEvents ORDER BY ComputerName ASC
OPEN ServerC
FETCH NEXT FROM ServerC INTO @Server
WHILE @@FETCH_STATUS = 0
BEGIN

DECLARE @ComputerName VARCHAR(128)
DECLARE @ID INT
DECLARE @TimeGenerated DATETIME
DECLARE DownUp Cursor
FOR SELECT ID, TimeGenerated, ComputerName FROM @BootEvents WHERE EventID = 6006 AND ComputerName = @Server ORDER BY TimeGenerated ASC
OPEN DownUp
FETCH NEXT FROM DownUp INTO @ID, @TimeGenerated, @ComputerName
WHILE @@FETCH_STATUS = 0
BEGIN
SET @Downtime = 0
SET @Down = (SELECT TimeGenerated FROM @BootEvents WHERE ID = @ID)
SET @ID = @ID + 1
SET @Up = (SELECT TimeGenerated FROM @BootEvents WHERE ID = @ID AND EventID = 6009)
SET @Downtime = @Downtime + (SELECT dbo.ufn_WorkSec(@Down, @Up))
FETCH NEXT FROM DownUp INTO @ID, @TimeGenerated, @ComputerName
END
CLOSE DownUp
DEALLOCATE DownUp

DECLARE @Message VARCHAR(500)
DECLARE DownUpHang Cursor
FOR SELECT ID, TimeGenerated, ComputerName, Message FROM @BootEvents WHERE EventID = 6008 AND ComputerName = @Server ORDER BY TimeGenerated ASC
OPEN DownUpHang
FETCH NEXT FROM DownUpHang INTO @ID, @TimeGenerated, @ComputerName, @Message
WHILE @@FETCH_STATUS = 0
BEGIN
SET @Up = (SELECT TimeGenerated FROM @BootEvents WHERE ID = @ID)
SET @Down = (SELECT dbo.ufn_DateUnexpected([message]) FROM @BootEvents WHERE ID = @ID)
SET @Downtime = @Downtime + (SELECT dbo.ufn_WorkSec(@Down, @Up))
--SELECT @ID2, @TimeGenerated2, @ComputerName2, @Message, @Up, @Down, @Downtime
FETCH NEXT FROM DownUpHang INTO @ID, @TimeGenerated, @ComputerName, @Message
END
CLOSE DownUpHang
DEALLOCATE DownUpHang

INSERT @UptimeSummary
SELECT @ComputerName, @Downtime AS DowntimeSec, @WorkSec AS WorkSec,
CAST((@WorkSec-@Downtime) AS Decimal(18,4))/CAST(@WorkSec AS Decimal(18,4))*100 AS UptimePercent

FETCH NEXT FROM ServerC INTO @Server
END
CLOSE ServerC
DEALLOCATE ServerC

TRUNCATE TABLE UptimeSummary

INSERT UptimeSummary
SELECT ComputerName, dbo.ufn_ConvertSec(DowntimeSec) AS Downtime, @StartDate AS UptimeSince, UptimePercent
FROM @UptimeSummary
ORDER BY UptimePercent ASC


Now that I have Summarized uptime data for each server in a SQL Table I can write some Reporting Services reports to show this data. I've written two reports. One that shows the contents of the UptimeSummary table and one that shows Boot Events for a particular server. From the Summary report you can drill through to the Boot Events report for a server by clicking on the server name.

All the above, with the inclusion of "Hangtime" in the downtime calculation, prompted us to monitor for currently hanging servers. So we don't only reactively report on servers that hung, but now proactively report on servers that are currently hanging. I'll cover this in a subsequent post.

One Excon to another: "I thought you was hung?" "I is!!"