Tuesday, July 5, 2011

View Reporting Services subscriptions and run them ad hoc

In some Reporting Services environments, subscriptions are extensively used to distribute business data. This results in 10's or even 100's or subscriptions. Its quite useful to have a view of these subscriptions and the associated sql agent jobs that actually run these. The thing is these jobs have GUID's as names. Very human unfriendly. There are many reasons I'd want to see this, for example, whne last was a subscription run? One of the main reasons I'd want to see a job linked to a subscription is if I wanted to run a subscription manually. I could of course edit the schedule of the subscription, or create a once off schedule, but that is quite tedious.

I've written some code to 1st of all view Subscriptions in a friendly way. This selects from 3 tables in the ReportServer database and shows the subscriptions and the associated job name. This is the data I display in my 1st Reporting Services report, i.e. show me all subscriptions and the associated job names, and some other data.

I then have a 2nd reporting services report to actually run the SQL agent job that runs the subscription and show the results of the job execution. So I'm using a RS report to execute a SQL agent job. From the 1st report I pass the job_name value to the 2nd report as an input parameter. This parameter is used to start a sql agent job, using the system stored procedure sp_start_job. I then wait for 30 seconds and get the last execution value of that job from sysjobhistory.

Herewith the code:
Select to show subscriptions and populate 1st report:

SELECT
c.Name AS ReportName
, rs.ScheduleID AS JOB_NAME
, s.[Description]
, s.LastStatus
, s.LastRunTime
FROM
ReportServer..[Catalog] c
JOIN ReportServer..Subscriptions s ON c.ItemID = s.Report_OID
JOIN ReportServer..ReportSchedule rs ON c.ItemID = rs.ReportID
AND rs.SubscriptionID = s.SubscriptionID
order by LastRunTime desc


Code to run report subscription based on job_name:

create proc usp_sub @JobName NVarchar(255)
as

begin
exec msdb.dbo.sp_start_job @JobName
WAITFOR DELAY '00:00:30'

select top 1 message, run_date, run_time, run_duration from msdb.dbo.sysjobhistory jh
inner join msdb.dbo.sysjobs j
on jh.job_id = j.job_id
where j.name = @JobName
and step_id = 0
order by jh.instance_id desc
end


The RS reports are very basic, just displaying the results of the above queries in tables. On the main report that shows the subscriptions, I have an action defined on the Subscription name column. The action is to go to the 2nd report which runs the job, taking the job_name (GUID) as an input parameter for the above stored proc.

Wednesday, June 22, 2011

Enable non sysadmins to run a profiler trace for a set period of time and view the results of the trace

My customer is a large corporate with a large Microsoft Development team. This team develops and suports numerous in house applications. These developers quite often need to troubleshoot problems on these apps and them having the ability to run a profiler trace greatly improves their ability to effectively troubleshoot problems. Obviously sysadmin rights on production servers for Developers is not allowed and not a good idea. Even the developers agree. With SQL 2005 and 2008 assigning sufficient permissions to achieve the above is not too difficult.

I've encapsulated everything the developers need into one stored procedure that takes 1 input paramater, run duration for the trace, with a default of 5 minutes.

The developers AD group is given login rights to a production SQL Server that they need to run the trace on. Then the ALTER TRACE Server permission is given to this group. The enables them to create a profiler trace (using sp_trace_create) and start (sp_trace_setstatus) the trace. The trace that's created by the stored procedure is set to capture RPC:Completed, SQL:BatchCompleted events as well as the CPU usage, Duration, Reads and Writes and some other information. The trace writes to a .trc file to the root of the C drive. The run duration input parameter for the sp specifies how long the trace runs for. Once the trace run time is done the trace is stopped and closed. The trace file is then imported into a table in msdb called PerfTrace. This is done using the fn_trace_gettable system function. Finally the PerfTrace table is queried and the results displayed to the user. The code is reasonably self explanatory.


USE [MSDB]
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[PerfTrace](
[TextData] [ntext] NULL,
[BinaryData] [image] NULL,
[DatabaseID] [int] NULL,
[TransactionID] [bigint] NULL,
[LineNumber] [int] NULL,
[NTUserName] [nvarchar](256) NULL,
[NTDomainName] [nvarchar](256) NULL,
[HostName] [nvarchar](256) NULL,
[ClientProcessID] [int] NULL,
[ApplicationName] [nvarchar](256) NULL,
[LoginName] [nvarchar](256) NULL,
[SPID] [int] NULL,
[Duration] [bigint] NULL,
[StartTime] [datetime] NULL,
[EndTime] [datetime] NULL,
[Reads] [bigint] NULL,
[Writes] [bigint] NULL,
[CPU] [int] NULL,
[Permissions] [bigint] NULL,
[Severity] [int] NULL,
[EventSubClass] [int] NULL,
[ObjectID] [int] NULL,
[Success] [int] NULL,
[IndexID] [int] NULL,
[IntegerData] [int] NULL,
[ServerName] [nvarchar](256) NULL,
[EventClass] [int] NULL,
[ObjectType] [int] NULL,
[NestLevel] [int] NULL,
[State] [int] NULL,
[Error] [int] NULL,
[Mode] [int] NULL,
[Handle] [int] NULL,
[ObjectName] [nvarchar](256) NULL,
[DatabaseName] [nvarchar](256) NULL,
[FileName] [nvarchar](256) NULL,
[OwnerName] [nvarchar](256) NULL,
[RoleName] [nvarchar](256) NULL,
[TargetUserName] [nvarchar](256) NULL,
[DBUserName] [nvarchar](256) NULL,
[LoginSid] [image] NULL,
[TargetLoginName] [nvarchar](256) NULL,
[TargetLoginSid] [image] NULL,
[ColumnPermissions] [int] NULL,
[LinkedServerName] [nvarchar](256) NULL,
[ProviderName] [nvarchar](256) NULL,
[MethodName] [nvarchar](256) NULL,
[RowCounts] [bigint] NULL,
[RequestID] [int] NULL,
[XactSequence] [bigint] NULL,
[EventSequence] [bigint] NULL,
[BigintData1] [bigint] NULL,
[BigintData2] [bigint] NULL,
[GUID] [uniqueidentifier] NULL,
[IntegerData2] [int] NULL,
[ObjectID2] [bigint] NULL,
[Type] [int] NULL,
[OwnerID] [int] NULL,
[ParentName] [nvarchar](256) NULL,
[IsSystem] [int] NULL,
[Offset] [int] NULL,
[SourceDatabaseID] [int] NULL,
[SqlHandle] [image] NULL,
[SessionLoginName] [nvarchar](256) NULL,
[PlanHandle] [image] NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

GO

USE [MSDB]
GO

CREATE PROC usp_startTrace5
@Runtime VARCHAR(10) = '00:05:00'
AS
BEGIN

DECLARE @RC int, @TraceID int, @on bit, @maxfilesize BIGINT, @stoptime DATETIME, @FileName nvarchar(256), @Dt VARCHAR(20), @VAL BIGINT
SET @Val = 1000
set @on = 1
SET @maxfilesize = 100
SET @Dt = CONVERT(VARCHAR(20), GETDATE(), 105)+'_'+CONVERT(VARCHAR(10), DATEPART(Hh, GETDATE()))+CONVERT(VARCHAR(10), DATEPART(mi, GETDATE()))
--SET @stoptime = DATEADD(mi, 1, GETDATE())
SET @FileName = 'C:\PerfTrace5_'+CAST(SERVERPROPERTY('MachineName') AS NVARCHAR(50))+'_'+@Dt

EXEC @rc = sp_trace_create @TraceID output, 0, @FileName, @maxfilesize, NULL
-- Set the events and data columns you need to capture.
EXEC sp_trace_setevent @TraceID, 10, 1, @On --1 = Text Data, RPC:Completed
EXEC sp_trace_setevent @TraceID, 12, 1, @On --1 = Text Data, SQL:BatchCompleted
EXEC sp_trace_setevent @TraceID, 10, 3, @On --3 = DatabaseID
EXEC sp_trace_setevent @TraceID, 12, 3, @On --3 = DatabaseID
EXEC sp_trace_setevent @TraceID, 10, 12, @On -- 12 = SPID
EXEC sp_trace_setevent @TraceID, 12, 12, @On -- 12 = SPID
EXEC sp_trace_setevent @TraceID, 10, 13, @On -- 13 = Duration
EXEC sp_trace_setevent @TraceID, 12, 13, @On -- 13 = Duration
EXEC sp_trace_setevent @TraceID, 10, 14, @On -- 14 = StartTime
EXEC sp_trace_setevent @TraceID, 12, 14, @On -- 14 = StartTime
EXEC sp_trace_setevent @TraceID, 10, 16, @On -- 16 = Reads
EXEC sp_trace_setevent @TraceID, 12, 16, @On -- 16 = Reads
EXEC sp_trace_setevent @TraceID, 10, 17, @On -- 17 = Writes
EXEC sp_trace_setevent @TraceID, 12, 17, @On -- 17 = Writes
EXEC sp_trace_setevent @TraceID, 10, 18, @On -- 18 = CPU
EXEC sp_trace_setevent @TraceID, 12, 18, @On -- 18 = CPU
EXEC sp_trace_setevent @TraceID, 10, 26, @On -- 26 = ServerName
EXEC sp_trace_setevent @TraceID, 12, 26, @On -- 26 = ServerName

-- Set any filter by using sp_trace_setfilter.
--EXEC sp_trace_setfilter @TraceID, 13, 0, 2, @Val This would filter out any queries that run for shorter than 1 second

-- Start the trace.
EXEC sp_trace_setstatus @TraceID, 1;

WAITFOR DELAY @Runtime

EXEC sp_trace_setstatus @TraceID, 0;
EXEC sp_trace_setstatus @TraceID, 2;

WAITFOR DELAY '00:00:05'

TRUNCATE TABLE msdb.dbo.PerfTrace;
declare @FN VARCHAR(100)
SET @FN = @FileName+'.trc'
--select @FN
INSERT PerfTrace Select * FROM ::fn_trace_gettable(@FN, default);

SELECT TextData, db.name as DatabaseName, Duration, Starttime, Reads, Writes, CPU, SPID
FROM [MSDB].[dbo].[PerfTrace] p INNER JOIN master.sys.databases db
on p.DatabaseID = db.database_ID
WHERE DatabaseID > 4

END

USE MASTER
GO

GRANT ALTER TRACE TO [Domain\group]

USE MSDB
GO

GRANT EXEC ON usp_startTrace5 TO [Domain\group]
GRANT INSERT, SELECT, DELETE ON PerfTrace TO [Domain\group]


Friday, June 3, 2011

Post SQL Installation Configuration Script

After installing SQL Server and applying the latest or required Service pack, there are always a number of tasks that may be completed in a particular environment. These can be quite repetative, especially in large environments with many servers being deployed.

Some of these tasks may include configuring SQL Mail, changing the default configuration for the Model database, setting up operators, etc.

Below is a list of these tasks:
Set up default maintenance plan
Configure SQL Mail
Change recovery model of Model database to simple
Change default file growth settings in Model
Set up SQL Agent notifications and DDSQLTeam operator
Change job history
Set up error logs to cycle and keep 31 logs

Most of these can be accomplished using T-SQL or powershell. The only one I didn't take the time to script was the default maintenance plan.

The rest of the configuration options are in the script below:


-- Create a Database Mail profile
EXECUTE msdb.dbo.sysmail_add_profile_sp
@profile_name = 'DEVS SQL Mail',
@description = 'DEVS SQL Mail' ;

-- Create a Database Mail account
EXECUTE msdb.dbo.sysmail_add_account_sp
@account_name = 'DEVS',
@description = 'DEVS',
@email_address = 'DEVS@za.com',
--@replyto_address = 'john.doe@domain_name.com',
@display_name = 'DEVS SQL Mail',
@mailserver_name = '10.10.10.1' ;

-- Add the account to the profile
EXECUTE msdb.dbo.sysmail_add_profileaccount_sp
@profile_name = 'DEVS SQL Mail',
@account_name = 'DEVS',
@sequence_number =1 ;

-- Grant access to the profile to the DBMailUsers role
EXECUTE msdb.dbo.sysmail_add_principalprofile_sp
@profile_name = 'DEVS SQL Mail',
@principal_id = 0,
@is_default = 1 ;

SELECT * FROM msdb.dbo.sysmail_profile
SELECT * FROM msdb.dbo.sysmail_account


-- Alter Model db recovery setting
ALTER DATABASE model
SET RECOVERY SIMPLE
GO

-- Alter Model file growth settings
ALTER DATABASE model
MODIFY FILE
(
Name = modeldev,
Size = 100,
FILEGROWTH = 100
)
GO

ALTER DATABASE model
MODIFY FILE
(
Name = modellog,
Size = 20,
FILEGROWTH = 20
)

-- Add SQL Team operator

USE [msdb]
GO

EXEC msdb.dbo.sp_add_operator @name=N'SQLTeam',
@enabled=1,
@weekday_pager_start_time=90000,
@weekday_pager_end_time=180000,
@saturday_pager_start_time=90000,
@saturday_pager_end_time=180000,
@sunday_pager_start_time=90000,
@sunday_pager_end_time=180000,
@pager_days=0,
@email_address=N'SQLTeam@gcn.local',
@category_name=N'[Uncategorized]'
GO

-- Change Job history
EXEC master..xp_regwrite @rootkey='HKEY_LOCAL_MACHINE', @key='SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.DEVS\SQLServerAgent', @value_name='JobHistoryMaxRows', @type='REG_DWORD', @value=100000
EXEC master..xp_regwrite @rootkey='HKEY_LOCAL_MACHINE', @key='SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.DEVS\SQLServerAgent', @value_name='JobHistoryMaxRowsPerJob', @type='REG_DWORD', @value=1000

-- Change number of error logs
EXEC master..xp_regwrite @rootkey='HKEY_LOCAL_MACHINE', @key='SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.DEVS\MSSQLServer', @value_name='NumErrorLogs', @type='REG_DWORD', @value=31

-- Add Job to cycle error logs
USE [msdb]
GO

BEGIN TRANSACTION
DECLARE @ReturnCode INT
SELECT @ReturnCode = 0
/****** Object: JobCategory [[Uncategorized (Local)]]] Script Date: 06/01/2011 13:30:22 ******/
IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'[Uncategorized (Local)]' AND category_class=1)
BEGIN
EXEC @ReturnCode = msdb.dbo.sp_add_category @class=N'JOB', @type=N'LOCAL', @name=N'[Uncategorized (Local)]'
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback

END

DECLARE @jobId BINARY(16)
EXEC @ReturnCode = msdb.dbo.sp_add_job @job_name=N'Cycle Error Logs',
@enabled=1,
@notify_level_eventlog=0,
@notify_level_email=0,
@notify_level_netsend=0,
@notify_level_page=0,
@delete_level=0,
@description=N'No description available.',
@category_name=N'[Uncategorized (Local)]',
@owner_login_name=N'sa', @job_id = @jobId OUTPUT
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
/****** Object: Step [Cycle Log] Script Date: 06/01/2011 13:30:24 ******/
EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id=@jobId, @step_name=N'Cycle Log',
@step_id=1,
@cmdexec_success_code=0,
@on_success_action=1,
@on_success_step_id=0,
@on_fail_action=2,
@on_fail_step_id=0,
@retry_attempts=0,
@retry_interval=0,
@os_run_priority=0, @subsystem=N'TSQL',
@command=N'exec sp_cycle_errorlog',
@database_name=N'master',
@flags=0
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_update_job @job_id = @jobId, @start_step_id = 1
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_add_jobschedule @job_id=@jobId, @name=N'Daily',
@enabled=1,
@freq_type=4,
@freq_interval=1,
@freq_subday_type=1,
@freq_subday_interval=0,
@freq_relative_interval=0,
@freq_recurrence_factor=0,
@active_start_date=20090420,
@active_end_date=99991231,
@active_start_time=0,
@active_end_time=235959
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_add_jobserver @job_id = @jobId, @server_name = N'(local)'
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
COMMIT TRANSACTION
GOTO EndSave
QuitWithRollback:
IF (@@TRANCOUNT > 0) ROLLBACK TRANSACTION
EndSave:

GO



--Enable SQL Agent Mail
USE [msdb]
GO
EXEC msdb.dbo.sp_set_sqlagent_properties @email_save_in_sent_folder=1
GO
EXEC master.dbo.xp_instance_regwrite N'HKEY_LOCAL_MACHINE', N'SOFTWARE\Microsoft\MSSQLServer\SQLServerAgent', N'UseDatabaseMail', N'REG_DWORD', 1
GO
EXEC master.dbo.xp_instance_regwrite N'HKEY_LOCAL_MACHINE', N'SOFTWARE\Microsoft\MSSQLServer\SQLServerAgent', N'DatabaseMailProfile', N'REG_SZ', N'DEVS SQL Mail'
GO


Just a bit about each option.
Configure SQL Mail is straightforward. Once you've done that you need to enable SQL Agent to use the SQL Mail profile you've created.

Changing the Model database growth and recovery model settings is quite important in a large busy environment. The default growth out of the box for the data file is 1MB, starting at 1MB. This is NOT ideal and if a database is created without changing this setting, the growth will cause severe fragmentation of the disk and of the mdf. In my particular environment there are 1600 databases. The policy is to have all db's use the SIMPLE recovery model. This means that if a database is created and the default setting is not changed, the recovery will be FULL. This new database log file will grow very big and will not be backed up.

The SQL Agent Operator is also quite obvious. Each SQL Server Agent has jobs that will inevitably fail sometime or another. Notifications need to be sent to a group of DBA's. This is a standard distribution list.

Changing the Job history is not so obvious. The default setting here is 1000 entries in the job history table with a max of 100 per job. This will be overrun very quickly on a busy server with many jobs that run frequently. If you want to see a jobs history, it may have been overwritten. Hence this change. Its two registry keys that are changed using xp_regwrite.

Changing the number of error logs to keep is quite handy when troubleshooting. I like to keep each days worth of SQL error logs in 1 log file. I then keep 31 error logs, so I have a months worth of error logs. I set up a job that cycles the error log at midnight. Neat, tidy, efficient. There is nothing worse than trying to open a huge SQL error log to try and find an error that may have happened 2 weeks ago.

Saturday, May 29, 2010

Automatically restore a list of databases

I recently had a situation at my client where we were troubleshooting a SQL server hanging and one candidate was a faulty RAID array or SCSI controller. One option to explore was to restore all the databases to another server and run an Index rebuild and Update statistics jobs against these restored databases. This was because these jobs caused the server to hang. So we had 20 odd databases to restore. The same requirement, that of restoring a list of databases to another server was also required by another DBA team at another client, so my quest for automation would be used by my team members at another client.

The databases in both scenario's are backed up by a maintenance plan. As such there is a backup folder with a subfolder for each database, with the sub folder name being the name of the database being restored. This is quite a common scenario. So I could copy the whole backup folder and then iterate through the sub folders and restore each database contained in each backup file. 2 minor challenges, the restore location may be, and in both cases, is different, the 2nd challenge is that each database may have more than one mdf and ldf. My script caters for both of these scenarios.

The strategy is to write a function (encapsulated single unit of work) that restores one database from one backup file. It takes the backupfile name and the database name as input parameters. I've hard coded the restore location in the function, but this could be passed in as a 3rd parameter.

Once I have this function working, I can iterate through the folder structure and for each sub directory, grab the folder name, which would be the database name, and restore the database in the backup file in that sub folder.

Herewith the code:




FUNCTION Restore-Db ($BackupFile, $DbName)
{
$RestP = "K:\MSSQL\MSSQL10.MSSQLSERVER\MSSQL\DATA"
$moveStr = ""
$q = "RESTORE FILELISTONLY FROM DISK = "+ $BackupFile
$b = Invoke-Sqlcmd -Query $q
$ln = $b | select logicalName, PhysicalName
foreach ($lnn in $ln) {$e = $lnn.PhysicalName.LastIndexOf("\")+1;$l =$lnn.PhysicalName.length; $ll = $l-$e; $phys = $lnn.PhysicalName.Substring($e,$ll); $moveStr = $moveStr+"MOVE '"+ $lnn.LogicalName + "' TO '"+$RestP+"\"+$phys+"', "}
$RestStr = "Restore Database "+$DbName+" FROM DISK = "+$BackupFile+" WITH "+$moveStr+ " STATS = 5"
Invoke-Sqlcmd -Query $RestStr -querytimeout 3600
}

$BackupsFiles = "F:\"
$dirs = dir $BackupsFiles | Where {$_.psIsContainer -eq $true}
$dbs = $dirs | select name, fullname

foreach ($dbx in $dbs) {$BackupFile = gci -path $dbx.fullname -include *.bak -recurse; $DbName = $dbx.name; write-host "Restoring database $DbName from file $BackupFile";$BackupFile = "'"+$BackupFile+"'"; Restore-Db $BackupFile $DbName}




One point to take note of, the code does not cater for cases where there is more than one backup file per sub folder. If this is the case then you'll have to edit the function accordingly. Another point is I've put in a -querytimeout value of 3600, or 1 hour. If a very big database restore takes longer than this, adjust this value accordingly.

Its a beautiful thing when you can fire off a script and watch a list of 30 databases being restored, automatically, while you do other stuff. :-)

Monday, May 10, 2010

Check that all databases on all servers have been backed up

At the client that I work at, there are over 1500 databases on more than 200 SQL servers, ranging from SQL 2000 to 2008. For me to have a good nights sleep, I want to know that each and every database is being and has been backed up. I could check each databases properties, I could go through 200 emails, and other such tedious methods?! Heck no, read the title of this Blog!!!

Prior to SQL 2008 and the Powershell provider it ships with (sqlps.exe) you could either write your own cmdlets or using SQLPSX from Codeplex, thanks Chad. The beauty with Powershell and SMO and .Net is that a database is an object and the lastbackupdate is a property of that object. Grab the databases collection of a SQL server and for each database get the last backup date. Using T-SQL you'd join a few system tables and write some code. Thats for 1 server, for multiple servers you could use any one of a number of methods to get this information. All this is quite a bit more tedious than the PS approach.

The high level PS approach is I have a list of SQL servers, for each server go to the databases collection and return the database name and the last backup date. That's as simple as it gets and the code is not much more complicated. Well sort of.

To automate the above I can use the SQL Agent on my DBA Management server that has SQL 2008 client tools installed. I can create a SQL Agent Job that runs a Powershell step. How cool is that?!! So in the command window I type vanilla PS code. Whoop!! Thing is, and this is what to me a while to figure out, the PS shell that SQL agent runs is the vanilla shell, not the sqlps shell with the SQL snapins. So to get around this I created a profile file that applies to the native powershell shell that is used by all users.

I create the file and add the two entries using powershell:



ni C:\Windows\system32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1 -type "file"
ac -path C:\Windows\system32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1 -value "Add-PSSnapin SqlServerCmdletSnapin100"
ac -path C:\Windows\system32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1 -value "Add-PSSnapin SqlServerProviderSnapin100"



The great thing now is that, besides from SQL Agent, I can browse SQL like a directory. What I do from the SQL Agent Job is just that. The code:



$svl = 'SQLsvr1', 'SQL-Dev01', 'SQL2k-03'
foreach ($s in $svl) {gci sqlserver:sql\$s\default\databases | select parent, name, lastbackupdate| where {$_.lastbackupdate -lt $(Get-Date).AddDays(-7)}}



Thats it! The above queries my server list ($svl) and gets the properties (gci or get-childitme) of each database in the databases collection for each server. The specific property is the LastBackupDate and where its older than 7 days ago.

Work smart, sleep easy. ;-)

Monday, March 8, 2010

Viewing SQL Agent Job History and Duration visually using floating bars

Viewing SQL Agent Job History and Duration visually using floating bars. The title only tells half the story. What this article will describe is this and then applying this view across a number of inter-related servers. The gem though, is the last bit, reverse engineering the Job schedule based on the new found view that was previously not visible. So you may see that there is a peak period between 4am and 7am where a number of jobs are running and some jobs in this period may be moved earlier or later. I can click on the job and change the schedule, all from reporting services. ;-).

The intro kinda gives you a good idea of the contents of this blog. Basically, when viewing performance and what happens when on which server an important place to look is at the SQL Agent and to see what jobs are running, what they do, how long they run for and if they could be contributing to performance issues. This is particularly relevant when multiple jobs doing different tasks are run on a number of servers.

So first we need a T-SQL Query that shows us the job history and the job duration. This is found in the msdb database in the sysjobhistory table and the sysjobs table. I could use the sp_help_jobhistory system stored procedure to return most, but not all the information that I need. The query I've written returns exactly the information that I need to get the required view in Excel. Prior to the actual query I need 2 user defined functions, these are to format the run_duration and run_date columns in the sysjobhistory tables into a DATETIME Value.

Herewith the two ufn's:



USE MSDB
GO

CREATE function ufn_Run_time (@Run_Date INT, @Run_time INT)
RETURNS VARCHAR(30)
AS
BEGIN
DECLARE @TL INT
DECLARE @T VARCHAR(30)
DECLARE @RTC VARCHAR(10)
DECLARE @DS VARCHAR(15)

SET @TL = LEN(@Run_time)
SET @RTC = CAST(@Run_time AS VARCHAR(10))

SET @DS = CAST(@Run_Date AS VARCHAR(15))
SET @DS = SUBSTRING(@DS, 1, 4)+'/'+SUBSTRING(@DS, 5, 2)+'/'+SUBSTRING(@DS, 7, 2)

IF (@TL = 5)
BEGIN
SET @T = '0'+SUBSTRING(@RTC,1,1)+':'+SUBSTRING(@RTC,2,2)+':'+SUBSTRING(@RTC,4,2)
END
ELSE
IF (@TL = 1)
BEGIN
SET @T = '00:00:0'+@RTC
END
ELSE
IF (@TL = 2)
BEGIN
SET @T = '00:00:'+@RTC
END
ELSE
IF (@TL = 3)
BEGIN
SET @T = '00:0'+SUBSTRING(@RTC,1,1)+':'+SUBSTRING(@RTC,2,2)
END
ELSE
IF (@TL = 4)
BEGIN
SET @T = '00:'+SUBSTRING(@RTC,1,2)+':'+SUBSTRING(@RTC,3,2)
END
ELSE
SET @T = SUBSTRING(@RTC,1,2)+':'+SUBSTRING(@RTC,3,2)+':'+SUBSTRING(@RTC,5,2)

SET @T = @DS + ' ' + @T
RETURN(@T)
END


--select (dbo.ufn_Run_time(20090815, 1234))





USE MSDB
GO

CREATE function ufn_Run_Duration (@Run_time INT)
RETURNS VARCHAR(30)
AS
BEGIN
DECLARE @TL INT
DECLARE @T VARCHAR(30)
DECLARE @RTC VARCHAR(10)
DECLARE @DS VARCHAR(30)
SET @TL = LEN(@Run_time)

SET @RTC = CAST(@Run_time AS VARCHAR(10))


SET @DS = '19000101'
SET @DS = SUBSTRING(@DS, 1, 4)+'/'+SUBSTRING(@DS, 5, 2)+'/'+SUBSTRING(@DS, 7, 2)

IF (@TL = 5)
BEGIN
SET @T = '0'+SUBSTRING(@RTC,1,1)+':'+SUBSTRING(@RTC,2,2)+':'+SUBSTRING(@RTC,4,2)
END
ELSE
IF (@TL = 1)
BEGIN
SET @T = '00:00:0'+@RTC
END
ELSE
IF (@TL = 2)
BEGIN
SET @T = '00:00:'+@RTC
END
ELSE
IF (@TL = 3)
BEGIN
SET @T = '00:0'+SUBSTRING(@RTC,1,1)+':'+SUBSTRING(@RTC,2,2)
END
ELSE
IF (@TL = 4)
BEGIN
SET @T = '00:'+SUBSTRING(@RTC,1,2)+':'+SUBSTRING(@RTC,3,2)
END
ELSE
SET @T = SUBSTRING(@RTC,1,2)+':'+SUBSTRING(@RTC,3,2)+':'+SUBSTRING(@RTC,5,2)

--SET @T = @DS + ' ' + @T
RETURN(@T)
END


--select dbo.ufn_Run_Duration(1234)



Right, so now for the T-SQL query:


select jh.Server + ' - ' + j.Name AS ServerJob,
CONVERT(VARCHAR(30),CAST(dbo.ufn_Run_time(Run_Date, Run_Time) AS DATETIME), 120) AS StartTime,
CONVERT(VARCHAR(30), CAST(dbo.ufn_Run_time(Run_Date, Run_Time) AS DATETIME) + CAST(dbo.ufn_Run_Duration(Run_Duration) AS DATETIME), 120) AS EndTime
from sysjobhistory jh inner join sysjobs j
on jh.job_id = j.job_id
where run_date = '20100308' AND Name NOT LIKE '%Biz%' AND Name NOT LIKE 'ASP%' AND Name NOT LIKE '%Mail%'
AND Name NOT LIKE 'Update DbIo' AND Name NOT LIKE '%distribution' AND Name NOT LIKE '%Capture DB%'
AND Name NOT LIKE '%Replication%' AND Name NOT LIKE '%Mobile - Process%' AND NAME NOT LIKE 'MSCRM_CONFIG.HardDelete'
and step_name = '(Job outcome)'
--AND Run_Duration > 30



This shows me the Server and Job name and the StartTime and EndTime of the Job. This is what I require in Excel to show floating bars, across a time scale. And in SQL 2008 Reporting Services, which has Dundas chart controls built in, you can do it to! :-). So the above query is the Dataset query.

I then create a range chart with the floating bars:




The Series data is configured as the two date fields, see below image:



I drop the Server - Job field in as the category field:



If everything is set up correctly, when you preview the report you should have a view of something you could never visualize so succinctly before, all your SQL Agent jobs and their durations across a time scale. Son, its a beautiful thing.... :-).




Now for the final bit, I'm gonna write some pseudo code. ;-). Basically I want to be able to click on a Job bar and be able to change its start time. So I'll need a second report that does the work of changing the schedule. A report that changes code?! Yip, it is possible. :-). This 2nd report takes the job name as an input parameter. On the 1st report I set a click action to navigate to the second report and populates the job name parameter. The only other input needed is the new start time, in the format of hhmmss, i.e. 031500 for a start time of 03:15am. I'll post the stored procedure code below to actually affect the schedule change. The report bit you'll have to figure out. If you don't come right, I can email you the rdl. ;-).

Code to change job schedule from Reporting Services:

ALTER PROC usp_Change_JobStartTime @JobName VARCHAR(255), @NewTime VARCHAR(50)
AS
BEGIN
DECLARE @ScheduleID INT
DECLARE @ReturnString VARCHAR(500)
DECLARE @ReturnTime VARCHAR(50)
--SET @JobName = 'Toets32'
--SET @NewTime = '122920'

SET @ScheduleID = (SELECT s.schedule_id FROM sysjobschedules s INNER JOIN sysjobs j
on s.job_id = j.job_id WHERE j.Name = @JobName)

exec sp_update_schedule @schedule_id = @ScheduleID, @active_start_time = @NewTime

SET @ReturnTime = (select active_start_time from sysschedules WHERE schedule_id = @ScheduleID)
SET @ReturnString = 'Job: ' + @JobName + ' Start time has been changed to ' + (dbo.ufn_Run_Duration(@ReturnTime))
SELECT @ReturnString
END

usp_Change_JobStartTime 'Toets32', '124935'

Thursday, January 14, 2010

Rarring files, Maintenance Plans and powershell

Hi Peeps. Been a while. Happy 2010 and all that.

Right, automation, powershell and maintenance plans. All meant to make our lives easier. Throw in some WinRar (licensed of course ;-)) and maybe some multi server administration (SQL) and you have a seriously powerful mix of stuffs!

Ok, so, first the problem statement: I want to be able to backup up all my SQL databases and then subsequent to that, compress all these backups and remove rar files older than say, 2 days.

Now the solution overview: Create a maintenance plan in SQL Management Studio (SSMS). The 1st task would be a Backup database task to back up all your databases. Then the 2nd maint plan task would be a T-SQL task. This would code would start a SQL Agent job that would run a Powershell script. This PS script would trawl through the SQL backup directory and rar all the backup files. The 3rd maint plan task would also kick off a SQL Agent Job to remove rar files older than say 2 days. Groovy.

This all seems simple enough. The funky bit in the code (below) is to tell Powershell to wait for just enough time for Winrar to run to compress the current file. Thats because winrar is running outside of the PS process. I've divised a very simple strategy here. Take the size of the current file being rarred, divide it by a number that will return an integer value that equates to a wait time just long enough for the rar process to complete. This number will depend on your server hardware, so you may want to tweak it a bit. It works, beeee-yute-ifully!

Here is the PS script:

$WinRAR = 'C:\Program Files\WinRAR\Winrar.exe'
$path = 'D:\MSSQL\MSSQL\BACKUP\'
sl $path
$FullBackup = gci $path -recurse -include *.bak
foreach ($File in $FullBackup) {$FileName = $File.Fullname
if ($FileName -ne $Null) {#echo "Compressing $FileName
&$WinRar a -v50000 m "$FileName.rar" $FileName; $ss = $File.Length/2500000; write-host "Rarring $filename with wait time of $ss seconds"; start-sleep $ss}
}



The above basically gets (Get-ChildItem gci) all the backup files (*.bak) in a specified path ($path). Preceeding this it creates a Winrar object ($Winrar). Now we create a foreach loop to rar each file in our $FullBackup list of files. The funky bit is to take the File size ($File.Length) and divide it by 2500000. This gives us our wait time (start-sleep $ss).

Putting it all together:

I save the above PS script on the SQL Server. I then create a SQL Agent Job that runs this script. I don't set a schedule because it will be called by the Maintenance Plan Job. I then create my Maintenance Plan with the 1st step being my backup database task and my 2nd step being a custom T-SQL Task that executes the below T-SQL code to start a SQL Agent Job. T-SQL Code:


exec msdb..sp_start_job 'Rarfiles'


The last step is to create a 2nd SQL Agent Job that calls a PS script that deletes rar files in the SQL Backup directory that are older than, say, 2 days. This is a one liner:


gci '\\devss165\D$\MSSQL\MSSQL.1\MSSQL\Backup\' -recurse -include *.bak | where {($_.CreationTime -le $(Get-Date).AddDays(-2))} | ri -force


This 2nd SQL Agent will be called from the 3rd task in the Maintenance plan, also a custom T-SQL task, that would run a sp_start_job as well.

Now if you roll this out using a multi server maintenance plan to all your SQL servers you could just save your corporate a heap of disk space and money. :-).