Tools to Analyzing Perfmon logs

               Now that we talked about perfmon counters for SharePoint and project server in the last post, the other challenging task after collecting these counters was to analyze this data and make out some sense out of those huge log file.  There may be different ways analyze them by applying knowledge gained experience of such incidents, but not all the time we can rely on that when the counters are in large in number. The standard Perfmon interface tends to be messy when many counters are involved.

I have two recommendations:

1.      Manual PerfMon Analysis Workbook   — When you want a tool that would give you quick overview of the server without having to delve through a thick report or through a cluttered perfmon diagram with lots of counters. 

2.      PAL  –  Performance Analysis of Logs (PAL) Tool  – A proper tool which prepares a beautiful report and graphs based on the counters and it also needs a threshold file for the counter/application we need detailed analysis.

PAL – Performance Analysis of Logs (PAL) Tool  – My favorite

This tool is all you need to complete your analysis. While you are looking to analyze SharePoint Counters and see how your server fares to the best practice results or a standard expected results of a well performing one, you need to feed the threshold file for SharePoint and the dependent threshold files as well, like for SharePoint server you may need a system overview, IIS, SQL thresholds as well as the counters we normally monitor for SharePoint server will also include IIS, process and system hardware related.

You can get a clue by looking at the sample report which this tool prepares at this URL

Manual PerfMon Analysis Workbook – You get detailed and latest update by visiting codeplex link provided above.

This tool  gives you an overview over lots of counters that are collected using perfmon. This release is workable. Too upload the data to the workbook: read the manual below:
Manual Perfmon Analysis Workbook.docx

Advertisement

Perfmon Counter Templates – SharePoint Server, Project Server and SQL Server

               When we run into issues and need further investigation to troubleshoot the root cause we look up to getting Servers and Database statistics. Always I look forward to gather as much data from Performance Counters. These counters are specific to application and Operating system and as a SharePoint Admins we need to gather data which makes sense and the following counters had been very helpful for up in such instances.

Every Situation is different and we need to add counters which are necessary and apt for that situation and below are one which I use most of the time for SharePoint and Project Server and while working with Microsoft we got these templates to collect the counters. I also uploaded these templates which you can download at:

SharePoint Server and Project Server Counters:

Processor(*)% Processor Time
Network Interface(*)Bytes Total/sec
Network Interface(*)Current Bandwidth
SystemProcessor Queue Length
Processor(*)% Privileged Time
PhysicalDisk(*)Avg. Disk sec/Read
PhysicalDisk(*)Avg. Disk sec/Write
LogicalDisk(*)Avg. Disk sec/Read
LogicalDisk(*)Avg. Disk sec/Write
MemoryFree System Page Table Entries
MemoryPool Nonpaged Bytes
MemoryPool Paged Bytes
MemoryAvailable MBytes
MemoryPages/sec
Network Interface(*)Output Queue Length
Processor(_Total)% Processor Time
Processor(_Total)% Privileged Time
SystemContext Switches/sec
Process(*)Private Bytes
Process(*)Handle Count
Process(*)Thread Count
Process(*)% Processor Time
Process(*)Virtual Bytes
LogicalDisk(C:)Free Megabytes
Processor(*)% Interrupt Time
Process(*)Working Set
MemorySystem Cache Resident Bytes
Process(*)IO Data Operations/sec
Process(*)IO Other Operations/sec
LogicalDisk(*)Disk Transfers/sec
MemoryPages Input/sec
Paging File(*)% Usage
Paging File(*)% Usage Peak
ASP.NET Apps v2.0.50727(*)Request Execution Time
ASP.NET Apps v2.0.50727(*)Requests/Sec
ASP.NET Apps v2.0.50727(*)Requests In Application Queue
ASP.NET(*)Worker Process Restarts
SQLServer:Locks(*)Average Wait Time (ms)
SQLServer:Locks(*)Number of Deadlocks/sec
SQLServer:Access Methods(*)Full Scans/sec
SQLServer:General Statistics(*)User Connections
SQLServer:Cache Manager(*)Cache Hit Ratio
SharePoint Publishing Cache(*)Publishing cache flushes / second
SharePoint Publishing Cache(*)Publishing cache hit ratio
SharePoint Publishing Cache(*)Publishing cache misses / sec
SharePoint Foundation(*)Sql Query Executing time
SharePoint Foundation(*)Executing Sql Queries
SharePoint Foundation(*)Responded Page Requests Rate
SharePoint Foundation(*)Executing Time/Page Request
SharePoint Foundation(*)Current Page Requests
SharePoint Foundation(*)Reject Page Requests Rate
SharePoint Foundation(*)Incoming Page Requests Rate
SharePoint Foundation(*)Active Threads
.NET CLR Exceptions(*)# of Exceps Thrown / sec
Web Service(_Total)Connection Attempts/sec
.NET CLR Loading(*)Current appdomains
ASP.NET Applications(*)Request Execution Time
ASP.NET Applications(*)Requests Queued
.NET CLR Memory(*)# Bytes in all Heaps
ASP.NET Applications(*)Application Restarts
MemoryCommitted Bytes
MemoryCommit Limit
Memory% Committed Bytes In Use

SQL Server Perfmon Counters Template:

LogicalDisk(*)* Memory* Network Interface(*)* Paging File(*)* PhysicalDisk(*)* Process(*)* Processor(*)* SQLServer:Access Methods* SQLServer:Backup Device(*)* SQLServer:Broker Activation(*)* SQLServer:Broker Statistics* SQLServer:Broker/DBM Transport* SQLServer:Buffer Manager* SQLServer:Buffer Node(*)* SQLServer:Buffer Partition(*)* SQLServer:Catalog Metadata(*)* SQLServer:CLR* SQLServer:Cursor Manager by Type(*)* SQLServer:Cursor Manager Total* SQLServer:Databases(*)* SQLServer:Database Mirroring(*)* SQLServer:Exec Statistics(*)* SQLServer:General Statistics* SQLServer:Latches* SQLServer:Locks(*)* SQLServer:Memory Manager* SQLServer:Plan Cache(*)* SQLServer:Replication Agents(*)* SQLServer:Replication Dist(*)* SQLServer:Replication Logreader(*)* SQLServer:Replication Snapshot(*)* SQLServer:Replication Merge(*)* SQLServer:SQL Errors(*)* SQLServer:SQL Statistics* SQLServer:Transactions* SQLServer:User Settable(*)* SQLServer:Wait Statistics(*)* System* Thread(*)*

 

SharePoint logs and IIS logs backup/archive and Compress to a file

The data keeps on growing as the SharePoint Farm being adopted more into the company and customers. It brings more challenges in terms of keeping up with the pace of requests and data administration. We are going to address today one part of that problem which is managing SharePoint LOGS weather it is ULS, Diagnostic and Analytic logging. Also it includes IIS logs. To address this challenge I did wrote a small script to migrate these logs to a different directory and them zip these logs and move to a network storage or to another directory on the local hard drive. 

Some of the challenges to achieve this were 

·        There are IIS logs with same file names in different directories (mainly IIS logs) we need to maintain the directories structure to not miss any logs. 

·        The criteria to move these logs out is based on a number of day , like I need to move all the logs 15 or older away from the default logs folder or drive. 

·        Then zip these logs and add to same archive file as this script has to run daily and we have a single archive file on that machine and to maintain the zip name to reflect the server name to easy recognition on the network store. 

·        All this to be scripted in batch scripting and using a free compress utility (7z) command line utility to include compress in the script. 

·        Donwload 7z and install it to program files as we need to add this install path to environment variable PATH to use it in script directly 

REM Sharepoint ULS and IIS Log cleanup script (backup and compress)
REM Venu Madhav
REM 18/july/2013
REM Enter the file path for the ULS log directory path
SET log=D:LogsIIS
REM ENTER no of days old logs to be archived
SET days=15
REM Enter the location for the temporary logs backup and archive file location
SET backup=d:archive
REM sets the compress utility directory  to PATH variable
REM SET PATH=%Path%;C:Program Files7-Zip
REM copy the current logs directory structure
mkdir %backup%
echo D | xcopy %log% %backup% /t
REM copies all the *.log files matching to the criteria in root directory initially
forfiles -p "%log%" -m *.log -d %days% -c "cmd /c copy /Y @path  %backup%@file"
PUSHD %log%
FOR /F %%v IN ('dir/ad/b') DO forfiles -p "%log%%%v" -m *.log -d -%days% -c "cmd /c copy @path %backup%%%v@file"
POPD
REM creating a compress file with hostname if it already exists it appends these logs either in local directory or network location
for /F %%v in ('hostname') DO 7z a d:logs%%v.zip %backup%*
REM once the logs are compressed remove the log files meeting the criteria
forfiles -p "%log%" -m *.log -s -d -%days% -c "cmd /c del @path /F /Q"
REM removing the temporary folder after the log compressed and zipped
rmdir %backup% /S /Q