Comments (0)

Created

Jul 23, 2012

Monitoring events in an automated fashion

Posted by Ankush Kumar

One good way of understanding functioning of CQ is via event monitoring. Events are fired in CQ for various operations like rendition creation, replication etc. This can be monitored from:

http://<server>:<port>/system/console/slingevent

It provides following data

  • Total Processed Jobs
  • Queued Jobs
  • Failed Jobs
  • Average Processing Time
  • Average Waiting Time

With this data, one can easily calculate performance of workflow processing, rendition generation speed, page activation speed etc.

Problem

In load testing or longevity runs, one needs to monitor above mentioned URL to collect meaningful trends out of it. However, this will become a hectic process if done manually.

Solution

gathereventingdata is a perl based script which has just following pre-requisites.

  • Perl should be installed and present in default PATH
  • Curl should be installed and present in default PATH

This script will regularly poll /system/console/slingevent page and will gather the data. And finally, it will dump this data in the form of csv on stdout. (Which can be redirected to file if required)

Data on this page is present in following format

Data on Sling Eventing Page

This table represents over all statistics from the slingevent page. Similar to this, we have tables of individual queues also. For example, if you have one replication queue and you are using it, you will find a table for the same present too on slingevent page.

The script converts this tabular data into CSV format. A row is added for each available table. As you can see in the example mentioned below, row is added for both the replication agent queues and overall statistics.

Moreover, the sequence of columns in CSV is same as above table rows. Only difference is, one time stamp is added as added as first column.

To get started, one will just have to modify the script properties to point to his own server.

#Polling interval
$interval = 10;

#Host Name or IP and Port of machine running CQ instance.
$host = ":";

#Path to OSGi Console page showing Sling Event info.
$target = "/system/console/slingevent";

#Admin Credentials of CQ Instance
$id = "admin:admin@";

#1st Part of Curl Command. Specify curl parameters here.
$curl = "/usr/bin/curl -s -o $temp_file";

#2nd Part of Curl Command
$command = "$curl http://$id$host$target";

And simply start the script.

This will capture data from all available statistics tables and will dump the data as csv with a single row for each table like shown below:


2012-05-16,08:38:04,Topic Statistics: com/day/cq/replication/job/publish1,08:38:04:617 2012-May-16,08:38:04:566 2012-May-16,20,0,0,20,217 ms,764 ms 2012-05-16,08:38:04,Topic Statistics: com/day/cq/replication/job/publish,08:38:04:335 2012-May-16,08:38:04:631 2012-May-16,20,0,0,20,248 ms,920 ms 2012-05-16,08:38:14,Apache Sling Eventing: Overall Statistics,08:35:20:939 2012-May-16,08:38:12:042 2012-May-16,08:38:12:217 2012-May-16,0,0,0,76,0,0,76,202 ms,799 ms

This data can be filtered out on basis of statistic table column (3rd column in csv) using Excel filter functionality or Unix grep. Once sorted, all columns (starting 4th column) in csv will correspond to rows of the selected table on sling eventing page in the same order.

A typical usage of script is,

./gathereventingdata > event_output.log 2>&1


The entire perl script, gathereventingdata, is pasted below for use

#!/usr/bin/perl

#*************************************************************************
# ADOBE SYSTEMS INCORPORATED
# Copyright 2012 Adobe Systems Incorporated
# All Rights Reserved.
# 
# NOTICE:  Adobe permits you to use, modify, and distribute this file in 
# accordance with the terms of the Adobe license agreement accompanying it.  
# If you have received this file from a source other than Adobe, 
# then your use, modification, or distribution of it requires the prior 
# written permission of Adobe.
#*************************************************************************



# URL to hit: curl  http://admin:admin@:/system/console/slingevent > slingevent.log
# 
# Useful data to parse
# 
# Queued Jobs				0
# Active Jobs				0
# Jobs						0
# Finished Jobs				7480
# Failed Jobs				0
# Cancelled Jobs			0	 
# Processed Jobs			7480	 
# Average Processing Time	913 ms	 
# Average Waiting Time		519 ms

#date,time,table,jobs/queue,jobs/active,jobs/jobs,jobs/finished,jobs/failed,jobs/cancelled,jobs/processed,average_time/process,average_time/wait


local $longString;
local $boolean = 0;


# URL to hit: curl  http://admin:admin@:/system/console/slingevent

#Temperory File to store result of curl command
$temp_file = "/tmp/curl_result.tmp";

#Polling interval
$interval = 10;

#Host Name or IP and Port of machine running CQ instance.
$host = ":";

#Path to OSGi Console page showing Sling Event info.
$target = "/system/console/slingevent";

#Admin Credentials of CQ Instance
$id = "admin:admin@";

#1st Part of Curl Command. Specify curl parameters here.
$curl = "/usr/bin/curl -s -o $temp_file";

#2nd Part of Curl Command
$command = "$curl http://$id$host$target";



#Main function which will poll Sling Event page in pre-defined interval.
while(1)
{
# print("command = \"$command\"\n");

    $status = system($command);

    if ($status != 0) {
        print "Error fetching\n";
        #exit;
    }
    else
    {
        open (FILE, $temp_file);
        #print ("File open success");
        &process_file;
    }
    unlink($temp_file);
    sleep $interval;
    #exit;
}


#Function to get date for time stamp in logs
sub timestamp
{
   local($date);
   local($sec,$min,$hour,$mday,$mon,$year) = localtime;
   $year += 1900;
   $mon += 1;
   $date = sprintf("%04d-%02d-%02d,%02d:%02d:%02d",
                  $year, $mon, $mday, $hour, $min, $sec);
   return $date;
}

#Function to parse property name and value out of a string
sub get_property_and_value
{
	local ( $str ) = @_;
	local ( $name, $value );

	$str =~ /([^<>]*)<\/td>([^<>]*)<\/td>/;
	$name = $1;
	$value = $2;

	return ( $name, $value );
}

#Function to analyze each table from Sling Event page and create property value pairs.
sub process_table
{
	while () {
		last if (/\) {
		chomp;
		@list = split(//);
	
		foreach (@list) {
				local $str = $_;
				local $s; 
				
			if (/\/)
			{
				#Apache Sling Eventing: Overall Statistics
				$str =~ /(.*)<\/span>/;
				$s = $1;
				#Table Name in $s
				$ts = ×tamp;

				# if the table is Job Queue, I don't want to print it out
				if (/Job Queue Configuration/)
				{
					last;
				}
				else
				{
					$boolean = 1;
					$longString = "$ts,$s";
				}
			}
			
			# if encounter a 
			elsif (/
			if (/\Start Time/) {
				( $label ,$value ) = &get_property_and_value($_);
				#print "$label : $value\n";
				$longString = "$longString,$value";
			}
			
			elsif (/\Jobs/) {
				( $label ,$value ) = &get_property_and_value($_);
				#print "$label : $value\n";
				$longString = "$longString,$value";
			}
			
			elsif (/\Last/) {
				( $label ,$value ) = &get_property_and_value($_);
				#print "$label : $value\n";
				$longString = "$longString,$value";
			}
			
			elsif (/Average Processing Time/) {
				( $label ,$value ) = &get_property_and_value($_);
				#print "$label : $value\n";
				$longString = "$longString,$value";
			}
			
			elsif (/Average Waiting Time/) {
				( $label ,$value ) = &get_property_and_value($_);
				#print "$label : $value\n";
				$longString = "$longString,$value";
			}
			
			elsif (/\/)
			{
				if (length($longString) > 0)
				{
				    print "\n$longString";
				}
                                $longString="";
				$boolean=0;
				last;
			}
		}
	}
}



#Function to process the temp file obtained by running curl command. This will find HTML tables from file and pass them for processing one by one
sub process_file
{
	
	# Skip everything until you hit the first 
 which is the start of the first 
 then print out the table name
			#
Topic Statistics: com/day/cq/replication/job/publish
/ && $boolean==0) {
				
				#print "ln. $. $_ \n";
				$str =~ /(.*)<\/th><\/tr>/;
				$s = $1;
			        
				if(/Job Queue Configuration/ || /Configuration/)
				{
					last:
				}
				else
				{
				    $ts = ×tamp;
				    #Table Name in $s
				    $longString = "$ts,$s";
				}
			}
			
			# Start Time16:36:36:511 2011-Nov-28TypeTopic Round Robin

	while () {
		last if (/\
 so extract it this way

			# 
			if (/\/)
			{
				#Apache Sling Eventing: Overall Statistics
			
				local $str = $_;
				local $s; 
				
				# print "\nProcess String: $_\n";
				$str =~ /(.*)<\/span>/;
				$s = $1;
				
				$ts = ×tamp;
				#print "\nTable Name: $s\n";
				$longString = "$ts,$s";
				$boolean = 1;
				
				last foobar;
			}
		}
	}

	while () {
		&process_table;
	}
	close (FILE);
}

exit;