What is Splunk?
I think the chaps from Splunk explain that the best…

Splunk is the engine for machine data. Use Splunk to collect, index and harness the fast moving machine data generated by all your applications, servers and devices — physical, virtual and in the cloud. Search and analyze all your real-time and historical data from one place.
Splunking your machine data lets you troubleshoot problems and investigate security incidents in minutes, not hours or days. Monitor your end-to-end infrastructure to avoid service degradation or outages. Meet compliance mandates at lower cost. Correlate and analyze complex events spanning multiple systems. Gain new levels of operational visibility and intelligence for IT and the Business.

Cool… but then why is this nice?
Traditional support organizations supporting large web farms often have monitoring in place and then when something goes wrong they go to the server and search the logs until they have the reason for the failure. At the bare minimum, which is where I am at, Splunk aggregates all the logs from your entire web farm and presents is as one complete picture. From this I can see trends and machines with difficulties (problematic to spot when you have alot of servers). I am sure Splunk does a lot more, but I am not yet there. When I have discovered them, I will share them but so far I am a happy camper.

Adding SDL Tridion (of course) as a data source
Splunk takes multiple sources of log data and the usual list of candidates are web server (in my case, IIS) logs and Windows Event Logs. I want my Content Delivery logs integrated and so I set about doing this.

For this I will leverage SDL Tridion 2011’s updated logging (LOGBack) and with this I am able to configure logging to output logs to Syslog. Syslog is a standard way of logging application messages that separates the software application and the system logging the messages. In essence, the messages can be pushed out much like a network broadcast and a logging system, Splunk, captures that.
SO I want to configure the logging in SDL Tridion Content Delivery to push the messages to Syslog. For this we need two things: 1) an Appender and 2) a reference to that appender for my log. The Appender configures my log output and looks like:

<appender name="SYSLOG">
     <syslogHost>localhost</syslogHost>
     <facility>LOCAL0</facility>
     <suffixPattern>[%thread] %-5level %logger %msg</suffixPattern>
</appender>

This is the example from the LOGBack help documentation (http://logback.qos.ch/manual/appenders.html) with the addition of the logging level (“%-5level” which captures “ERROR”, “INFO” etc.). Then I can set a given log to push it’s messages to my Syslog appender:

<logger name="com.tridion.deployer">
     <appender-ref ref="rollingDeployerLog"/>
     <appender-ref ref="SYSLOG"/>
</logger>

I’ve kept in the original logging (“rollingDeployerLog”) just for testing and validation.

In Splunk I need to add a data source to listen to this output. So I add a new local Syslog data source specifying the (standard) port. I don’t need to specify a hostname because I did that in my logging configuration on Tridion. I told it to push the messages to “localhost” but that could be any server.

After that, once Tridion logs messages, Splunk will pick them up and store them.

So in November we’ll see the first of these sessions which we hope to repeat over the course of 2011/2012.

So what is the idea? The idea is that anyone can join in and if desired present on a community organized and driven event. The sessions, covering 10-15 minutes, can cover any topic related to a SDL Tridion (e.g. a recently implemented project, bit a useful template). It can be in demonstration or slide format… or both.

Towards the end of October I will post here on the final format of the sessions but we are looking for community volunteers to present the sessions. Do you want to present something? If so, comment on this post and I will get back to towards the end of October.

© Jeff Golden (Flickr)

If you can’t manage what you can’t measure, then it makes sense to start measuring any SDL Tridion environment so that you better understand what you are doing with your environment and help predict the future. I’ll get onto measuring what you publish and how often at a later date but I recently looked at measurement of content for a customer so I thought I would quickly share that.

At the base level I want to know something about how many creates or updates I do on a given SDL Tridion implementation. Now, doing this via the API will be troublesome because the API does not lend itself to doing operations over the entire content repository in one call and if I want to report I don’t want to end up with expensive reporting activities. By far and away the best way to do this is directly on the database and below I have listed out some queries that will give you some ideas on what information you could report on.

You could easily take these queries and knock them into a nice ASP.NET page to display your results or even include them in a bona fide reporting tool like Crystal Reports. The queries are all read only so they are safe to run on any environment.
Getting the number of publications; easy to see in the Tridion interface but why not start with a basic query:

SELECT count(*) FROM publications;

Not too hard I would say…

Components created for a given day with Schema Title; this was my main aim of the reporting, show what components are created and what schema they are based upon. This query gave me some headaches because the Schema (title) is actually a row in the same table that you are querying for your component information. You could take the easy way out and just retrieve the ID of the schema and look it up later but why take the easy way out?

SELECT DISTINCT itm1.title, COUNT(*) total  
FROM items itm1
JOIN
(
  SELECT itm3.title, itm3.schema_reference_id, itm3.item_reference_id, itm3.publication_id, itm3.version
  FROM items itm3
  WHERE TRUNC(itm3.modification_date)=TO_DATE('26-11-2009','DD-MM-YYYY')
  AND itm3.version = 1
  AND itm3.item_type=16
) itm2
ON (itm2.schema_reference_id = itm1.item_reference_id)
GROUP BY itm1.title
ORDER BY total DESC

So the crucial parts to this query are:

The JOIN, the sub query is the list of components that were created on the date in the WHERE clause. If you want those items they were updated you can better put “AND itm3.version > 1” or just leave it out completely if you want both created & updated in the same result. For a nice report you would want probably want to parameterize the date so you can report any given day but if you are just running the SQL you’ll need to update by hand. The “item_type = 16” refers to components and it is worth noting this query will not work for pages (item type of 64) because pages don’t have schemas, they have templates and they can also inherit them rather than having them set explicitly so it makes discovery of what template a page was created against a mind bender I did not feel inclined to take a look at (yes, I took the easy route).

Total new components for a given day by user; a good way to check who is doing what is looking, by day, who created content. It says nothing about how many meetings the users attended or even how many coffees they made for the team, just how much they created (i.e. made a version 1 component):

SELECT trs.name, count(*) total_new_items
FROM items itm, trustees trs
WHERE TRUNC(itm.modification_date)=TO_DATE('26-11-2009','DD-MM-YYYY')
AND itm.version = 1
AND itm.item_type=16
AND trs.id=itm.trustee_id
GROUP BY trs.name
ORDER BY total_new_items DESC;

As for the first query, updates can be found by using “> 1” for the version.

Total updates/creates of items per day; this query will show you, over time, what the total of items (e.g. components, pages, folders, publications) were created per day is.

SELECT modification_date, COUNT(*) total
FROM items
GROUP BY modification_date
ORDER BY total DESC

Hopefully that gives you a little insight into what you could report on your own environment. Collecting statistics is key to a successful environment and without it you do not know if your investment is well spent or if your project is really being successful.

TGIF?
So there I am, its 5 PM on a Friday and I am trying to get out that article that I have written in the afternoon. I am waiting for it to publish and it is taking longer than I want because I have better things to do. It’s Friday after all! I have to wait in that queue for my job to be processed after all the other jobs that were submitted before mine. But it is a queue, right? That’s what happens, I join the back and I wait until I get to the front for my turn. (At least that’s how we do it in the UK.  Also, we mumble.)

So what determines how long the queue takes?
Certainly not the user! The user determines how many jobs are in the queue but not how long it takes to complete the jobs.  The duration is mostly determined by a) the task, b) the templates, c) the servers and d) the configuration.

The Task
I am sure many of you have queued in the bank.( I am sure that is also why many of you turned to online banking.) There are 5 people in front of you but how long are you waiting? 5 minutes or 10 minutes? The answer is, you have no idea… it depends on what those people want from the bank. Some might want information, some cash and others might want a loan. Each person represents a complexity that will take x amount of minutes of handling by the servers (and yes and I expressly used the word “server” rather than “clerk”). Publishing Jobs submitted by the users are the same, in that the jobs vary in complexity and that cannot be determined by looking superficially at the job (or person) in question.

The complexity is determined in two ways. Firstly, what the user tried to publish and secondly, how the data model is constructed.  Now I know I said the user does not determine the queue duration and now I contradicted myself. But, I don’t believe that it is the user’s job to determine whether or not something should be published they need to publish what they need to publish. However, it is important to note that some Tridion items, when published, can take more items along for the ride. A Structure Group, for example, has pages and nested Structure Groups which need to be published also.

The data model determines our relationships between items. So when I publish an item, additional items will be taken because they complete an item. This typically is a small number of items, but the data model could need attention if publishing a single item leads to excessive numbers of additional items.

The Templates
When I get to the front of the queue at the bank, I am most likely going to be presented with some forms to fill in. Those legal documents that lets me get the money to buy a car or get a new credit card. How long it takes me to fill out those forms will determine how long it will be before I am finished. The smaller and simpler the form, the better! My publishing job will execute templates to create some sort of output (e.g. HTML, XML, Java or ASP.NET code).  The templates take time to execute and many templates may have to be executed for one job. The larger and more complex these templates, the longer it will be before I see my publish job completed.

The Servers
The speed the servers work at and the amount of simultaneous activities they can complete affects the overall speed of publishing a job. The servers must therefore be scaled to meet the load requirements of the environment. Much like the bank, the overall throughput effects the waiting time of any job, with a single server I will wait the longest, with multiple parallel servers the waiting time will be reduced. In most scaled environments the tendency will be to separate out publishing from other server functions (database, management interface), this dedication of a task means that the server can concentrate on the same repetitive task without being interrupted with other business and therefore improve the overall publishing throughput of this server.

The Configuration
The configuration options with SDL Tridion, allow you to manipulate how the queue is managed; it in essence all publishing jobs are equal but with Publishing Priorities and Filters some jobs are more equal than others. Priorities can be set by the end user at the time of publish or as a rule on a given Publishing Target. The priority (high, normal or low) allows the most important tasks to go first (or the least important tasks last) and works like any priority system would do.

Filtering adds an extra dimension to this and the overall way items are removed from the queue for publishing. Many banks have separate desks for different tasks. If you go to deposit some money you use a different desk to the desk where you get a loan. Filtering does the same thing, in that it allows us to specify certain servers to complete certain jobs depend upon its configuration. Filtering is possible by Publication (e.g. German Website), Publication Target (e.g. Live) or Priority (e.g. High) – or a combination of multiple filters with multiple values. So for certain areas of your business you could, for example, dedicate servers to complete just those tasks; so in times of lots of house buying, we have more servers on the loans desk and we divide our throughput unevenly across our customers.

4 actions to improve to improve publishing
I have not encountered a single organization yet who could not do with faster publishing. Even when you think it is as fast as it can be, there will still be room for improvement somewhere. In summary, I have four points you can act upon that can help you love publishing that little bit more…

1. Understand what you publish! You can’t manage what you can’t measure, so start measuring! The information you need in available in the API, so a simple report will be enough to help you collect the data needed:
1. What is your total job load per day?
2. How many items do these jobs consist of?
3. How long does it take to render the items in the job?
2. Investigate to see if adjusting the configuration can ensure that important publications get the time they deserve and those publications lower down the content management food chain get less time.
3. Have IT analyze the performance of your servers under stress, can the performance be improved? If so, then you will gain in throughput.
4. Undertake and analysis of the templates and the data model to see where performance can be improved. Having the data from the measurements can really help zoom into what is performing badly.

Event Horizon

As I mentioned in one of my previous post the event system has been revised to me more modular and, probably more importantly, to be native .NET.

An event, an action from a user, triggers the event system which is one or more .NET classes which will act upon this event in some way. In versions of Tridion prior to 2011 the event system has been a single class which then leads to having to combine different event activity into the same piece of code. For instance, additional logging and workflow elements might have to be combined; this leads to unnecessary complexity in the code itself. It also means you cannot easily turn off (and on) part of the code.

In SDL Tridion 2011, this has changed (although you can still run older event systems) to allow you to hook into events using multiple separate event systems. As Bart Koopman explains in this article, when we hook into the events we concern ourselves with the phase of the event, the type of event and the object itself.

The phase relates to where in the event you are, for example, initiated, processed, committed etc. The type refers to the type of event, e.g. Publishing, BluePrint, or Data Modification events. Lastly our object (or in more familiar terms “item”) itself; our Component, Page, Schema etc. With these I can easily define that I would like to fire an event on a Component, when it is published and after that publish action has started.

Two more additional features are important. Firstly, it is quite possible that you have more than one event system hooked into the same event, so with the “Event Subscription Order” you can also specify the order in which your code is triggered (creating a event system stack). Secondly, you can also specify an event to run asynchronously rather than wait for code to complete before other code can continue (synchronously).

Event System Example

We need to do two simple things to get this to work:

1)      Define out event system class

2)      Configure SDL Tridion to load the event system and make it active

The Event System Class

We need to define our event system and this being .NET we can pretty much do anything we want from here, however, we need to at minimum hook (or rather subscribe) to an event.

Our event is going to log everything that happens to a Windows Event log called “Message Log”. We open our class and declare all the variables we need etc:

using System;
using System.Diagnostics;
using System.Threading;
using Tridion.ContentManager;
using Tridion.ContentManager.Extensibility;
using Tridion.ContentManager.Extensibility.Events;

namespace Tridion.Utilities. ExampleEventSystem {
[TcmExtension("ExampleEventSystemEventHandlerExtension")]
      public class ExampleEventSystem : TcmExtension {
           #region Local Variables
           private static EventLog messageLog = new EventLog();
           #endregion

In our constructor we will initialize out Windows Event Log and call our subscribe method:

          public AuditLogger() {
               Init();
               Subscribe();
          }

Create the event log:

          private static void Init() {
               if (!EventLog.SourceExists("TridionMessages")) {
                   EventLog.CreateEventSource("TridionMessages ", "Message Log");
               }
               messageLog.Source = " TridionMessages ";
          }

Subscribe to our events:

          public void Subscribe() {
               EventSystem.Subscribe<IdentifiableObject, TcmEventArgs>( LogMessage, EventPhases.Processed);
          }

In this case I have used all “IdentifiableObject” which is everything, the method, LogMessage, that will called when the event fires and the phase is “processed”. The method signatures for subscribing are:

Subscribe<TSubject, TEvent>(TcmEventHandler<TSubject, TEvent> eventHandler, EventPhases phases, EventSubscriptionOrder order)
SubscribeAsync<TSubject, TEvent>(TcmEventHandler<TSubject, TEvent> eventHandler, EventPhases phases, EventSubscriptionOrder order)

In our case we defined no subscription order so it will be “normal” order.

And finally define the method that will actually log our message:

          private static void LogMessage(IdentifiableObject subject, TcmEventArgs args, EventPhases phase) {
               String message = "Session user: " + subject.Session.User.Title.ToString();
               message = message + "\nTarget URI: " + subject.Id;
               message = message + "\nItem Type: " + subject.GetType().Name;
               message = message + "\nEvent: " + args.GetType().ToString();
               message = message + "\nPhase: " + phase.ToString();
               messageLog.WriteEntry(message, EventLogEntryType.Information, 0, 0);
          }
     }
}

Configuring Tridion

To load our event system, we must load the class responsible. This we do in the Tridion.ContentManager.config which can be found under the %TRIDION_HOME%\config directory and we add a line to the extensions section to add our assembly:

<extensions>
<add assemblyFileName="C:\myevevntsystem\messagelogger.dll"/>
</extensions>

Online Marketing Explorer

It has been released and I decided to list out five things that I think are very important to the release of 2011. Mostly talk has been about the new Content Manager Explorer (and it’s cross browser functionality) but this is just the surface. Underneath there are many changes of which just five is a small snippet of what you can find in the box…

The five:

Scalable Deployer

To meet the demands of a large scale enterprise the deployer is now much more scalable than before allowing organizations to constantly grow their environment to meet the demands placed upon it by a growing content organization. The scalable deployer allows multiple processes to simultaneously process deployments as well as updating the publisher on how much load they can handle to avoid overloading.

Storage Layer

The new Content Delivery storage layer is based upon the Java Persistence API (JPA) and its concrete implementation, Hibernate.  With this you are able to expand the single Content Delivery storage layer to encompass multiple different data sources (e.g. product information or user generated content)  into one single layer.

Online Marketing Explorer

Drive Customer Impact with the new Online Marketing Explorer. Giving an overview of your marketing activities with a centralized model of campaigns, reporting and actionable insights.

Content Services

Content Services is the new RESTful webservice on Content Delivery based upon the oData and oAuth standards. You now have your published content available through a webservice to any application, mobile app, affiliate, white label site, 3rd party the list goes on…

Event System

The new fully .NET event system is modular. Does not sound like much but you can add one or more separate event systems to the same Content Management environment. Each event system can work alone hooking into different CM activities or can work together as part of coordinated event driven activities.

As I love SDL Tridion I wanted to share with you the video released this week that tells, from our employees point of view, why we all love SDL Tridion so much…

PS: Don’t forget to switch it to HD!