What is Splunk?
I think the chaps from Splunk explain that the best…

Splunk is the engine for machine data. Use Splunk to collect, index and harness the fast moving machine data generated by all your applications, servers and devices — physical, virtual and in the cloud. Search and analyze all your real-time and historical data from one place.
Splunking your machine data lets you troubleshoot problems and investigate security incidents in minutes, not hours or days. Monitor your end-to-end infrastructure to avoid service degradation or outages. Meet compliance mandates at lower cost. Correlate and analyze complex events spanning multiple systems. Gain new levels of operational visibility and intelligence for IT and the Business.

Cool… but then why is this nice?
Traditional support organizations supporting large web farms often have monitoring in place and then when something goes wrong they go to the server and search the logs until they have the reason for the failure. At the bare minimum, which is where I am at, Splunk aggregates all the logs from your entire web farm and presents is as one complete picture. From this I can see trends and machines with difficulties (problematic to spot when you have alot of servers). I am sure Splunk does a lot more, but I am not yet there. When I have discovered them, I will share them but so far I am a happy camper.

Adding SDL Tridion (of course) as a data source
Splunk takes multiple sources of log data and the usual list of candidates are web server (in my case, IIS) logs and Windows Event Logs. I want my Content Delivery logs integrated and so I set about doing this.

For this I will leverage SDL Tridion 2011’s updated logging (LOGBack) and with this I am able to configure logging to output logs to Syslog. Syslog is a standard way of logging application messages that separates the software application and the system logging the messages. In essence, the messages can be pushed out much like a network broadcast and a logging system, Splunk, captures that.
SO I want to configure the logging in SDL Tridion Content Delivery to push the messages to Syslog. For this we need two things: 1) an Appender and 2) a reference to that appender for my log. The Appender configures my log output and looks like:

<appender name="SYSLOG">
     <syslogHost>localhost</syslogHost>
     <facility>LOCAL0</facility>
     <suffixPattern>[%thread] %-5level %logger %msg</suffixPattern>
</appender>

This is the example from the LOGBack help documentation (http://logback.qos.ch/manual/appenders.html) with the addition of the logging level (“%-5level” which captures “ERROR”, “INFO” etc.). Then I can set a given log to push it’s messages to my Syslog appender:

<logger name="com.tridion.deployer">
     <appender-ref ref="rollingDeployerLog"/>
     <appender-ref ref="SYSLOG"/>
</logger>

I’ve kept in the original logging (“rollingDeployerLog”) just for testing and validation.

In Splunk I need to add a data source to listen to this output. So I add a new local Syslog data source specifying the (standard) port. I don’t need to specify a hostname because I did that in my logging configuration on Tridion. I told it to push the messages to “localhost” but that could be any server.

After that, once Tridion logs messages, Splunk will pick them up and store them.

So in November we’ll see the first of these sessions which we hope to repeat over the course of 2011/2012.

So what is the idea? The idea is that anyone can join in and if desired present on a community organized and driven event. The sessions, covering 10-15 minutes, can cover any topic related to a SDL Tridion (e.g. a recently implemented project, bit a useful template). It can be in demonstration or slide format… or both.

Towards the end of October I will post here on the final format of the sessions but we are looking for community volunteers to present the sessions. Do you want to present something? If so, comment on this post and I will get back to towards the end of October.

On Thursday the 2011 MVP weekend started with 11 of the 15 MVPs, or Most Valued Professionals, flying in from across the globe to hotel by the sea near Lisbon (technically, some of the 11 are community builders which are a sort of MVP for SDL employees). If you are not familiar with what an SDL Tridion MVP is then, in short, it is a person who distinguishes themselves by sharing their knowledge over and above what their role demands. They can be anyone from any discipline but they must of course share SDL Tridion knowledge and promote the SDL Tridion community.

Sharing is not an easy thing to do and I, like many of the MVPs, have a busy job. But on the whole the MVPs are contributing to the community and our efforts grow steadily each year as we get more and more familiar of not only what it takes to be an MVP but also what it takes to show it. To quote Tom Jones by Henry Fielding  “Make sure Tom that your actions are not all good, take care that they appear good as well.”  Quoting is very fake elitist sort of thing to do, people do that to sound smart so before any one gets any ideas about me, I watched the BBC adaptation on DVD; the phrase just stuck with me.

Over the week and eventual weekend, the MVPs have been discussing, planning and coding their hearts out from dawn till dusk it what might be describe as a little calmer weekend compared to last year. The coding outcome is the beginning of the rebuilding of the SDL Tridion PowerTools; a hugely popular and useful set of tools to go with any SDL Tridion implementation. The current version does not run on 2011 since 2011 had a graphical interface overhaul allot of the dependencies have since vanished and thus a rebuild is required. More about this in another post but none the less I must mention that the project is looking for volunteers from any part of the community that would like to write a PowerTool or just help with testing.

What little time left of the weekend was spent relaxing on the beach or in Lisbon. The hotel where the MVPs stayed is on the Atlantic coast of Portugal near to Lisbon – or technically nearer to Cascais, a town which looks like a smaller version of Monaco. The hotel, an old fort is shouldered on either side by golden sand beaches with some of the biggest waves I have even seen (ok, I have not really seen _that_ many waves up this close). But something towards 3 meters is enough to knock myself and 10 other MVPs back onto the shore like slightly disgruntled penguins.

The flights we took down to Portugal were with the airline TAP. An airline, much like Alitalia, that has a troubled existence of bankruptcy, sale, cut back etc. Leaving Amsterdam proved to be our first challenge of the weekend with the check-in line taking nearly an hour to check-in a handful of customers. For many, online check-In had not worked. For one of us, it had not worked because the ticket he had did not actually exist. Making the plane via the originally, but ultimately incorrect gate, we boarded with a minute to spare and promptly sat waiting, as waiting turned into a delay of 30 minutes. Why did I run again? I can run, when I need to – and quicker than you would expect – but it is not something I do on a regular basis and needless to say, I took some time to recover. Having made it to Lisbon airport still disgruntled by the rushing in the morning, we were approached by Jose of the local baggage handling firm. The airline, he informed us, had left my bag in Amsterdam because the plane was too heavy. I know I do not pack light but I felt, for a moment, slightly offended until I found out nine other people were in the same boat. Directed off to lost property to fill in the appropriate forms, Jose, who magically reappeared in the lost property office, assure me that the bag would make it on the next flight at 2 PM and would probably, would be at my hotel by 8 PM. In the meantime, fellow MVP, Peter had discovered that his bag was also nowhere to be seen and that, after visiting lost property, it was nowhere to be found. Both bags turned up at around 10 PM that evening, much to our relief.

The flight back, Peter remarked, “would hopefully not be as bad as the way to Lisbon.” Never could words be further from the truth. On Sunday night I slept in a hotel in Costa da Caparica, Almada; an area on the other side of the river from Lisbon. Not many might know this, but Lisbon has alot in common with San Francisco; steep hills, funny little trams, a prison on an island and a Golden Gate Bridge (known as the 25th April Bridge) and it is on one end of this look alike bridge that Almada is situated and it is also where TAP has decided to put its refugee passengers from the evening’s antics.

Firecrews standing by...

Firstly the flight was delayed by 45 minutes and then no sooner as we were bussed out to the plane, we were bussed right back to the terminal where we were “debussed” back into the terminal, only for three minutes later to be “rebussed” and taken back to the plane; the pilot, having changed his mind about whether not we would be allowed to fly with him. After take-off I began to sense that things were not completely good. The sensation of the landing gear going in, then out, then in again did not seem to fit, but then was probably I misinterpreting the various groans, creeks and clunks of the aging Fokker 100 (Jose from lost property told me about these aging aircraft, it seemed he was a plane buff too as well as a bag finding bearer of bad tidings). The second thing that did not seem right was that the steward, when signalled he could get up, got up, but then came back again and sat down. Normally they run into a frenzy of getting coffee ready. As it turned out, the pilot had discovered that the landing gear would not go up correctly (do I fly too much?) and would have to return to the airport; the steward clarified for me that the problem was only that it did not go up, not that it would not go down so I was somewhat relieved.  Standard operating procedure for such a situation is that the stewards hand out a cups of water – something akin to men boiling water when a baby will be born, an act whilst moderately useful is more designed to keep people busy – and the pilot needs to lose fuel and that takes about an hour. After that hour we returned somewhat nervously to the airport. The landing was smooth and with a big sigh of relief the pilot got his customary round of applause from the passengers for saving them from a fiery death.

After we were bussed back to the terminal, we stood in the exit of the arrivals (I got the hint immediately). The staff of TAP then tried to organize the situation. This they did by starting a stock market style trading frenzy of boarding passes and negotiations about bags and changing the flights. With my name, they will rarely say it right and I was tuned into all 15 variations that people use (mostly variations that are other names e.g. Wright). With shock and awe I heard my name correctly pronounced and I squeezed my way through to the hand holding up the ticket. The Portuguese are not that tall on average and surrounded by 50 Dutch, who are average allot taller, the only thing I could see was that hand. After much handing around and waiting to get my own bag I followed a trio of feisty brunet, brown eyed Portuguese TAP ground crew who, on the whole, looked much like each other. One declared in a loud voice normally reserved for large groups of Japanese tourists, that “those going to Amsterdam should follow her”. We continued to follow the three as the “argued” and grumped at each other as to how the whole situation was developing. As I looked and glanced around I was shocked to witness the TAP ground crew equivalent of a “Thach Weave”, an aerobatic manoeuvre where those on the left split right and those on the right split left (leaving the picture of interlocking weave). Which one of these motor mouths was I to follow again? I stuck it out only heading in the same direction hoping that I had chosen correctly. I did and at least for the time being I was on a bus going to a hotel. The hotel was 30 minutes away and I did at points wonder if I was on the right bus going to a hotel or a bus taking up to Amsterdam. I had given up caring and was only looking forward to a hot shower and some sleep.

For the weekend, I would like to thank Nuno Linhares. His dedication and devotion to knowledge sharing over and above what his job already asks (which is over and above what is normal) is amazing. A great host and an awesome guy; just don’t let him borrow your car…

[Update] The flight “drama” made the news.

See! White and Fluffy

Sadly, “The Cloud” is a term much abused these days; much like the word “Hoover” which apparently, in the UK at least, applies to any vacuum cleaner whether or not it is actually a Hoover. Everyone and anyone seems to label their service with the term Cloud so long as it appears like it might actually be the cloud. So what actually is the cloud?

In history, the cloud is that thing that appears on an infrastructure diagram, the icon the represents the Internet and all that goes with it. Like a 21st Century equivalent of the remark “ere be dragons”, the cloud picture represents something that is too hard to draw – even if you wanted to try – but it is something that is an essential part of our infrastructure. I’ve drawn thousands of web architectures with the cloud on the right (or left, depending on the day) and it was there to deliver our visitors to our site; without it out efforts would have been worthless.

So the Cloud is the Internet and any service labeled as being “Cloud” is not the Cloud unless it is actually in the Cloud. Alot of hosting providers label their services as Cloud because that is what the market demands, not because they actually are in the Cloud. They offer a service on the other side of the Cloud. Like East and West Berlin with the Cloud being the gap between the two, hosting providers are no more the Cloud than your own Infrastructure. That is not to say they can’t offer virtualization, per minute billing and elasticity (many don’t) but it is more to say they have a protected network or data center that they maintain. Amazon, which is in the cloud, provides you with a service that is somewhere. You will be unable to draw it on a diagram because you have simply no idea where it is or even what it looks like.

© Jeff Golden (Flickr)

If you can’t manage what you can’t measure, then it makes sense to start measuring any SDL Tridion environment so that you better understand what you are doing with your environment and help predict the future. I’ll get onto measuring what you publish and how often at a later date but I recently looked at measurement of content for a customer so I thought I would quickly share that.

At the base level I want to know something about how many creates or updates I do on a given SDL Tridion implementation. Now, doing this via the API will be troublesome because the API does not lend itself to doing operations over the entire content repository in one call and if I want to report I don’t want to end up with expensive reporting activities. By far and away the best way to do this is directly on the database and below I have listed out some queries that will give you some ideas on what information you could report on.

You could easily take these queries and knock them into a nice ASP.NET page to display your results or even include them in a bona fide reporting tool like Crystal Reports. The queries are all read only so they are safe to run on any environment.
Getting the number of publications; easy to see in the Tridion interface but why not start with a basic query:

SELECT count(*) FROM publications;

Not too hard I would say…

Components created for a given day with Schema Title; this was my main aim of the reporting, show what components are created and what schema they are based upon. This query gave me some headaches because the Schema (title) is actually a row in the same table that you are querying for your component information. You could take the easy way out and just retrieve the ID of the schema and look it up later but why take the easy way out?

SELECT DISTINCT itm1.title, COUNT(*) total  
FROM items itm1
JOIN
(
  SELECT itm3.title, itm3.schema_reference_id, itm3.item_reference_id, itm3.publication_id, itm3.version
  FROM items itm3
  WHERE TRUNC(itm3.modification_date)=TO_DATE('26-11-2009','DD-MM-YYYY')
  AND itm3.version = 1
  AND itm3.item_type=16
) itm2
ON (itm2.schema_reference_id = itm1.item_reference_id)
GROUP BY itm1.title
ORDER BY total DESC

So the crucial parts to this query are:

The JOIN, the sub query is the list of components that were created on the date in the WHERE clause. If you want those items they were updated you can better put “AND itm3.version > 1” or just leave it out completely if you want both created & updated in the same result. For a nice report you would want probably want to parameterize the date so you can report any given day but if you are just running the SQL you’ll need to update by hand. The “item_type = 16” refers to components and it is worth noting this query will not work for pages (item type of 64) because pages don’t have schemas, they have templates and they can also inherit them rather than having them set explicitly so it makes discovery of what template a page was created against a mind bender I did not feel inclined to take a look at (yes, I took the easy route).

Total new components for a given day by user; a good way to check who is doing what is looking, by day, who created content. It says nothing about how many meetings the users attended or even how many coffees they made for the team, just how much they created (i.e. made a version 1 component):

SELECT trs.name, count(*) total_new_items
FROM items itm, trustees trs
WHERE TRUNC(itm.modification_date)=TO_DATE('26-11-2009','DD-MM-YYYY')
AND itm.version = 1
AND itm.item_type=16
AND trs.id=itm.trustee_id
GROUP BY trs.name
ORDER BY total_new_items DESC;

As for the first query, updates can be found by using “> 1” for the version.

Total updates/creates of items per day; this query will show you, over time, what the total of items (e.g. components, pages, folders, publications) were created per day is.

SELECT modification_date, COUNT(*) total
FROM items
GROUP BY modification_date
ORDER BY total DESC

Hopefully that gives you a little insight into what you could report on your own environment. Collecting statistics is key to a successful environment and without it you do not know if your investment is well spent or if your project is really being successful.

TGIF?
So there I am, its 5 PM on a Friday and I am trying to get out that article that I have written in the afternoon. I am waiting for it to publish and it is taking longer than I want because I have better things to do. It’s Friday after all! I have to wait in that queue for my job to be processed after all the other jobs that were submitted before mine. But it is a queue, right? That’s what happens, I join the back and I wait until I get to the front for my turn. (At least that’s how we do it in the UK.  Also, we mumble.)

So what determines how long the queue takes?
Certainly not the user! The user determines how many jobs are in the queue but not how long it takes to complete the jobs.  The duration is mostly determined by a) the task, b) the templates, c) the servers and d) the configuration.

The Task
I am sure many of you have queued in the bank.( I am sure that is also why many of you turned to online banking.) There are 5 people in front of you but how long are you waiting? 5 minutes or 10 minutes? The answer is, you have no idea… it depends on what those people want from the bank. Some might want information, some cash and others might want a loan. Each person represents a complexity that will take x amount of minutes of handling by the servers (and yes and I expressly used the word “server” rather than “clerk”). Publishing Jobs submitted by the users are the same, in that the jobs vary in complexity and that cannot be determined by looking superficially at the job (or person) in question.

The complexity is determined in two ways. Firstly, what the user tried to publish and secondly, how the data model is constructed.  Now I know I said the user does not determine the queue duration and now I contradicted myself. But, I don’t believe that it is the user’s job to determine whether or not something should be published they need to publish what they need to publish. However, it is important to note that some Tridion items, when published, can take more items along for the ride. A Structure Group, for example, has pages and nested Structure Groups which need to be published also.

The data model determines our relationships between items. So when I publish an item, additional items will be taken because they complete an item. This typically is a small number of items, but the data model could need attention if publishing a single item leads to excessive numbers of additional items.

The Templates
When I get to the front of the queue at the bank, I am most likely going to be presented with some forms to fill in. Those legal documents that lets me get the money to buy a car or get a new credit card. How long it takes me to fill out those forms will determine how long it will be before I am finished. The smaller and simpler the form, the better! My publishing job will execute templates to create some sort of output (e.g. HTML, XML, Java or ASP.NET code).  The templates take time to execute and many templates may have to be executed for one job. The larger and more complex these templates, the longer it will be before I see my publish job completed.

The Servers
The speed the servers work at and the amount of simultaneous activities they can complete affects the overall speed of publishing a job. The servers must therefore be scaled to meet the load requirements of the environment. Much like the bank, the overall throughput effects the waiting time of any job, with a single server I will wait the longest, with multiple parallel servers the waiting time will be reduced. In most scaled environments the tendency will be to separate out publishing from other server functions (database, management interface), this dedication of a task means that the server can concentrate on the same repetitive task without being interrupted with other business and therefore improve the overall publishing throughput of this server.

The Configuration
The configuration options with SDL Tridion, allow you to manipulate how the queue is managed; it in essence all publishing jobs are equal but with Publishing Priorities and Filters some jobs are more equal than others. Priorities can be set by the end user at the time of publish or as a rule on a given Publishing Target. The priority (high, normal or low) allows the most important tasks to go first (or the least important tasks last) and works like any priority system would do.

Filtering adds an extra dimension to this and the overall way items are removed from the queue for publishing. Many banks have separate desks for different tasks. If you go to deposit some money you use a different desk to the desk where you get a loan. Filtering does the same thing, in that it allows us to specify certain servers to complete certain jobs depend upon its configuration. Filtering is possible by Publication (e.g. German Website), Publication Target (e.g. Live) or Priority (e.g. High) – or a combination of multiple filters with multiple values. So for certain areas of your business you could, for example, dedicate servers to complete just those tasks; so in times of lots of house buying, we have more servers on the loans desk and we divide our throughput unevenly across our customers.

4 actions to improve to improve publishing
I have not encountered a single organization yet who could not do with faster publishing. Even when you think it is as fast as it can be, there will still be room for improvement somewhere. In summary, I have four points you can act upon that can help you love publishing that little bit more…

1. Understand what you publish! You can’t manage what you can’t measure, so start measuring! The information you need in available in the API, so a simple report will be enough to help you collect the data needed:
1. What is your total job load per day?
2. How many items do these jobs consist of?
3. How long does it take to render the items in the job?
2. Investigate to see if adjusting the configuration can ensure that important publications get the time they deserve and those publications lower down the content management food chain get less time.
3. Have IT analyze the performance of your servers under stress, can the performance be improved? If so, then you will gain in throughput.
4. Undertake and analysis of the templates and the data model to see where performance can be improved. Having the data from the measurements can really help zoom into what is performing badly.

Event Horizon

As I mentioned in one of my previous post the event system has been revised to me more modular and, probably more importantly, to be native .NET.

An event, an action from a user, triggers the event system which is one or more .NET classes which will act upon this event in some way. In versions of Tridion prior to 2011 the event system has been a single class which then leads to having to combine different event activity into the same piece of code. For instance, additional logging and workflow elements might have to be combined; this leads to unnecessary complexity in the code itself. It also means you cannot easily turn off (and on) part of the code.

In SDL Tridion 2011, this has changed (although you can still run older event systems) to allow you to hook into events using multiple separate event systems. As Bart Koopman explains in this article, when we hook into the events we concern ourselves with the phase of the event, the type of event and the object itself.

The phase relates to where in the event you are, for example, initiated, processed, committed etc. The type refers to the type of event, e.g. Publishing, BluePrint, or Data Modification events. Lastly our object (or in more familiar terms “item”) itself; our Component, Page, Schema etc. With these I can easily define that I would like to fire an event on a Component, when it is published and after that publish action has started.

Two more additional features are important. Firstly, it is quite possible that you have more than one event system hooked into the same event, so with the “Event Subscription Order” you can also specify the order in which your code is triggered (creating a event system stack). Secondly, you can also specify an event to run asynchronously rather than wait for code to complete before other code can continue (synchronously).

Event System Example

We need to do two simple things to get this to work:

1)      Define out event system class

2)      Configure SDL Tridion to load the event system and make it active

The Event System Class

We need to define our event system and this being .NET we can pretty much do anything we want from here, however, we need to at minimum hook (or rather subscribe) to an event.

Our event is going to log everything that happens to a Windows Event log called “Message Log”. We open our class and declare all the variables we need etc:

using System;
using System.Diagnostics;
using System.Threading;
using Tridion.ContentManager;
using Tridion.ContentManager.Extensibility;
using Tridion.ContentManager.Extensibility.Events;

namespace Tridion.Utilities. ExampleEventSystem {
[TcmExtension("ExampleEventSystemEventHandlerExtension")]
      public class ExampleEventSystem : TcmExtension {
           #region Local Variables
           private static EventLog messageLog = new EventLog();
           #endregion

In our constructor we will initialize out Windows Event Log and call our subscribe method:

          public AuditLogger() {
               Init();
               Subscribe();
          }

Create the event log:

          private static void Init() {
               if (!EventLog.SourceExists("TridionMessages")) {
                   EventLog.CreateEventSource("TridionMessages ", "Message Log");
               }
               messageLog.Source = " TridionMessages ";
          }

Subscribe to our events:

          public void Subscribe() {
               EventSystem.Subscribe<IdentifiableObject, TcmEventArgs>( LogMessage, EventPhases.Processed);
          }

In this case I have used all “IdentifiableObject” which is everything, the method, LogMessage, that will called when the event fires and the phase is “processed”. The method signatures for subscribing are:

Subscribe<TSubject, TEvent>(TcmEventHandler<TSubject, TEvent> eventHandler, EventPhases phases, EventSubscriptionOrder order)
SubscribeAsync<TSubject, TEvent>(TcmEventHandler<TSubject, TEvent> eventHandler, EventPhases phases, EventSubscriptionOrder order)

In our case we defined no subscription order so it will be “normal” order.

And finally define the method that will actually log our message:

          private static void LogMessage(IdentifiableObject subject, TcmEventArgs args, EventPhases phase) {
               String message = "Session user: " + subject.Session.User.Title.ToString();
               message = message + "\nTarget URI: " + subject.Id;
               message = message + "\nItem Type: " + subject.GetType().Name;
               message = message + "\nEvent: " + args.GetType().ToString();
               message = message + "\nPhase: " + phase.ToString();
               messageLog.WriteEntry(message, EventLogEntryType.Information, 0, 0);
          }
     }
}

Configuring Tridion

To load our event system, we must load the class responsible. This we do in the Tridion.ContentManager.config which can be found under the %TRIDION_HOME%\config directory and we add a line to the extensions section to add our assembly:

<extensions>
<add assemblyFileName="C:\myevevntsystem\messagelogger.dll"/>
</extensions>