IT Process Automation (ITPA) – How I See It

I thought for this blog, I'd write something slightly different rather than just the usual Powershell piece.

I have been working in the IT Process Automation (ITPA) arena for around 2 years now. It's a fascinating industry that's starting to emerge, and as more and more large IT companies realise it's potential, the core concept is really starting to take hold.

The main idea behind it is automating processes which are generally highly repetitive, quite often long winded and are prone to user errors. Although, this does pose a large block in front of any manager when your staff ask: "But doesn't this mean we'll be out of a job?". It's the largest problem I've faced when trying to get various processes rolled out. Personally, I feel that by removing these time wasting tasks, the staff that you already have will have more time to work on interesting projects and further their skill sets and thus career. This is still a tough sell....

The main product that I have worked with from an workflow perspective is NetIQ Aegis, but I've also seen Symantecs Altiris workflow product and Microsoft's Opalis. These are great products, although, with one flaw. The way they are marketed is such that "Any business process owner can put together a workflow". Incorrect. I'd really like to see a business manager attempt to throw together a technical workflow without any prior IT experience. To deliver this ease, the approach tends to be to abstract the code from the user, and present them with thousands of buttons and options to click. The reality is that most users who will design workflows are going to be either DevOps or a developer. Both of which would much rather have keyboard access straight into editing variables etc. SQL SSIS seems to have gone slightly the other way with this, but I think I'd prefer that approach.

You may ask, "Well, if you want to just start typing out scripts, then why do you bother with workflow in the first place?". Well, workflow has a great deal to offer. It allows for a very robust process to be put together which in most cases can be easily watched, tracked and reported upon. This alone is worth a lot to most businesses.

The biggest topic next to workflow, is "How do you initiate workflow?"

The way most workflow products out there seem to work, relys on a manual button press from someone. Whether that's triggered through the built in UI, or via an API call or a database trigger from an 3rd party app, who knows. This is great if you want to load up your service desk with a thousand potential buttons to press in the event of something happening, but in most cases, that's not the best move....

Complex Event Processing (CEP) - This is the area where it's all moving to, and it's one of the key reasons why I've been working with NetIQ Aegis for the past 2 years. Aegis cleverly combines the StreamBase CEP engine, which allows you to watch for patterns in all of your data streams "while it is in flight", simultaneously. Therefore, you can configure a custom trigger to watch for something like:

In any 5 minute window
If you see a ping timeout from a server in data centre 1
AND you see a URL timeout for a domain in this group
AND you see a disk capacity warning for one of these servers

Trigger something.....

You can then have a workflow which specifically caters for that scenario. No users needed, it just gets on with it.

Unfortunately, there are a few occasions where you might want your workflow engine not to react i.e. during a planned maintenance window or during a major incident. The trick to this is designing your architecture so you can restrict the flow of data into your CEP engine at the right times, therefore, creating some form of data flow controller in front of it.... pie in the sky stuff though.

Either way, CEP is fantastic. Annoyingly, it's really marketed in the financial sector due to the fact it can watch for stock trade patterns etc and allow traders to make choices quickly. However, as you can see, there are some fantastic uses for it in a data centre environment.

So, what's the best of both worlds?

Well, as I mentioned above, I've been using NetIQ Aegis, and it was the product we picked for a lot of reasons, but mainly because of the CEP engine integration. The downside is that the StreamBase engine is wrapped by the product and you therefore don't get a huge amount of access to how it actually works, which can be quite frustrating sometimes... although, I know it's a Java based product and it's not overly easy to integrate with .net, so I'm not too worried.

As you all know, I'm a massive Powershell advocate and believe that there are boundless time savings to be had from it in large IT enterprises. However, imagine Powershell bolted to a decent CEP? Powershell that ran automatically when it knew something needed to be done.....

You can sort of do this with Aegis, but I'd like a much closer to the metal approach, and believe that this could easily be delivered by Microsofts new StreamInsight product.

StreamInsight is Microsofts new CEP engine, and it ships as part of SQL 2008 R2, although it's not actually a server product. If you get a license for it, you will receive 10 or 12 dll's which you can import into a .net c# project and build your own engine. You can connect to as many different data streams as you like, process them how you wish and then fire them off to any system you deem fit to deal with them all in real time. If you had a service which fired off Powershell at the end, then you'd be done.....

I'd love to see a product like this to come on the market, but I think we're a bit of a way off, so I'm currently working on my own.

If you're interested in looking at Stream Insight or CEP in general, Alan Mitchell did a great session on StreamInsight at SQLBits in 2010 and 2011, you can see his deep dive video here.

Hope you found this post interesting, and give me a shout if you're interested in process automation or CEP.