In my previous posts I discussed the foundational principles of DevOps – known by their acronym of ‘CALMS’ and looked in detail at the first three of these. We now take an overview of the fourth – ‘M’, which we all know stands for ‘Measurement’.
So far, we’ve focussed on how to transform your organisation into a leaner, fitter, more cohesive and agile state. But the question remains, how on earth do you know if what you are doing is correct? What changes are providing improvement, and which are costing more but delivering less?
There’s no point in making changes to the status quo unless you have some method of tracking whether those changes are delivering results or are proving detrimental to your business – so you need to track and to measure these changes over time to determine what does, and – more importantly – does not, work.
So, what sort of metrics do we need?
This question is not quite so facetious as it first sounds. The instant gut feel is to instrument and record everything. After all, we routinely hoard data every day, so why not apply some metrics to this data to track item counts, time tickets spend in queues, time applications take to compile or deploy, or any one of an almost infinite number of data points that can be tracked and measured?
Whilst you can take this approach, you will rapidly come to learn that 99.99% of these (at least) impart no real meaning outside of any specific context and will very soon leave you drowning in data that you are unable to action, so you need to think about defining a narrower set of KPIs, or Key Performance Indicators.
So, what differentiates a KPI from any other metric?
In order to answer this, I’m going to refer to the excellent piece written by Avinash Kaushik from his book Web Analytics 2.0. Here, he gives the following definition for a KPI:
A key performance indicator (KPI) is a metric that helps you understand how you are doing against your objectives.
There, it’s as simple as that. Whilst you may have many metrics, your KPIs are those that you will reference again, and again, to measure your success. The word objectives being the key word in that quote.
By way of an example, by adopting DevOps, you may have set as an objective that you want to increase customer satisfaction with your product. One of the goals you may have set to help achieve this is to respond more rapidly to customer requests for product features or to reduce the number of defects in delivered product.
Thus, we have a couple of obvious metrics (and maybe a not so obvious one). We have:
- The obvious
- Number of reported defects per release
- Average time to closure of requests for features
- Number of new features per release
- The not-so obvious
- Overall client satisfaction
How could you measure client satisfaction? Well, not using a direct metric from your SRM tool or deployment tool perhaps, but you could run regular customer satisfaction surveys throughout the year (say, monthly or quarterly), track a Net Promoter Score and view this trend over time. The survey doesn’t need to be complex. In fact, the best survey should just be a single question – “How likely are you to recommend this product/service?”
Once you have identified the metrics that relate to your objective(s), how do you find the ‘best’ ones. One simple way is that a KPI should be measurable, actionable, simple, relevant and timely. If a KPI isn’t one of these things, then it is a poor candidate. KPIs that fulfil more of these, are better.
Poor KPIs, better known as vanity metrics, impart little knowledge. Simply put, to see if you have a vanity metric ask yourself two simple questions:
- Does this value going up make me feel good?
- Does this value going up help me make decisions?
If the answer to (1) is Yes, but the answer to (2) is No, then you have yourself a vanity metric.
Once you have your KPIs defined, and reports produced showing trends over time, share these with your DevOps teams, and together determine what needs to change next to continue improving against your objectives. If this means that your existing KPIs become outdated or obsolete, then drop them in favour of new measures that track the new systems and processes.
Continuous improvement is a cornerstone of any Agile methodology, and this applies to your metrics and KPIs as much as any other area of your organisation.
Next time, I’ll bring the study of foundational principles to a close when we look at the final link in the chain ‘S’ – for Sharing, which will bring us full-circle.
Meanwhile, to learn more about presenting good metrics and KPIs, why not check out the blogs by my colleague Toby Isaacson?
Kaushik, A. – Web Analytics 2.0: The Art of Online Accountability and Science of Customer Centricity (Sybex, 2009) ISBN: 978-0-470-52939-3