Avoid APM Vanity Metrics – Use What Matters

Metrics are how we assess aspects of our business, applications, and infrastructure. They allow us to measure and report on key indicators, and we can then use that information for making decisions, or to adjust for improved performance, stability and overall quality.

While collecting and analyzing metrics can be hard work, in many respects, the most challenging aspect of working with metrics is identifying which ones you should collect in the first place. You want to spend your time collecting metrics that matter, and avoid “vanity metrics”, by which I mean those that don’t accurately reflect your environment, or provide extraneous information that might seem useful but does not actually improve outcomes.

In this article I explain why vanity metrics pose a challenge and how to avoid them so that you collect and analyze only those metrics that help to drive positive change.

Vanity metrics: Feel awesome, but bad for action

Vanity metrics might seem attractive, but they don’t deliver true value.

There is an argument to be made for picking metrics which make your product look good, but it’s not advisable. Sure, they’ll look great to stakeholders, but it will also lull you into a false sense of security.

Another problem with vanity metrics is that if you are using them to drive action, you might be set up to do a whole lot of work maximizing the wrong things.

Metrics should drive action and should highlight aspects of your system that are critical to its health and continued performance, so it is vital that you identify the right metrics to track.

Metrics that matter

Now that you know why vanity metrics are not useful, let’s discuss how to identify the metrics that do matter.

I’d like to introduce you to the four A’s of effective metrics.

• Accountable
• Actionable
• Accessible
• Auditable

Here’s what each of these types of metrics mean.

Accountable

Before you measure anything, determine who the owner is. If the ownership of a metric is ambiguous or shared between multiple teams, you may encounter problems tying the success or failure of the metric to a particular individual or group; or worse, the success of one team may be eclipsed by the sub-optimal performance of another.

Establish a clear owner for each metric.The importance of this will become more apparent as we continue with the rest of the characteristics of metrics that matter.

Actionable

Any metrics which you track should be actionable and should be tied to specific performance targets. If there isn’t an action you can take in response to a given metric, the value of that particular metric becomes questionable.

Accessible

You can have the greatest minds in your organization curate a report with every possible permutation of raw and derived metrics, but if the info isn’t in an easily digestible format, it’s not going to be implementable by those in the trenches. Metrics and the reports containing them should be easy to understand for anyone who might read them.

Auditable

Finally, ensure that you can validate the accuracy of metrics. Using metrics as a foundation from which to drive progress is only relevant if the improvement can be measured, and if the particular measurements being instrumented are accurate.

Metrics are key to monitoring

Monitoring is the process of collecting metrics from an application in real time or near-real time. The metrics are then used to determine the current state and health of the application and the infrastructure on which it installed.

When implemented correctly, with the right collection of metrics, an effective monitoring plan can ensure that your applications are running efficiently and effectively.

Additionally, if things take a turn for the worse, developers and production support personnel should be able to use the data available from your monitoring system to quickly isolate the cause and formulate a way to resolve it.

Tips for the right metrics

In addition to ensuring that the metrics you choose meet the requirements outlined by the four A’s, there are a couple more guidelines which will ensure you’re making the most of your metrics system.

Monitor rates, not totals

First, avoid the temptation to monitor total counts or run total metrics. Total counts are a big part of the vanity metric problem. What you want to monitor instead is a specific rate over time. Calculating the number of transactions being performed over a distinct period will be significantly more useful to developers and stakeholders alike. You can even measure the rate of change over time to highlight whether your system is improving or deteriorating.

Visibility is important

Second, make your metrics visible. The right collection of metrics should highlight both successes and failures. Ensure your metrics are displayed prominently to the owners of the metrics and other key stakeholders within the organization. Emphasize that a downturn in the metrics represents an opportunity for improvement, whereas good metrics should spur on the process of continuous improvement forward.

Get good tools and expert help

Finally, if you need help identifying, analyzing and displaying metrics related to your system, seek out the help of an expert, and investigate tools built specifically to help you expose useful and actionable metrics. Retrace by Stackify is one such tool, and you can try it free for 14 days. You can sign up for the trial or request a demo.

Author

Alexandra

More articles by Alexandra