CIO.com – June 27, 2022

By: John Edwards

Virtually every CIO uses metrics and key performance indicators (KPIs) to gauge the performance of their IT teams and strategies, but not always correctly. Here’s a look at what you’re doing wrong.

Mark Twain famously remarked that there are three kinds of lies: lies, damned lies, and statistics. Today, many CIOs feel the same way about metrics.

Metrics are only as good as their source. “Too often, technology companies pay consulting or analyst firms to create metrics based on the best characteristics of their offerings,” says Judith Hurwitz, CEO of Hurwitz Strategies, an emerging technology consulting firm. “Therefore, CIOs must be cautious about taking metrics at face value [and] leaders need to understand the data behind the metrics.”

Metrics interpretation is essentially a numbers game, and as with any numbers game, it’s possible to win or lose. Here are seven ways IT leaders are often misled by key performance indicators (KPIs) and other critical business and IT metrics.

1. Not considering the source

When studying a metric, it’s important to know who created it and the data source. Results may be based on a survey, for instance. If so, ask how many people were surveyed and the roles they played in their respective organizations. Check as well to see whether the metrics are based on a well-proven methodology. “It’s important to understand the research and data behind the metrics,” Hurwitz says.

Also consider the metric’s purpose. Will it be used as a planning tool? If so, will it help determine a business strategy, a technology selection, or some other need? “Metrics are only one tool for decision-making,” Hurwitz notes. “Therefore, approach metrics with skepticism.”

2. Failing to collaborate with front-line personnel

By now, most enterprises have reached data maturity. “If your company has data, you’re definitely leveraging it and trying to use insights from analytics to drive positive business outcomes,” says John Loury, president and CEO of Cause + Effect Strategy, a business intelligence consulting firm. “It’s 2022, we’re past the age of DRIP — data rich, insight poor.”

Loury believes that most organizations don’t dig deep enough when communicating with the front-line business personnel who will ultimately use collected metrics to make decisions and drive actions. Before building analytics, he recommends collecting business requirements from all involved parties. This means distilling metrics down to the data points most relevant to drive outcomes, Loury notes. “Prioritize what most directly impacts the business decision your user is trying to make.”

Loury advises building and honing communication skills to convey metrics-based insights to team members. “Modern CIOs and analytics leaders need to be adept at pulling together the key metrics that will drive the most impact for a team and presenting them in a way that makes sense to the user and will help guide their behavior,” he says.

Loury adds that it’s also time for CIOs to task their teams with truly understanding their users and building them tailored, effective analytics solutions. “The days of data leaders and their teams scrambling to build something — anything — and ship it to business teams are behind us,” he explains. “We’re living with the results of those days, where teams are inundated with wall-to-wall dashboards that tell them everything and nothing.”

3. Overlooking the importance of ownership, involvement, and balance

Metrics present an excellent opportunity for ownership and staff involvement, as well as continuous improvement and process control. “The key to correctly interpreting metrics is to engage your whole team and use the metrics to collectively improve processes,” says Paul Gelter, coordinator of CIO services at business and technology consulting firm Centric Consulting.

When evaluating metrics, Gelter believes it’s essential to strike a balance between cost, quality, and service. Cost metrics, for example, could be tracked in completed tickets per individual, yet ticket quality could be degraded by rework/repeated tickets. “Service could then be impacted by the response time, backlog, and uptime,” he notes. It’s all about obtaining an optimal balance.

4. Chasing the wrong numbers

Time really is money, so don’t squander precious hours scrutinizing irrelevant metrics. Clearly identify all goals before deciding which metrics to study. In most cases, metrics that don’t support or reflect future decision options are unnecessary and, worse yet, distracting and time-wasting.

Once the goal has been fully defined, allocate sufficient time to understanding the factors that cause individual metrics to fluctuate, suggests Alex Levin, co-founder of technology and design studio L+R. Next, investigate how individual metrics are tied to one another, and what’s likely to happen during different stages within an initiative’s or project’s lifecycle that might directly affect the KPIs being tracked.

Meanwhile, don’t waste staff time by concealing or hoarding conclusions. Levin advises sharing study results with your team, ensuring that each individual can use metric-driven insights to improve performance and/or outcomes.