Thoughts from the Berlin X Change
A good framework for digital analytics is a core ingredient in a good enterprise measurement program. A framework provides a common language for the organization to talk and think about digital. It provides the foundation for enterprise reporting and it is absolutely essential in achieving some degree of comparability across different digital (and non-digital) channels. Comparability is surprisingly important. If you want to be able to create enterprise-wide dashboards, you’ve got to be able to show data from multiple channels (Web, Mobile, Social, Branch, Call, etc.) side-by-side in a way that makes them somehow comparable.
Two-Tiered Segmentation as a Framework
Over the last four or five years I’ve put a lot of time and thought into trying to create a really good framework for digital analytics. That work has led us to create one of the foundational elements of our practice – Two-Tiered Segmentation. The idea is simple enough. Metrics, to be meaningful, need to be put in the context of a segmentation: a segmentation that covers who is being measured and what they were trying to accomplish. Two-Tiered segmentation provides that context. The first tier captures the who. This is classic visitor segmentation. Understanding who someone is and what their relationship is to you (Customer/Prospect, High-Value or Low-Value, Male or Female). But in digital measurement, this traditional segmentation is only half the story. Maybe less than half the story. Because what really drives measurement in digital isn’t who somebody is, it’s what they were trying to accomplish. That’s why if you’re going to pick one tier to focus on in digital, it’s the second tier that really matters. There is nothing more important to our measurement foundation than figuring out the visitor’s intent.
So while I thought I had a pretty good handle on this particular topic, I wasn’t really sure what to expect from a discussion of a digital analytics framework and, as often happens at X Change, it was the unexpected that I got.
Generalizing Metrics Across Mobile Apps
The part of the discussion that both surprised and intrigued me most concerned the challenge of generalizing an analytics framework across different digital channels – particularly mobile apps. On the surface, two-tiered segmentation is unaffected by switching from Web to Mobile. It’s part of the strength of the framework that it generalizes so well. Two-Tiered segmentation doesn’t just work well for the Web. It works for Mobile Apps, it works for Social Community posts, it works for calls to the Call-Center. It even works for visits to the ATM.
That ability to Two-Tiered Segmentation to generalize is a profound demonstration of its power. It’s hard to create a truly general framework and the more problems a framework can encompass, the better is the evidence for its usefulness.
But prior to X Change, I hadn’t deeply considered the metrics inside that framework. Metrics like visits and even page views have survived in our framework – contextualized by the Two-Tiered segmentation.
Does that really make sense?
People outside of Web analytics have long complained about the opacity of our metrics. That very opacity is troublesome. If you can’t understand metrics from digital, it may well be that it’s because the metrics aren’t very generalizable. Uniques is an obvious example. Can uniques be used in any other channel? It seems unlikely.
But visit is a much simpler and more generalizable metric.
Visits to an ATM certainly make sense. Visits to a Branch fits perfectly.
Visits to the Call-Center? It’s not quite the way we’d talk, but it fits reasonably well.
But how about Visits to a Mobile App? Visits to a Twitter Feed? Visits to Pandora content? Visits to a TV Channel?
Visits doesn’t seem to fit that well and these channels and here’s why.
In the Web context, we think of visit as a container with a time dimension, an intent (what the visitor wanted to accomplish), and an outcome (success/failure/etc.).
Where the visit concept fits, you can see those elements. A visit to an ATM has a time dimension, it has an intent (usually but not always to withdraw money), and it has a success (often time to complete transaction). Perfect.
Streaming mediums, like Twitter, break all three paradigms.
Mobile Apps are somewhere in between. It’s true that when you open a Mobile App there is a time dimension, a success, and an outcome. But what’s tricky is how those dimensions play out. With Mobile Applications, the time dimension is hard to control. Of course, you might reasonably object that the time dimension on traditional Websites is hardly pure. Unlike an ATM visit or Call, time on the Website is rather fuzzy, especially in a tabbed browser world. Since we don’t actually know when the user stopped looking at content, we measure visits by creating an arbitrary time cut-off of thirty minutes without a server request.
Consider these two scenarios.
Scenario A
I visit my brokerage site and do a search on IBM. I read two pages of research. I then do nothing for twenty-five minutes. My next click is to the home page where I then initiate a trade, selling my shares of Chevron.
Scenario B
I visit my brokerage site and do a search on IBM. I read two pages of research. I then create a new tab and do a series of Google searches on IBM and IBM related matters. After thirty-one minutes, I return to the IBM tab, view two more pages of research, and then execute a buy order on IBM.
In the world of Web Analytics, Scenario A generates a single visit and a rather confused visit intent. Scenario B generates two visits, one of which is a research visit and the other of which is a buying visit.
There’s no logic to this. It’s obvious we’re simply the victim of an arbitrary time cut-off. Unfortunately, removing the time cut-off isn’t an option. Not on the Web and certainly not on Mobile. I suspect that my wife has Apps running on her iPhone from when she last upgraded or, at the very least, since she last traveled. People don’t turn their phones off and many people never terminate their apps.
Nor is there some magically correct time cut-off. Any time we choose will necessarily be arbitrary and there is no inherent reason to believe that any other number is better than thirty minutes or that there is any time that would be optimal across multiple applications or Websites.
Working this through didn’t necessarily make be believe that Mobile Apps are fundamentally different than Fixed Web. Instead, it made me think that perhaps our concept of a Visit (as a container for Time, Intent, and Success) is wrong on the Web.
Given the challenges around time, it might make more sense to abandon the idea of a Visit as a container for a single time/intent/outcome triplet. To people who haven’t adopted our Two-Tiered Segmentation, that conclusion might warrant no more than a shrug. Other measurement frameworks never committed the visit to be a container of intent and outcome and when a visit isn’t so defined, the arbitrary nature of the time cut-off may not seem particularly important.
Within our framework, however, it means re-thinking metrics where Intent not Time becomes the defining dimension for splitting activities. I’ve proposed the concept of Unit of Work as a way to think about Mobile Apps. If I open Kayak and search for flight, that’s a unit of work. If I then search for a hotel, it’s another unit of work. Same deal with Opentable. If I book a restaurant for Thursday, that’s a Unit of Work. Book another for Friday, that’s a different unit.
With Units of Work, it’s not clear we need the idea of a visit at all. Here are three scenarios:
Scenario A
Start App. Search on Trips from SFO to LA on Dec. 21st. Leave for 10 minutes. Refresh my search and Filter on Morning Non-Stops. Leave for 31 minutes. Refresh my search and click through to an airline site.
Scenario B
Start App. Search on Trips from SFO to LA on Dec. 21st. Filter on Morning Non-Stops. Click through to airline site.
Scenario C
Start App. Search on Trips from SFO to LA on Dec. 21st. Go back to main menu. Search for Hotels in Denver on December 4th. Click through on a Hotel.
In what way does the concept of a visit help with measurement in these Scenarios? I don’t think it does. It seems clearer to me to measure by Unit of Work. I start a Search on LA. for Dec. 21st. That Unit of Work starts. When I do something different, that Unit of Work stops or is suspended. For a Unit of Work, I may care about how long it took from start to finish and how much App foreground time was consumed. But why would I separate Scenario A into 2 Visits? What measurement value do I get?
This may not seem terribly radical when it comes to Mobile Apps. Lots of people might expect that Visits isn’t that interesting when it comes to Mobile Apps. If we are trying to generalize our framework, though, this implies that perhaps Visits is a bad metric on the Fixed Web as well.
Given our Two-Tiered segmentation approach (which effectively generates Units of Work), we could just as easily remove the concept of the Visit from the Fixed Web as from Mobile Apps.
I think that’s intriguing.
A framework like this:
Person
Unit of Work (intent)
Time
Outcome
Generalizes in every channel that visit covered (Branch, ATM, Web), makes the framework extensible across multi-purpose touches in all of these channels, and gets rid of the arbitrary time borderline that’s used to create visits and which feels particularly unnatural in Mobile Apps. It also creates a whole new set of potentially interesting metrics (such as Total Task Elapsed Time and Total Task Working Time) that are hard to elucidate in a world of artificially bounded visits.
Conclusion
The idea of using generalization as a test of metrics crystallized for me in the X Change discussion. I’m not going to argue that there are no channel specific metrics. Certainly there are. But there is a strong imperative in a measurement foundation to create a set of metrics that are maximally generalizable so that multiple channels can be treated identically and compared fairly.
Visit fails the test. Not spectacularly. Not like page views (which fails spectacularly). Visits is powerfully generalizable across multiple channels. But in its current time-bounded Web form, it still seems less than ideal.
Of course, if you’re using a Web analytics tool as your primary source of truth, you’ve got no dog in the hunt. You’re stuck with visits and page views as the metrics. But as with so many other digital measurement problems, reporting from a warehouse leaves you free to fundamentally change your measurement framework. If that’s the case and you’re faced with multiple touchpoints in digital or non-digital channels, it’s worth considering which metrics should survive.
[If you’re intrigued by measurement discussions that range from the highly practical to the deeply theoretical (like this), then X Change is the right Conference for you. And it’s high-time to register for the September X Change at the Ritz Carlton in Laguna Niguel. If the sunshine doesn’t blow your mind, the metrics will!]
