Notes from the X Change Berlin:
Two years ago, in a Huddle on Site Testing at the X Change in Monterey, I found myself getting quite frustrated by the way most organizations run their test program. There were a number of organizations in that Huddle doing aggressive A/B and multivariate testing. Many evinced a strong commitment to testing every change they made on the site. But it wasn’t the amount of tests or the commitment to testing that I thought problematic. That was great to hear. The problem was that in almost every case, there was no method behind deciding what to test. People identified potential tests and tried them. In the vast majority of cases, these tests were site-wide and lacked any initial segmentation. The general idea was that you’d run a test, then identify which populations it worked best for. In organizations where testing had really taken off, lots of people were coming up with test ideas and generally there was some kind of “Testing” Committee to prioritize and approve actual tests. Again, there’s nothing structurally wrong with any of this. But where’s the link to analytics? If you’re analytics department isn’t driving your testing, what’s your analytics department for? Worse, as I pointed out at the time, without initial segmentation, coming up with effective, targeted creative is nearly impossible. The idea of site-wide tests followed by segmentation is completely backward.
At the time, I wrote a series of posts showing how our Use-Case Analysis could bridge analytics into testing and provide a more formal methodology for constructing a test program. Since then, we’ve elaborated on that approach to building testing programs based on our Two-Tiered Segmentation methods (which are really just an extension of Use-Case Analysis) . There are many things I like about this approach: it provides a comprehensive testing approach to every site function and user-type, it provides a very rich framework for developing targeted creative, it allows for the effective integration of online surveys and voice of customer data in designing test alternatives, and it allows for multiple simultaneous tests without danger of overlap. By bridging the gap between analytics and testing, it creates a unified approach to the two most important aspects of digital measurement.
So in signing up for John Hogan’s Segmented Testing Huddle, I certainly had a pretty strong set of pre-conceived notions about how testing and segmentation fit together and I was mostly interested to see how organizations in Europe were handling the two. Certainly I was encouraged by the very existence of the Huddle which promised a deep interest in Segmentation as a part of testing.
I wasn’t disappointed, but I was surprised. The way Virgin Media handles their segmentation is rather different than our Two-Tiered Segmentation approach. They rely heavily on Persona-based segmentation. Indeed, it was the development of their persona-based segmentations that appears to have really kick-started their program and driven widespread marketing adoption.
I found that extremely interesting and worth pondering.
We’ve been doing persona-based Segmentation for many years. Indeed, it was the main technique we adapted from days in credit-card marketing to the Web. Persona-based segmentations (typically built using either Cluster Analysis or Neural Nets), are designed to aggregate many separate variables into a multi-dimensional map in which visitors tend to cluster. This sounds very complicated (and mathematically it is – though fortunately other statisticians and programmers have done all the hard work building the tools), but the end result is designed to deliver a much simpler view of digital behavior to marketers. I often describe this type of segmentation as providing the “flesh” on the bones of digital measurement.
Here’s an example I sometimes show from the travel industry of one of our persona-based segmentations:
In this case, we described sixteen different persona-based segments to the business (entirely created from digital behavior – though they can and often should integrate offline data and demographics as well) organized along several “uber” dimensions: visitor’s focus on specific destinations, their focus on specific travel-times, their tendency to do trip research, and their tendency to book.
In the screenshot above, one of the personas “Snake Eyes” (gamblers slang for rolling two ones) that focused almost exclusively on trips to Las Vegas is expanded in the spreadsheet to the right showing how this group can be described to the business. The variables we used don’t look much like traditional Web analytics: Hotel Propensity, Package Propensity, Flexible about Dates (well above Index Average for Snake Eyes), and Flexible about Destinations (way below Index Average for this group of people) are typical examples.
The beauty of this type of segmentation is that it provides a much easier grip on the data than almost any other method of presenting Web analytics. Marketers can really use this type of data – and you can see how it lends itself to testing. Targeting a family-friendly vacation offer is as simple as finding the segments that are strongest in family-oriented travel. Since that’s a variable in the analysis and a dimension along which every segment is ranked, the marketer need to do nothing more than scan a list of top segments by that dimension to create a truly powerful targeting mechanism.
So how does this approach differ from the Two-Tiered Segmentation? The two aren’t mutually exclusive at all, but they aren’t necessarily the same thing either. A persona-based segmentation is a visitor-based clustering. It lacks the second, visit type dimension that we use.
On the other hand, it could easily be the Visitor-level segmentation of a 2-tier scheme. That’s something I’ve always realized, but never, perhaps, thought about enough or pushed hard enough for. It’s a great technique for creating a really rich Visitor-based segmentation for ANY type of client regardless of whether or not visitors are known or anonymous. In situations where a large percentage of behavior is anonymous, it may be the only really effective technique for fully describing a visitor.
With persona-based segmentation, you get rich, powerful, data-driven segments that lend themselves to effective approaches to testing and suggest to marketers rich ways to use the data.
By adding our visit-based segmentation dimension to it, you refine your ability to run segmented tests on every aspect of the site including ones (like Customer Support) that may not be particularly suggestive to marketers. You also create a method for drilling down into site areas to understand how effective they are (and where they fail) for each type of visitor. This makes the tie between analytics and testing that much stronger.
The effectiveness of persona-based segmentation in getting Marketers to understand and use the data is something I’ve probably underestimated. It makes me think that more extensive and aggressive integration of personas into our approach would help marketers leverage the system more effectively and generate more and better tests.
