I had a busy week hopping between the OMS Conference in San Diego (where I co-presented with Ali Behnam of Tealium) and the WAA Symposium in Austin. In San Diego, Ali and I crafted a presentation built around “Common Myths of Web analytics”. This is pure Web analytics and since we’ve both been doing Web measurement since the dawn of the internet, it was great to trade experiences and talk about some of the “best-practices” (including reliance on best-practices) that aren’t really all that wise. We both enjoyed doing it so much we’re going to try and get a Webinar scheduled on the same material sometime in the next couple of months.
The WAA Austin presentation was a different kettle of fish and it’s the one I’m going to focus on here because it comes closer to my recent themes. The essence of the Austin presentation was simple: digital optimization in today’s enterprise is split between classic site analytics, A/B and MV Testing, and personalization. Each of these three paths to digital optimization have unique strengths and weaknesses; what’s more, they are very complementary. Each can facilitate the more effective use of the others. But while these three paths to optimization should be complementary, they are mostly deployed in a siloed manner that keeps each discipline working sub-optimally.
My goal in the presentation was to show how each discipline potentially fits into a larger Digital Optimization strategy. That was the first half of the presentation (yes, I should have stopped there since that’s plenty of material). In the second half, I showed how specific analytic techniques that we use (Behavioral Use Cases, 2-Tiered Segmentation, and Functionalism) help bridge the gaps between these disciplines.
This diagram is really at the heart of the whole presentation. It shows the good and bad of each discipline.
Classic Site Optimization (Web analytics) is great for identifying site problems, finding and defining customer segments, and measuring “deep” metrics like Lifetime Value or Engagement. On the other hand, classic Web analytics has great difficulty in controlling for self-selection; this makes it rather poor at choosing between creative alternatives. It’s also pretty hard to do and takes significant expertise.
Site Testing, on the other hand, provides truly definitive answers between creative versions. It eliminates most self-selection and control issues (at least if done well), it’s pretty easy to do, and it generates consistent value. On the other hand, Site Testing methods have little or no ability to identify promising test areas, identify good customer segmentations or support “deep” success metrics. Site testing also can be difficult to scale. Every experiment needs new creative – making the use of micro-segmentation almost impossible.
Personalization is yet another, and very different, beast. Like Testing, it can eliminate most self-selection and control problems. Unlike Testing, it can scale and take advantage of micro-segmentation. It’s generally limited to problems that don’t require deep creative and, depending on the approach, it either lacks segmentation or provides a black-box solution to segmentation.
On even a cursory reading, it should be apparent that the strengths of classic Web analytics are a perfect complement to Site Testing and Personalization. It should also be obvious that there is a much bigger role for Site Testing in the development of Personalization than is commonly realized.
Here’s the way I showed it in my presentation:
You should be using Web analytics to figure out what to test and to generate your segmentation (as I’ve written before, all tests should be segmented). Classic Web Site analytics is also the best way to develop and sustain the “deep” metrics you may need to understand true lift in both Testing and Targeting.
You can get around the scale limitation of Testing by focusing on tests of personalization rules. This is a completely different way to think about Testing, but it makes it possible to develop powerful “white-box” approaches to targeting and have a truly logical, integrated path to Targeting and personalization.
This isn’t to say that every aspect of Web analytics, Testing or Personalization can or should be integrated. Web analytics has many functions that are simply unrelated to testing. If you need to measure before and after site re-designs, measure the absolute performance of a tool (such as internal search), measure campaign performance, and much more, Web analytics on its own is fine.
Likewise, Testing of Landing Pages often has no necessary or even desirable path or integration to Targeting and Personalization. There are many creative tests that don’t lead directly to Personalization or micro-segmentation.
Still, to really get full value from each of these components, you need the type of integrated approach I’ve outlined.
As we at Semphonic see it, Behavioral Use-Case Analysis provides the perfect bridge between classic Web analytics and testing. It creates a digital segmentation, a framework for testing, and a set of likely test targets. It’s the perfect way to answer the “where to test” and “how to segment” questions.
Functionalism then provides a more detailed drill-down into the individual performance of site components. This is an excellent method for answering the “what to test” question within the context of a Use-Case and Segment.
Our Two-Tiered Segmentation provides a ready-made integration of traditional visitor segmentation and digital visit types. This is just what’s needed to identify personalization options worth testing.
It’s one thing to say that site analytics, testing and personalization should be closely integrated. It’s another to show how they can be. In Austin, I tried to do both.
Drop me a line if you’re interested in the complete presentation or finding out more about how we do this!
