Something I’ve learned from years of attending Conferences on Web analytics, it’s that if there’s one topic you won’t hear much about, it’s how to do analysis. That isn’t necessarily a knock on Conferences – it’s a pretty hard topic to handle in a presentation. Kind of like going to a Conference on baseball – you probably won’t see a presentation on how to swing a bat! But in X Change’s conversational format, you can tackle some topics that wouldn’t necessarily work in a more formal presentation.
In “Getting the Data to Tell It’s Secrets”, I was filling in for Matt Fryer who ran a very successful session on this topic last year. My plan for the session was fairly straightforward – I wanted the group to work through the major steps in analysis starting with how analysts choose topics for research and the extent to which open-ended data exploration is possible and effective. Next, I wanted to see how and whether you can regularize the process of analysis. Are there methods for making analysts more productive in exploring data and answering questions. I also wanted to discuss exactly how analysis works – what are the core techniques we use when actually deep-dive into a problem. I knew this part of the discussion might be problematic but I think it’s critically important to reflect on. Next, I wanted to talk about the process of putting an analysis together and presenting it to stakeholders. Socialization is almost as important as actual analysis in the effective use of information. Finally, I hoped to talk about what to do when an analysis goes right and you find something big and, conversely, how to handle it when you screw up an analysis. If you’ve never messed up an analysis, pulled the wrong data, used the wrong math, or drawn the wrong conclusion it can only be because you’ve never done an analysis. Mistakes are inevitable, so handling them right is always on the agenda.
Any researcher will tell you that the single most important step in the process is deciding what question to answer. How do you know where to start?
Not surprisingly, our group at X Change had a wide array of answers. Some analysts seem to prefer a customer-driven approach. Start with Voice of Customer (VoC) data – whether from feedback systems like OpinionLab or online survey solutions. Look for places where customers express dissatisfaction or give the site poor ratings and then focus analysis on broader behaviors to see if the problems are widespread or unique. I’ve certainly seen this technique used very effectively and I’m confident that it’s a part of any complete answer on how to steer a research programme. I’m also fairly certain that it’s not the whole enchilada. There are too many critical business questions that simply aren’t going to be surfaced by VoC. At a fundamental level, customers don’t care whether you’re Website is accomplishing your business goals. They don’t care if you’re persuading them. They don’t care if you’re strengthening your brand. They don’t care if you’re maximizing your AOS or putting the right mix of merchandising levers on a page. It’s critical to find ways to surface research questions that are business driven not just customer driven.
Another common driver of analysis is variation in reporting. Since reporting mostly focuses on business drivers (traffic, conversion, margin, revenue), this seems, intuitively, like a nice counterpart to customer-driven research. When an important business metric changes significantly, it almost always requires an analysis to figure out why. So reporting is a natural and important table-set for analysis.
Even combining these two methods, however, I think you’re well short of a complete research programme. Both customer and report based methods are event-driven. They’re great methods for surfacing unforeseen and important questions. What they won’t necessarily do is find and focus attention on big business problems that aren’t captured by key metrics or that exist within the current status quo.
One of the unique pieces we like to create when we do digital strategy engagements at Semphonic is a data science roadmap. The idea is to identify key analysis opportunities to create a formal plan for the analytics team. To create the plan, we don’t use ANY event driven techniques. Instead, the plan evolves in three stages.
First, we create a high-level model of the client’s business. This isn’t some fancy predictive model or anything – it’s just a conceptual model of how the business works. In the second step, we identify any gaps in understanding that make it hard for the business to measure success or optimize the process. These gaps become target areas for analysis. In the third step, we match these target areas to an existing library of internal analysis methods to see if we have practical methods for answering the research problem and prioritize methods based on potential business impact.
I’d describe this approach to creating a research programme as methodological. It forces the analyst to start with the business and guides research toward areas that are of highest value. It’s a great way to map out a high-level programme. It also ties in very well with our approach to reporting. When building segmented reporting systems, we step clients through a formal process of audience identification, visit intent segmentation, matching business goals to visit types, and then defining success metrics for each business goal with respect to the audience and visit type. Sometimes, there are straightforward metrics for making that tie. Often, there aren’t. If you decide that the measure of success for a Facebook Campaign is the percentage of “engaged” Facebook Fans you acquire within your target audience, you’re faced with a research problem – how do you measure engagement and audience? If you’re doing your report definition right, you should expect additions to your research programme.
Of course, the methodological approach has its own limitations. If you build a research programme but don’t pay attention to customer and report driven questions, you’ve lost a critical element of responsiveness. These first two methods are much better at surfacing NEW and unforeseeable business problems.
Wrapping up this part of the Huddle, we explored the role of data visualization and unguided exploration in generating research questions. Can an analyst really just sit down with a tool and explore the data to find interesting points or important problems?
I’ll admit to deep skepticism on this point. It’s never really worked for me. However, there was a pretty strong consensus in our Huddle that, in fact, it does work for some analysts. Given the right tool (something like Spotfire or Tableau seemed to be the consensus), unguided data exploration was forcibly supported as a valid method of generating interesting avenues of exploration.
I’m going to be a recidivist here and claim (with no real evidence) that what’s probably going on here is sub-conscious application by an experienced analyst of something like our methodological approach.
Whatever you believe about that, what’s more important from a practical standpoint is that unguided exploration – with the right tool – can and does work. Perhaps my attitudes here are colored by the fact that we’re a consulting company. Nobody pays us (rightly in my view) to do unguided exploration of the data. As an enterprise, however, it probably makes sense to create enough space in your research programme to allow your analysts some unguided time to explore the data.
Which brings us to the next stage in getting the data to tell its secrets – actually doing an analysis. In my next post, I’m going to digress briefly to introduce a new Whitepaper that Semphonic created for IBM on Choosing a Big Data Technology Stack. This is one of the hottest topics in our practice and it’s a piece I’m very happy with. Then I’ll return with more on what I think was a fascinating discussion of how to tackle an analysis project.
[And since I’m on the topic of Web analytics methodologies, if you’re in the Philadelphia area, please join me at the DAA Philadelphia Symposium on October 18th. My topic is on the application of Statistical Analysis Techniques to Digital Analytics and why it’s much more challenging than people think. This is the subject of yet another upcoming Whitepaper and I think it’s groundbreaking work. If you’re in the Midwest (I’m an Indiana boy), I’m also going to be in Indianapolis for ExactTarget’s Connections 2012 Conference on the 17th. Michael J. Fox and David Blaine are both Keynoting there – how cool is that!]
