One really aspect of my recent series of posts around social media is that they’ve generated quite a few thoughtful and useful comments. Like every other blogger, I love comments. Next to someone ringing us up and asking Semphonic for a proposal because they “follow the blog”, it’s the best evidence I have that people are actually engaged. As an extra bonus, really good comments are grist for the mill – it’s much easier to write a conversation than a monologue (though perhaps less easy to manage). And these issues and questions around methodology in social media measurement interest me deeply, so it’s pleasing that my interest is not isolated.
In particular, I wanted to address this three part question/comment from Jason that I got this week. Here’s the comment:
I liked your post very much. A few points, if you could further elaborate: 1. “most social chatter has absolutely nothing to do with PR effectiveness”. How would you define PR effectiveness? Is it computable according to your definition? 2. “Those influencers have nothing to do with measuring traditional consumer awareness”. What about traditional consumer awareness. I am interested in the computational aspect of the term. 3. Don’t you believe there should be a holistic approach, in any case we are talking about a unified brand that we try to quantify. Thank you in advance.
Some of this presages issues that I hoped to talk about in more depth in the future, but I’ll take at least a first cut at them here. I’m going to start at the back of these comments and take one more whack at the concept of Total Mention Counts and measurement samples as a way of thinking about a holistic approach to brand measurement. I’m not really sure that this is what Jason has in mind, but I feel that while I’ve taken several whacks at the “Total Mentions” problem, I have yet to hit a clear home run.
Let me start by re-visiting the difference between traditional opinion research and PR brand monitoring. In traditional opinion research, the goal was to capture and understand consumer sentiment and attitudes. Questions like “how many people know our brand” and “what do people think about our brand” and “what would make people buy more of our brand” are all answerable with primary research. They are answerable not because you can talk to every consumer, but because you can sample a small but representative set of consumers and, from that sample, extrapolate with some confidence to the total population.
Building a representative sample for traditional research may sound simple but in fact, it isn’t. If you call people during the day, for example, you’re likely to reach mostly the unemployed, the retired or housewives. That’s likely not a representative sample for most businesses of their customer base. It may be that women are more likely to take a survey than men. Or that seniors are more likely to give you their time than young adults. There are a thousand different factors that can influence a sample and most commercial (as opposed to academic) research doesn’t attempt to do more than control for the most obvious biases.
But there is one bias that nearly all survey samples control for – they exclude anyone with any expertise in the industry. If you’ve ever been called (and I’m sure you have) for a survey, you’ll remember that there’s almost always a few qualifying questions. If the survey is for a health care provider, they want to make sure you don’t work for the client or their competitors. Likely, they want to make sure you don’t work in the industry at all!
Why? Because if you are in the industry, you know too much, you have too much bias, and your views are almost by definition different from the general consumer population. Nor are your opinions likely influencable by the normal channels of marketing. So you’re answers are going to distort the research, not make it better.
Keep this in mind – it isn’t just that traditional opinion research isn’t designed to study influencers, it’s specifically constructed to AVOID their opinions.
PR monitoring services, on the other hand, focus exclusively on one subset of people that are excluded from opinion research. PR monitoring services want to know what they influencers are saying. They want to know about mentions in the NY Times. They want to know whether you’re mentioned at expert Conferences. They don’t care at all about Joe Consumer’s opinion of your service or drivers of choice. That stuff simply doesn’t matter when it comes to PR.
This is all clear, simple and straightforward when it comes to traditional research. No PR person would ever have thought to use random digit dialing to figure out whether he’s influencing key people. No survey designer would ever have bought of list of journalists and experts to target for his consumer opinion research (though I can see some merit in the idea if you’re planning a PR strategy).
With Social Media, there’s a misleading sense that these lines are erased. Blurred, perhaps is true, but they are by no means erased. Everybody is most certainly not an influencer. With blogging and twitter, there has been a vast increase in the amount of user generated content (UGC). But UGC is still readily bucketable into distinct types and categories that (should) mirror these previous distinctions.
Here’s a personal example. If you’re a Web analytics tool vendor then Gary Angel is an influencer. You’d be wrong to survey me if you’re goal is to understand what would make an enterprise buyer purchase your product. I’m not an enterprise buyer even though, heaven knows, I talk endlessly about Web analytics. In fact, my profligacy of output on the topic is generally hard evidence that I’m not a consumer – and this topic and posting density is a metric we use when we create automated samples of consumers and influencers. On the other hand, if you’re Kayak.com or ESPN and your interested in consumer attitudes about travel or sports, I’m perfect. I’m not an influencer in those fields though I most certainly have a blog.
Or consider the Web analytics twittersphere. The vast majority of #measure tweets are generated by tiny number of devotees (nearly all with a strong interest in self or company promotion) who generate a vast amount of chatter amongst themselves. Certainly my own somewhat infrequent posts on Twitter fall in this category. In what sense is this interesting if you’re trying to understand what an enterprise buyer of analytics might care about? These are the LAST people you should be thinking about – far from measuring their attitudes you need to be going out of your way to exclude them from your sample. They are as unrepresentative a bunch as you could possibly imagine.
So yes, Social Media has created many more niche influencers. However, most of these influencers are topic specific. Within that topic, these influencers need to be treated as if they were traditional media or company spokespeople not consumers. That means you need to EXCLUDE their opinions if your goal is to understand your customers. Fortunately, that’s quite possible even in an automated fashion. By measuring their output and topic focus, you can nearly always identify a professional signature even when you don’t know who they are.
So let me sum up the argument here one last time (I hope). If your goal is to understand your consumers, then you need to be monitoring their conversation. By definition, experts and influencers ARE NOT your consumers. Their opinions are different from your true consumer population and they are vastly overrepresented in the opinion-sphere of social media. So unless you specifically exclude them from your consumer research, you aren’t measuring what you think you’re measuring and you aren’t measuring what you want to be measuring accurately.
The converse is equally true. Not everyone is an influencer worthy of PR attention. Kayak.com doesn’t need to send me their press releases just because I have a blog. It won’t help and I won’t read it. And if they are measuring the effectiveness of their PR, they don’t need to be sampling my blog.
The simple necessity to create a sample that is representative of your research target is NOT removed by Social Media. Far from it. You need to devote even more attention to this than you might in traditional research precisely because the bounds are blurred. When you random digit dial, the chances of influencing your research by oversampling influencers is quite small. With social media, this is most definitely NOT the case.
So Jason, I don’t know if I’ve really answered your last question because I’m not sure I interpreted it right, but I guess I’d sum it up this way. I’m most definitely not against a holistic view of the brand. At Semphonic, we routinely try to create just that in our Social Media reporting. But a holistic view of the brand doesn’t come from counting everything in a single bucket. It comes from careful sampling of different audiences each of which is important to a holistic view of the brand, but none of which should simply be added together. Creating a unified view of a brand might well encompass a report that showed how strong the brand was with influencers AND how strong the brand was with consumers, it would NEVER mix the two samples when generating those answers or even mix the two answers.
I hope that makes sense. This is probably my third or fourth cut at this point and I hope, at last, that I’ve gotten it right!
Now I’m going to skip back and tackle the first question (that middle question may just have to wait). How do you measure PR effectiveness and is it computable? The answer to this last part is yes – it’s computable if you have a good sample of PR influencers and that having a good sample of PR influencers is the critical ingredient in measuring PR effectiveness.
I think of PR effectiveness as quite distinct from viral marketing in that it targets a specific group of people (Influencers) and has two readily measurable goals: getting influencers to talk more about your topics and to talk more and more favorably about your brand.
A PR sample will surely include all traditional media sources as well as careful identification of influence-based online sources. This doesn’t include all blogs or all tweets – but only those generated by those who qualify as topic influencers.
I’ve shown these before, but I’m going to trot them out again. Here’s a PR report we created for a client that shows how topic focused potential influencers are:
The gist of this report is to help show which online sources are important for a given topic. The NY Times is probably influential across almost any topic. Online sources, however, are much more likely to be topically influential. If you’re interest is in ASP, then Blogs.MSDN.Com is authoritative. If you’re interested in Ajax, not so much.
We’ve refined this work recently into a generalized segmentation approach to social reporting. We identify individuals as influencers based on their posting frequency and topic specificity. This allows us to sample these influencers separately without prior identification (the list above was based on a pre-selected list). Equally important, the converse sample allows us to create a much cleaner set of true consumers.
Once you’ve identified a set of influencers, you can now track your mindshare and message share with them over time. Here’s an example:
The affinity graph is showing how much each source overlaps our client’s core topics. The Us vs. Them shows how much that source discusses our clients vs. their competitors. The shift measures how the Us vs. Them has trended in the past month. If you’re PR is working, you should be seeing shift – either with specific targets (as in this report) or with the overall set of influencers (depending on your tactics).
I think this is a perfect example of PR effectiveness measurement. Your goal with PR is to shift influencers to talk more about your topics and your brand. If you can effectively identify your target population, you have every opportunity to do just that.
I hope all this makes clear that while I have deep doubts about certain aspects of Social Media measurement I am confident that it has considerable value when used properly. Measurement of PR effectiveness is an entirely appropriate use of Social Media and an excellent example of a case where social media lets us measure a task far more effectively than was ever possible in the past!
I still have that post on survey research in my plans – but I don’t think I’ll be trotting it out for Christmas! Perhaps that will be the post to kick-off the New Years with…
Since I may not write again before the holidays, I’ll wish everyone a happy holiday right now.
From what was, for me, a troubled beginning, this has been a much better span of months than I had any right to expect. Business has been wonderful and interesting. Building Semphonic is more enjoyable and fulfilling now than at any time in the past 14 years. I finally, after many years, left the City (as we title San Francisco) and made the short hop across the Golden Gate to Marin. Though I miss the burritos and the Asian food, we have a beautiful new home and I most certainly don’t miss the fog! Best of all, of course, my daughters have given me the endless charm of their last years of true childhood. Their enthusiasms, their music (so much better than my own), and even their Macbeth (from theatre this summer) were simple gifts that graced the year.
In my first post of 2011 I quoted the same famous lines that title this post. In work and life are we not always and forever going “once more into the breach”? It seems so to me. I hope your year, too, has been good and that you’ll continue to keep company with me in 2012.
