Is it surprising that tools influence the builder? Give a man a hammer and nails and he’ll build one sort of house. Give him only a saw, and you’ll get a rather different sort of building. In Web analytics and Social Media, our tools have a surprisingly large influence on what we choose to measure. They frame the problem, provide a vocabulary, and suggest certain types of measurement. In Web analytics, our tools fire once per page. So we talked of page views and we looked at utterly meaningless reports like “Top Exit Pages”. We use cookies to track people, and it took us years to free ourselves from the tyranny of utterly stupid, tool-induced measures like “daily uniques.” By making some measurements easy, some hard, and some quite impossible, tools frame the measurement problem in ways that often lead us down remarkably unproductive directions.
In Social Media, we are once again learning just how influential tools can be in framing what we measure and how we report. The measure I have in mind is none other than the single most ubiquitious metric in Social Media: the mention count.
Most social media tools have roots, somewhere, in traditional PR measurement. What, after all, is a listening tool but a glorified electronic clipping service? Clipping services scanned print media searching for mentions of a brand. When they found something, they “clipped’ the article and passed it on to the company. They even counted the number of clippings.
In social listening tools, clipping is the “culling” process – the sampling and selection that I’ve been writing about. Of course, the process is no longer manual and it’s no longer confined to traditional media. And therein lies the problem.
The whole point of social listening tools is that they listen to a huge variety of sources. From traditional media to the blogsphere to Twitter to communities both public and private. Now if you’re a tool builder, it’s obvious that the more sources you have the better. An easier selling point can hardly be imagined.
It’s also obvious that you can’t just report on “articles”, “blogs”, “tweets”, and “community posts” as if they were all discrete systems. It’s too confusing. You need something simple that unifies these things and allows you treat them holistically. You need, in short, the MENTION.
The Mention is a perfect abstraction. It’s common across ANY imaginable communication. It should work on anything from Telepathy to Phone Books. It’s wonderfully countable. You can aggregated Mentions and trend them over time. You can break them out by sentiment. You can divide them from a larger pie and calculate share of voice. You can even multiply them by influence weights to calculate impact.
Believe me, we at Semphonic have done all of these things.
But does any of it make any sense? Is there really an interesting metric called Mentions that can be aggregated across all of these sources?
I’m starting to think not.
What, really, is there between a story in the NY Times and in a tweet for Customer Support that is comparable and countable? I can’t think of anything.
If our tools didn’t give us a measure this stupid, would we ever actually want it?
And does multiplying it by an Influence Weight somehow make it right?
Again, I’m thinking not.
I started this series by describing the different uses of Social Media and the ways that measurement maps to those functions. It seems so obvious, but it’s so easy to miss. If you’re intent is to measure PR effectiveness, then you need to measure the subset of social media sources that are targets of PR. By source, that means traditional media and the blogsphere. It also means selected elements of Twitter and Facebook because those aren’t purely social communities – there are plenty of influencers using those platforms for what amount to traditional media purposes. On the other hand, the vast amount of consumer chatter about your brand has nothing to do with PR effectiveness. Let me repeat that, most social chatter has absolutely nothing to do with PR effectiveness.
Flip this around the other way. If you’re intent is to measure brand awareness (ala Brand Tracking studies), then you should have no interest in traditional media or the blogsphere or in the group of professional Twitter and Facebook users. Those influencers have nothing to do with measuring traditional consumer awareness – they are the people your survey folks take great care to screen out when they build their studies! It’s not that they aren’t adding to your study – they are positively destroying it. It’s one of the great challenges of Social Media Measurement for consumer research, this removing influencers and their immediate effects from your sample.
It seems to me that most functions (PR, Consumer Research, Community) live within a subset of Sources and topics. It’s a subset based on a simple Segmentation whose top level is probably something along the lines of “Media/Influencers” and “Consumers” and is further sub-divided from there. The aggregation of Mention counts across source necessarily cuts against the idea of analysis by function. It’s also clear that Source is often, even within function, a very distinct set of Mentions and that our segmentation often cuts across newer Social sources.
In measuring PR effectiveness, my analysis probably needs to be at the Source level and below. Comparing Mentions between traditional media and the blogsphere or Twitter is naturally misleading – even if you’ve segmented to include only Media/Influencers from these non-traditional sources. If you don’t weight the numbers, they are silly. If you do weight them, they are opaque. What measure is there to compare that NY Times mention with a blog mention? Readers? I don’t think so.
It is so much harder to create a good influence weighting across Source than it is within Source (and that’s hard enough) that I can’t imagine that the effort will be worthwhile. If you’ve ever really studied the Influence weightings that tools use, you know they are mostly a bunch of made-up rules stretched into a formula whose sole purpose is to yield some number that can be charted and trended. Using these numbers within a single Source type might work. Use them across Source types at your peril!
The more I think about this, the less attractive and interesting does the idea of Mention Counts across Source types and functions appear. I see why it’s there. If I were building a tool, I would have done the same thing. But it’s our job – and it’s not easy – to reject what doesn’t make sense in our tools.
We don’t need to report on Mentions totals. We don’t need to aggregate sources that don’t make sense. Share of voice isn’t mentions across all sources added up as if they were so many grains of indistinguishable rice. Influence metrics aren’t comparable across source. And the whole tool edifice of reporting that aggregates all these very distinct things into abstract mentions is just not useful!
We need to understand that the measurement of consumer sentiment and PR effectiveness (and Influencer sentiment) are two fundamentally different tasks that need almost completely differentiated samples. The fact that our tools give us counts across these things (and across other categories that we don’t care about), shouldn’t lure us into believing those counts mean something.
I’ve often remarked how similar is the current experience of Social Media Measurement with the early days of Web analytics that I remember. We are all making so many of the same mistakes – Semphonic included. But here’s where our deep experience in digital measurement is actually bearing some fruit and proving to be a real advantage. Because this time around, we are learning much faster. In Web analytics, it took us years to learn the metrics to ignore and the metrics with real meaning. It’s taken even longer to understand many (if not all) of the ways our tools both help and hinder us in our task. But we’ve learned that just because a tool reports a metric, that doesn’t make it meaningful.
In Social media, I know of no likelier candidate for the scrap-bin of history than the Total Mention Count across all Sources. May it “Rest In Peace” in our tools, unused, unloved, and, most of all, unreported.
