Short answer: No. I mean Yes. Let me explain.
There is no such thing as a completely unbiased review. So if you’re looking for one and only one source of B2B software info to help you decide which software is “best” for your business, you’re bound to be disappointed.
Even detailed reviews — whether put out by prominent analyst firms like Forrester and Gartner, boutique analysts, or review sites like G2Growd, TrustRadius, or GetApp — have biases inherent in any research. Reviewers are biased by their own experiences, data sources easily available, etc.
According to consultant Matt Heinz of Heinz Marketing:
I’ve worked with “review” sites like this for literally 15+ years. None of them – be they analyst firms like Gartner or Forrester, or verticalized sites like G2 and others – are unbiased or objective. That’s not necessarily bad.
Scott Brinker, “MarTech” thought leader and Co-founder & CTO of ion interactive, says that while review sites help improve transparency, …
Even in the absence of explicit “gaming”, there’s a selection bias — both among the vendors who choose to embrace these sites (versus those who don’t) and the customers who choose to participate.
Exactly. That’s why I start with a emphatic “no.” Don’t put all your trust in any one source. Multiple sources will help you “triangulate” on a short list of options to consider more carefully.
That’s the process I used 4+ years ago to decide to use Mailchimp for email distribution at CustomerThink. We had previously used an enterprise-class email system for several years. When the contract was up I took a look around and found a number of options by doing Google searches, reading reviews, and visiting vendor web sites.
That helped me narrow down choices to Mailchimp, Infusionsoft and a couple of others I’ve since forgotten. I then did a detailed “test drive” to see how the solutions met our key use cases, evaluated prices, and made a decision. This is, of course, a very common pattern in B2B software decisions then, and now.
Software Review Sites Go Mainstream
Since then, B2B software review sites have been launched and have grown dramatically. In early 2014 I blogged about G2 Crowd, which in about a year had accumulated 10K reviews. Since then, that number has grown to nearly 50K.
TrustRadius, which I’ll be writing about in more depth in this post, has a similar history. According to Founder/CEO Vinay Bhagat, the site was launched in May 2013 with a focus on marketing automation (MA) products. Since then, it has has expanded to cloud software and 1,000 unique products.
In roughly 2 1/2 years TrustRadius has accumulated 19K reviews/ratings, enabling it to create “TrustMaps” (kinda like Gartner Magic Quadrants or Forrester Waves) in 30 categories. Impressive.
Yeah, but Can You Trust the Reviews?
Software vendors are now fully aware of the importance of reviews. Let’s not debate how far buyers are through their evaluation process before contacting sales. The vast majority of software buyers will research options with public sources along with peers, so it’s critical for vendors to be found online.
That’s given rise to the massive increase in content marketing the past few years. One of the many tactics are review sites, which are an example of online communities. Voice of Customer marketer Ernan Roman of ERDM says they play a critical role “helping businesses and consumers make informed decisions.”
However, Roman also notes that “anything which compromises the value of reviews is a significant disservice to consumers.”
Unfortunately, some vendors do that disservice by attempting to “put their thumb on the scale” of the evaluation process. It’s one thing to put out material on a vendor web site that portrays the company and its products in a flattering light. Nothing wrong with that, and prospects can filter accordingly.
But when a prospective software buyer visits an independent software review site, they expect, um, independent reviews.
A warning light went off for me early this year at a “customer success” conference where I found Influitive touting its ability to motivate customers to become advocates. Activities (asks) could mean providing a quote for a case study, speaking at a conference, or, writing a review.
On the one hand, I’m happy to see more tools to “activate promoters.” But when it comes to review sites, it gives me pause. If vendors are cherry picking their best customers and motivating them via incentives to post “awesome reviews that will influence people just like them to work with you instead of your competitors,” how can software buyers trust the reviews?
Left unchecked, the influx of glowing reviews will make these sites just as useful as Yelp. Meaning, take the reviews with a big shaker full of salt.
TrustRadius Ups the Ante
Fortunately, leaders of review sites are aware of the problem. G2Crowd and TrustRadius both require users to register with LinkedIn, which helps limit outright fraudulent reviews.
To deal with bias generated by vendor advocacy programs, TrustRadius just announced additional measures.
- Reviews now disclose review source and the use of incentives.
- Average rating calculations give greater weight to more recent, non-incentivized reviews
These measures do depend on users being honest when they fill out the review forms. Still, Bhagat assured me they have other techniques to detect vendor-driven reviews, and penalties to discourage such behavior.
The review process is very nicely designed, by the way. Note how source and incentives are shown in my Mailchimp review (red circle emphasis mine):
Reports Are Useful, But…
While taking TrustRadius for a test drive, I also registered for and downloaded several reports. In general, I found them really well done and useful. An impressive example of what you can do with community input.
Marketing consultant David Dodd agrees:
I thought the TrustRadius report that you provided was balanced and well-written. In addition, I thought they were wise to expressly state that “User ratings indicate sentiment, but should not be the sole determinant for product selection.”
One quibble: to get a report you have to check a box agreeing that “TrustRadius may share my information with vendors.” It’s certainly fair that, in exchange for access to a free report, sponsors may contact you. We use that process on CustomerThink, but it’s disclosed which sponsor is offering each specific paper, e-book, etc.
For TrustRadius, however, it’s not clear what happens to my contact info (“name”). Will it go to a dozen different vendors? Will I start getting phone calls from reps? Will I be put on a nurturing campaign using marketing automation technology?
I raised this issue with Bhagat and learned that TrustRadius puts these names (unqualified leads) through a qualification process. Given that I said I was a small business and just starting my research process, that may explain why I’ve not been contacted by vendor reps. Yet.
I’d like to see more transparency here. And perhaps an option to purchase the report instead of opting to be a “lead.” I suspect software buyers in big enterprises would prefer not to be targeted by vendors as the result of early research online.
Speaking of large enterprises, vendors that cater to them are underrepresented. Bhagat says it’s harder to get people in big companies to submit reviews. As a result, the CRM report was missing one notable vendor: Oracle. It will be included when it gets at least 10 reviews, still a small number in my view.
Trust, But Verify
MarTech analyst David Raab of Raab Associates points out a limitation of simple matrices that rank along two generic dimensions, one usually related to the size of the customer base of the vendor.
In practice, this has the very harmful effect of leading people to consider only the top few systems, even though there might be some other systems that better match their particular needs. It’s simply up to buyers to be smart and avoid this error; you can’t blame the vendors for making it possible.
In summary, I want to emphasize that TrustRadius and other B2B software review sites are a welcome and very useful resource for software buyers. I’m impressed by TrustRadius’ efforts to improve transparency and discourage vendor manipulation.
Still, I suggest taking the advice of former US President Ronald Reagan: “Trust, but verify”. That’s a translation of an old Russian proverb “doveryai, no proveryai” that Reagan used when negotiating with Russian President Mikhail Gorbachev during the Cold War.
P.S. You may wonder how I came to write this review. TrustRadius VP of Marketing Bertrand Hazard emailed me about their new announcement and offered a phone call to discuss. I’m glad he did, and want to thank Hazard and Bhagat for a candid conversation.