Douglas Laney left META Group to start Evalubase -- a research service that gathers and analyzes the real-world experiences of IT buyers and users. Ben Bradley recently spoke to Laney, founder and CEO of Evalubase about the power of the IT community’s collective wisdom, building credibility and the state of IT research.
BRADLEY: Who knows better, the analysts or the IT buyers themselves?
LANEY: It depends on who you ask. When I was an IT industry analyst, I recognized my clients as a whole knew a lot more about my area of coverage than I did. IT professionals shouldn’t be scared to ask research analysts, “Say, when was the last time you ever used the products you’re advising us on anyway?” For some reason, analyst firms only capitalized on one side of the Internet—the outbound side. That is, they were quick to use the Internet for research distribution, but failed to capitalize on its democratizing power. Most IT research organizations continue to turn their backs on the collective wisdom of the IT community. It’s no secret that the IT community increasingly is concerned about the credibility of speculative, anecdotal, vendor-influenced research, and rehashed news posing as research. Most analyst-based research is based on the opinions of the analysts who rely on heavily on vendor marketing pitches and canned demos, but never actually use or see products being used in the real world. Cozy analyst-vendor relationships can also affect objectivity. Vendors regularly pay analysts to write whitepapers or speak at conferences, and invite them to events at fancy resorts. It’s hard to write something negative about a vendor after skiing with its CEO! Also, the written-research model introduces a lag time that results in reports being stale by the time they’re published. A study done by the IT Align Group showed that by nearly 3-to-1 IT professionals believe the insights of their peers are more valuable than those of so-called experts. Even though the traditional analyst-based model is well-entrenched in the market place and in the minds of IT professionals, I believe the time is right for a “wisdom of crowds” approach to IT research. Eventually, we’ll see hybrid approaches in which analyst postulates are fueled and tested by continuous primary research. The bottom line is that IT professionals throughout the world should be, must be, and will be incorporated into the research process.
BRADLEY: Now that you are an owner instead of an employee, what keeps you awake at night?
LANEY: When we started this business, what kept me awake was coming up with the secret sauce for getting people to want to and enjoy participating in our ongoing study of the enterprise technology marketplace. Then it hit me: IT professionals and executives crave scorecards and external benchmarks. So that’s just what we give them as a thank-you for participating—an instant, dynamic scorecard that shows participants how they’re doing with the technologies and vendors they’ve evaluated versus various peer groups. IT organizations can benchmark their entire technology portfolio using Evalubase for free, within days. Managing expectations is also a big challenge for us. Because of Evalubase’s open-participation research model, we’re expected to cover more vendors and more products than analyst firms or industry publications. And people expect that we have a statistically-significant set of data on every product and vendor we track. Within months of our launch we had more ratings on more products and vendor than even the most gargantuan analyst firm. And while we can’t promise statistical relevance on all but the most ubiquitous vendors thus far, how statistically-relevant or empirical are dots on a two-dimensional analyst-surmised magic quadrant anyway?
BRADLEY: Credibility is important for an analyst firm. How can you maintain objectivity?
LANEY: We’re not an analyst firm. We let the data do the talking. All of our research is based exclusively on the real quantified feedback of real IT professionals who are really using enterprise technologies. There’s no speculation, conjectured probabilities, re-hashed news or vendor-influence in our model. Yes, vendors can and do pay to subscribe to our research, but the only way they can influence it is to perform better in the eyes of their customers. Our biggest challenge (and our true strength) is that truly objective, empirical research provides no "infotainment" value. People enjoy IT horror and success stories, and they eat-up analyst stand-up routines no matter how dated. Pure empirical research flies in the face of the IT "rock star" aura cultivated by analyst firms for the past 20-plus years. We also work hard to prevent vendors from stuffing the ballot box. Who would trust a vendor comparison study that allowed vendors to participate? So we require participants to register with a real business email address. Then we require them to check a box indicating that they have no relationship with the vendor that would unduly influence their evaluation. We also allow anyone to question the validity of any evaluation. Evalubase subscribers can browse and view actual evaluations, although the evaluator’s identity of course remains confidential.
BRADLEY: What was your biggest mistake and what was your biggest gamble?
LANEY: Early on we had a couple false starts trying to partner with the consulting community, and for some reason we just gave up on the idea. We were trying to herd some really big systems integrators. One 10,000+ consultant firm in particular had committed to submitting a large number of evaluations each month, but their executive support for any extra-curricular activities fizzled. The other firm was a loose conglomeration of IT execs, who really weren’t hands-on enough to provide useful feedback. Now we're focusing on building a global research alliance with the attention-starved 10-1000 person boutique consultancies, media outlets and user groups.
Also, we assumed that everyone would want self-service access to the real-time data. So we built-in a query tool that enables users to browse and chart Evalubase data any-which-way and setup alerts to notify them when data about vendors, products or markets they’re tracking change. However, we’ve learned that many people still want to read reports and be spoon-fed analysis. Stay tuned. I think that leaving my VP and analyst job at META Group was the biggest personal gamble. The timing turned out to be good, as META was snatched up by Gartner a few months later. When the first strategic move of a new CEO is to eliminate the free coffee…well…the writing was on the wall. Many analysts lost their jobs. I could have been out on the street too. From a business perspective, the launch timing was perfect too. Many IT managers are looking for a research alternative.
BRADLEY: What advice would you give others about growing their businesses?
LANEY: In your business plan there should be a list of key success factors and key risks (or red herrings). These should be top of mind and top of whiteboard. If any one of these things falters, they need to be addressed head-on and not swept under the rug. Also, time management is HUGE in a young company. You need to think in “startup years.” Things that took you a month to do in your last corporate job need to get done in a week. Things that used to take a week need to get done in a day. And so on. I think it’s critical to have someone on the executive team with start-up experience to keep an eye on the clock more than anything. A business can be grown organically and conservatively if competitive pressures are nominal. But in the Internet age and global market place, this is rarely a luxury. The alternative is to fund growth and ownership, or worse cede control. The lucky few can create buzz, a hyper-catalyst for organic growth. But don’t ever count on it.
