7 Questions to Evaluate SaaS

Back in the desktop software era, magazines ran software reviews in which the side-by-side comparisons of features took up an entire page. Buyers used these reviews to shortlist vendors, trying to anticipate which features they’d need over the next five years. Typically, the software with the most features won. Feature-itis ruled.

No more. With software as a service, the focus has become whether the tool is good enough on day one and how well it will adapt over time. Take, for example, the Family Service Agency of San Francisco, which replaced its ailing paper-based system with SaaS donated by the Foundation, improving productivity and accountability along the way. Speaking today at the SIIA’s eGov event in Washington, D.C., Bob Bennett, the agency’s CEO, explained how the agency turned a salesforce automation tool into a social services management tool.

The point here is that the initial feature set didn’t matter much. Indeed, in order to evaluate SaaS, those page-long feature comparisons can be whittled down to just seven critical questions:

  • Adaptability: How easily can you modify the application? This can be as simple as adding fields or building dashboards, or as advanced as a programming platform.
  • Reliability: How much can you depend on the system to function well? This boils down to four things: Performance, availability, scalability and security.
  • Task productivity: How effectively can your users accomplish their goals? How many cases-per-minute or entries-per-day can workers do, and how many errors do they make?
  • Price: How much will it cost — really? Because SaaS offerings are so varied in pricing, it’s hard to compare them. A better model is to create several benchmark subscribers (a 10-, 100-, and 1,000-person organization) and compare upfront and ongoing costs for them.
  • Back-end integration: Can you plug it in to other things? Any enterprise SaaS offering will have to work with other systems, for everything from authentication to data sharing.
  • Longevity: How long will the SaaS company be around, and what’s your exit strategy? With ISVs, you could ask for software in escrow. But as the sudden disappearance of Coghead shows, when a SaaS provider closes down, your entire IT systems can vanish with the flick of an “off” switch. Offers from Intuit and others to help stranded customers notwithstanding, this is a big problem.
  • Ecosystem: How many third-party developers and integrators surround a particular platform with plug-ins and add-ons, and how active are they? A vibrant ecosystem means a more extensible, flexible solution.

The key point, however, is that features on day one don’t matter as much as the efficiencies and cost savings you can squeeze out of the SaaS tool within 30 days of adoption — and how confident you are that those efficiencies and cost savings will endure.