Software Testing Gets New Respect

Businesses turn to automated testing to boost confidence in their IT systems

ong ensconced in backroom IT offices as the Rodney Dangerfield of the application development process, software testing--specifically of the automated type--is gaining newfound respect. Fueled by widespread business expeditions into the Internet economy, testing has surfaced not only as a critical IT issue, but also as an even more critical business issue.

Companies intent on transacting revenue-generating business or offering enhanced customer services online are increasingly turning to automated testing solutions to gain confidence in their IT systems, fully understand how applications will behave under real-world conditions, uncover and rectify issues, and systematically manage growth.

The worldwide market for automated software quality tools, including mainframe and distributed environments, reached $931 million in 1999, a 23.6% increase over 1998 figures, according to recently published research from International Data Corp. And the market is slated to grow to more than $2.6 billion by 2004.

Dick Heiman, IDC's research director and analyst, says the widespread growth in the adoption of automated testing solutions is being fueled by the steps that businesses are taking to leverage the Web. "The old paradigm of forgoing a structured quality initiative in exchange for faster deployment--with plans to address quality issues with application updates--doesn't fly in the Internet economy."

Kevin Gallagher, VP of research with Newport Group Inc., an IT research and reporting firm, agrees. "Businesses must know how applications will perform and behave once they're open to the Web," he says. As businesses move from having isolated front-end Web applications to integrated Web-enabled enterprises, with multiple application interdependencies within or between businesses, the issue of testing is especially crucial, says Gallagher. "It's a simple matter of mitigating business risk, maintaining integrity, and gathering knowledge and confidence in the IT systems that are so heavily relied upon to transact daily business."

Business acceptance of automated testing as a mainline business practice hasn't come quickly or easily. For traditional client-server environments, it's been more common for testing efforts to be shortchanged in exchange for more development time or faster deployments.

Internal conflicts that result in poor communication between developers and testers have also contributed to weakened efforts, along with a loss of focus on the fact that both facets of IT must work together to achieve the end result. Too often, automated tools have been shelved due to inadequate test process support, lukewarm business management support, and staff turnover.

However, in the ever-widening Internet economy, much is changing. According to Brian Chase, quality assurance manager for Thomson & Thomson, a provider of legal research, "Today, automated testing tools are viewed as a necessary purchase and we have annual budget allocations for them." Initially approved for purchase on the heels of the company's Y2K testing efforts, Chase says "we've proven that automated testing solutions work to help us deliver our Web-based product to market sooner, with more accuracy and less user-found errors. In turn, business support continues to strengthen."

The equation is simple: Application performance and transaction precision equate to the efficient business services that lead to customer satisfaction, which ultimately boils down to revenue. So it goes that IT managers and business executives are speaking the same language--bottom-line revenue--and, therefore, the business investments in automated testing solutions are more easily justified and understood.

GlobalFulfillment.com's senior VP and chief technology officer Phil Wilkerson is an industry veteran of test-process engineering and test automation. Wilkerson worked to maintain the quality of the systems that support Levi Strauss & Co. and Gap Inc. before taking on his post at startup GlobalFulfillment.com, a Los Angeles provider of fulfillment solutions for retailers.

With the unique experience of obtaining business management buy-in for automated tools in closed client-server environments and open, Web-enabled enterprises, Wilkerson says the growing volume of business via the Web has increased business support for test automation. "There's more education about automated testing at the business level, and a greater awareness about the consequences that result when things don't work," he says.

Further, because Web application delivery cycles are so much faster than with traditional client-server or legacy systems, the potential for errors is greater, creating, in turn, a greater need for testing.

CIO Nathan Harper of Guild.com Inc., an online source for purchasing contemporary art, also recognizes a major upward shift in the business value that automated testing has gained as a direct result of doing business online. Harper received no business resistance to his request to purchase automated tools and consulting services for the E-commerce startup.

"The recognition of the value that automated testing brings to the business has changed enormously," says Harper. "In the old days, we were writing applications for a set number of users. We knew things like how many database connections were needed and what the maximum load requirements were for the system. Building to known requirements was tough, but at least it was manageable and quantifiable. The problem with the Internet is the surge activity."

For example, according to Harper, Guild.com averages 10,000 daily site visitors. Advertising campaigns typically cause traffic to spike to five or six times the daily average. And when Time magazine named Guild .com a top art site just before Christmas, traffic surged to nearly 20 times the daily average.

"These types of scenarios are difficult to plan and test for," says Harper, "so we need to use the tools in order to know ahead of time what will happen when these types of traffic surges occur। Will performance slow, will it grind to a halt? How much business could we potentially be losing? How much frustration are we causing our customers and, as a result, how much damage to our business reputation?"


s the E-commerce hype has cooled and more robust Web technologies have emerged, developers are overcoming numerous learning curves on newer technologies and businesses are working to re-architect Web infrastructures. Web development is embracing more structured processes, and automated testing is an integral part of those processes.

Thomson & Thomson's Chase agrees. As a proprietor of legal research services for trademark, copyright, and script clearance, Thomson & Thomson developed its Saegis Web application to provide its subscribers with online access to research. Within the last 24 months, the Saegis user base has grown from 250 to 10,000.

"As growth continues to accelerate, we're focused on reconfiguring software and rearchitecting the Web infrastructure," says Chase. "As a result of a better-thought-out Web architecture and testing processes that included structured and repeatable functional regression and load tests, we're more confident in our system."

The "gut instinct" testing mentality that previously existed for determining readiness has been replaced with quantifiable facts about how the application will perform under real-world conditions. Chase reports that communication with the development team has improved as a result of leveraging automated tools. "Now when we uncover problems, we can back up our findings with the information our developers need to resolve the issues."

Many an IT manager has pitched a case for purchasing automated testing solutions and been challenged to prove the business value. The cost justification: How long before the investment outweighs the cost? After all, the foray into automating the testing process is expensive. Automated tools are costly, as are the qualified staff and hardware resources necessary to operate and maintain them.

According to Harper, Guild.com spent $100,000 on testing tools in 1999 and has allocated another $60,000 this year. This doesn't include the multiple quality-assurance hires, required hardware, or the outsourced testing consultant the company retains. This is why justifying the purchase and a commitment to automated testing have traditionally been uphill battles. However, the cost of not testing applications and eradicating errors carries an even higher price tag, a fact that's easing the budgeting pain.

OurBeginning.com CEO Michael Budowski has changed his views on the importance of automated testing. "When our IT team came to us with the suggestion of purchasing an automated testing solution via a hosted testing service, our executive team didn't give it much consideration because we were looking at it in terms of dollars and cents. However, when we stepped back and evaluated our seven-figure investment into the development of our site, and the fact that we were set to advertise during the Super Bowl, we decided to approve the purchase. "

In the end, it was a good business decision because the testing did find errors. After making the appropriate adjustments, OurBeginning.com was able to handle the 1,600% surge of visitors that flooded its site after the Super Bowl ad ran.

Budowski says that although getting buy-in prior to the Super Bowl was difficult, getting buy-in now wouldn't be nearly as tough. "I guess I had to have that experience to gain a first-hand technical perspective on the value that automated testing was able to deliver," he says.

What tends to most easily convince businesses of the need to establish formal automated test processes is a bad experience with an application providing poor service, or one that fails altogether.

The upside of that experience is that application quality becomes a bigger issue and gains the business support needed to survive. The downside is that adopting automated tools in response to a crisis can seem to worsen problems if it's assumed that having the tools will immediately equate to better application quality. Tools require process support and qualified people to run them.

According to Steve Marshall, CEO of T-Plan Ltd., a provider of test process-management tools, "typical client consulting engagements require 75% of the training to focus on the test process, with the remainder spent on learning how and where to leverage automated tools."

GlobalFulfillment.com's Wilkerson says automating testing "takes a lot of commitment, hard work, and adult supervision. You can't just install these tools and walk away. You must really understand what you're testing and be sure that processes are in place." Automating the testing process is a formidable challenge and it's an ongoing process, forging a continuous path of quality assessment and application improvement.

In response to growing business acceptance of automated testing solutions, vendors are strengthening their tool suites and emerging with flexible testing-service options. That's what Web services vendors such as Compuware, Cyrano, Mercury Interactive, Rational Software, and Segue Software are doing.

In addition to the traditional vendors responsible for turning automated testing into a distinct discipline, there exists a new breed of vendors that are providing automated testing solutions built exclusively for the Web. Those vendors include Keynote Systems, RadView Software, and Teradyne-owned RSW Software

Both RSW and RadView focus on product sales, whereas Keynote offers load testing to complement the company's performance management services. For all these vendors, the Web-testing space is realizing the highest customer demand and, thus, potential for growth. For example, RSW and RadView experienced triple-digit percentage growth between 1998 and 1999 as a direct result of their Web focus.

These newer market entrants are likely to expand their offerings into other areas as Web technologies develop deeper roots in the complete IT enterprise. RSW Software, for example, has already begun such efforts with the introduction of EJB-test, a solution to test the scalability and functionality of Enterprise JavaBeans middle-tier applications.

As more companies work to develop and deploy applications for the Internet economy, by default there's a larger audience faced with the challenges of maximizing quality. Constant time-to-market pressure is only one issue. For many startup companies, financial constraints limit the ability to purchase tools, hardware, and hire people.

Recognizing these pressures, vendors of automated solutions have begun to provide "test on demand" solutions by leveraging their technology and expertise to provide testing as a hosted service via the Internet for a fee. These services can range from a one-time load test to a series of on-going tests with script development and maintenance, application performance measurement and monitoring, and real-time reports.

Load testing services from both Mercury Interactive and Keynote--an offering that exists as a result of the company's May 10 acquisition of Velogic Inc.--are gaining popularity. These services are beneficial for companies under pressure to deliver but short on resources and time.

In addition, these services give businesses the chance to have their applications evaluated by a third party, which adds an element of objectivity. Also, large volumes of realistic user loads can be leveraged through these services, which take into account realistic Web usage scenarios across geographic locations.

Because these services are geared for a fully functional application, they do detract from the life-cycle testing approach, which requires that testing begin early and continue throughout the development process. Therefore, the best solutions for Web-enabled enterprises will require some combination of in-house tools and outsourced services.

Another trend extending the value of automated testing is application monitoring in the production environment, from the perspective of the end user. For testing-tool vendors, application monitoring stretches quality practices into production. This functionality is critical for Web environments because businesses must constantly be aware of how users experience their applications.

Still a ways off, the next trend on the horizon for tool vendors will be a shift in business model to that of an application service provider, so customers can manage their own testing processes by renting the tools they require for testing. This will help to ease the requirement for hardware resources and tool licensing, allowing customers to pay for only what they need.

For many, obtaining automated tools via the ASP model will be attractive from a pricing standpoint because it will remove the big-ticket, up-front cost, which in turn will remove the entry barrier that some businesses face.

Given the undeniably complex array of technologies and the unpredictability of users' loads inherent in the Web-enabled enterprise, automated testing practices will continue to gain business acceptance for companies participating in the Internet economy.

The amount of test automation that's chosen to support the quality of any IT system comes down to finding the right balance within individual companies, but it's clear that there are more choices available today to help the testing effort along.

It will always be a challenge to blend new technologies with existing ones and test that they work together synergistically. Rapid-release cycles and continuous changes make test automation a more practical and reliable way to ensure quality IT systems.

Testing is a business investment, and what's put into it will directly correlate to the business value that's derived. Furthermore, like any good investment, businesses must think of their returns over the long term.

Twenty-five years ago, programmers never imagined their applications would live to see the 21st century. They were wrong about the longevity of their applications, and Y2K remediation carried a heavy price tag.

In the end, it's impossible to predict how far into the future any application may carry a business. Yet with the increasing reliance on component-based development strategies, at least pieces of applications will likely be around for many years to come.

Perhaps the time has arrived for businesses to tear a page out of Henry Ford's lesson book and leverage test automation in a manner that works to make quality "Job One" for the Web-enabled enterprise.

SOURCE www.informationweek.com