Software Testing Guide BookPart I: Fundamentals of Software Testing 1. The Software Testing Guide Book Forward Software Testing has gained a phenomenal importance in the recent years in the System Development Life Cycle. Many learned people have worked on the topic and provided various techniques and methodologies for effective and efficient testing. Today, even though we have many books and articles on Software Test Engineering, many people are fallacious in understanding the underlying concepts of the subject. Software Testing Guide Book (STGB) is an open source project aimed at bringing the technicalities of Software Testing into one place and arriving at a common understanding. This guide book has been authored by professionals who have been exposed to Testing various applications. We wanted to bring out a base knowledge bank where Testing enthusiasts can start to learn the science and art of Software Testing, and this is how this book has come out. This guide book does not provide any specific methodologies to be followed for Testing, instead provides a conceptual understanding of the same. Note to the Reader: It is not our intention to tell you that this is a one-stop place for learning Testing. This is just a guide. Many eminent scientists have researched every topic you find in this book. We have just compiled everything in one place and made sure we explained each topic relating it to the practical world as we experienced it. If you find any subject matter that might look like we have copied from any existing book, we request you to let us know. It is not our intention to copy any material, and we brought out this book just to help Testing aspirants to have a basic understanding of the subject and guide them to be good at their job. All the material in this document is written in plain English, as we understand testing. Please send in your comments, suggestions or a word of encouragement to the team . Regards, The SofTReL Team About SofTReL The Software Testing Research Lab (SofTReL) is a non-profit organization dedicated for Research and Advancements of Software Testing. The concept of having a common place for Software Testing Research was formulated in 2001. Initially we named it ‘Software Quality and Engineering’. Recently in March 2004, we renamed it to “Software Testing Research Lab” – SofTReL. Professionals who are currently working with the industry and possess rich experience in testing form the members of the Lab. Purpose of this Document This document does not provide the reader with short cut’s to perform testing in daily life, but instead explains the various methodologies and techniques which have been proposed by eminent scientists in an easy and understandable way. This guide book is divided into three parts: Part I – Fundamentals of Software Testing This section addresses the fundamentals of Software Testing and their practical application in real life. Part II – Software Testing for various Architectures This section would concentrate in explaining testing applications under various architectures like Client/Server, Web, Pocket PC, Part III – Platform Specific Testing This section addresses testing C++ and Java applications using white box testing methodologies. This is Part I. All updates on the project are available at http://www.SofTReL.org/stgb.html. Authors The guide book has been authored by professionals who ‘Test’ everyday. Ajitha - GrayLogic Corporation, New Jersey, USA Mahesh, iPointSoft, Hyderabad Muhammad Kashif Jamil, Avanza Solutions, Naveed Mohammad – vMoksha, Prathima Nagaprakash – Wipro Technologies, Bangalore Harinath – SofTReL, Intended Audience This guide book is aimed at all Testing Professionals – from a beginner to an advanced user. This book would provide a baseline understanding of the conceptual theory. How to use this Document This book can be used as a guide for performing the Testing activities. A ‘guide’ here, we mean that this can provide you a road map as to how to approach a specific problem with respect to Testing. What this Guide Book is not This guide book is definitely not a silver/gold/diamond bullet which can help you to test any application. Instead this book would be reference help to perform Testing. How to Contribute This is an open source project. If you are interested in contributing to the book or to the Lab, please do write in to stgb at SoFTReL dot org. We need your expertise in the research activities. Future Enhancements This is the first part of the three-part Software Testing Guide Book (STGB) series. You can visit http://www.softrel.org/stgb.html for updates on the Project. Copyrights SofTReL is not proposing the Testing methodologies, types and various other concepts. We tried presenting each and every theoretical concept of Software Testing with a live example for easier understanding of the subject and arriving at a common understanding of Software Test Engineering. However, we did put in few of our proposed ways to achieve specific tasks and these are governed by The GNU Free Documentation License (GNU-FDL). Please visit http://www.gnu.org/doc/doc.html for complete guidelines of the license or alternatively you can find the license towards the end of this document 2. What is Software Testing and Why is it Important? A brief history of Software engineering and the SDLC. The software industry has evolved through 4 eras, 50’s –60’s, mid 60’s –late 70’s, mid 70’s- mid 80’s, and mid 80’s-present. Each era has its own distinctive characteristics, but over the years the software’s have increased in size and complexity. Several problems are common to almost all of the eras and are discussed below. The Software Crisis dates back to the 1960’s when the primary reasons for this situation were less than acceptable software engineering practices. In the early stages of software there was a lot of interest in computers, a lot of code written but no established standards. Then in early 70’s a lot of computer programs started failing and people lost confidence and thus an industry crisis was declared. Various reasons leading to the crisis included: q Hardware advances outpacing the ability to build software for this hardware. q The ability to build in pace with the demands. q Increasing dependency on software’s q Struggle to build reliable and high quality software q Poor design and inadequate resources. This crisis though identified in the early years, exists to date and we have examples of software failures around the world. Software is basically considered a failure if the project is terminated because of costs or overrun schedules, if the project has experienced overruns in excess of 50% of the original or if the software results in client lawsuits. Some examples of failures include failure of Air traffic control systems, failure of medical software, and failure in telecommunication software. The primary reason for these failures other than those mentioned above is due to bad software engineering practices adopted. Some of the worst software practices include: q No historical software-measurement data. q Rejection of accurate cost estimates. q Failure to use automated estimating and planning tools. q Excessive, irrational schedule pressure and creep in user requirements. q Failure to monitor progress and to perform risk management. q Failure to use design reviews and code inspections. To avoid these failures and thus improve the record, what is needed is a better understanding of the process, better estimation techniques for cost time and quality measures. But the question is, what is a process? Process transform inputs to outputs i.e. a product. A software process is a set of activities, methods and practices involving transformation that people use to develop and maintain software. At present a large number of problems exist due to a chaotic software process and the occasional success depends on individual efforts. Therefore to be able to deliver successful software projects, a focus on the process is essential since a focus on the product alone is likely to miss the scalability issues, and improvements in the existing system. This focus would help in the predictability of outcomes, project trends, and project characteristics. The process that has been defined and adopted needs to be managed well and thus process management comes into play. Process management is concerned with the knowledge and management of the software process, its technical aspects and also ensures that the processes are being followed as expected and improvements are shown. From this we conclude that a set of defined processes can possibly save us from software project failures. But it is nonetheless important to note that the process alone cannot help us avoid all the problems, because with varying circumstances the need varies and the process has to be adaptive to these varying needs. Importance needs to be given to the human aspect of software development since that alone can have a lot of impact on the results, and effective cost and time estimations may go totally waste if the human resources are not planned and managed effectively. Secondly, the reasons mentioned related to the software engineering principles may be resolved when the needs are correctly identified. Correct identification would then make it easier to identify the best practices that can be applied because one process that might be suitable for one organization may not be most suitable for another. Therefore to make a successful product a combination of Process and Technicalities will be required under the umbrella of a well-defined process. Having talked about the Software process overall, it is important to identify and relate the role software testing plays not only in producing quality software but also maneuvering the overall process. The computer society defines testing as follows: “Testing -- A verification method that applies a controlled set of conditions and stimuli for the purpose of finding errors. This is the most desirable method of verifying the functional and performance requirements. Test results are documented proof that requirements were met and can be repeated. The resulting data can be reviewed by all concerned for confirmation of capabilities.” There may be many definitions of software testing and many which appeal to us from time to time, but its best to start by defining testing and then move on depending on the requirements or needs. 3. Types of Development Systems The type of development project refers to the environment/methodology in which the software will be developed. Different testing approaches need to be used for different types of projects, just as different development approaches. 3.1 Traditional Development Systems The Traditional Development System has the following characteristics: · The traditional development system uses a system development methodology. · The user knows what the customer requires (Requirements are clear from the customer). · The development system determines the structure of the application. What do you do while testing: · Testing happens at the end of each phase of development. · Testing should concentrate if the requirements match the development. · Functional testing is required. 3.2 Iterative Development During the Iterative Development: · The requirements are not clear from the user (customer). · The structure of the software is pre-determined. Testing of Iterative Development projects should concentrate only if the CASE (Computer Aided Software Engineering) tools are properly utilized and the functionality is thoroughly tested. 3.3 Maintenance System The Maintenance System is where the structure of the program undergoes changes. The system is developed and being used, but it demands changes in the functional aspects of the system due to various reasons. Testing Maintenance Systems requires structural testing. Top priority should be put into Regression Testing. 3.4 Purchased/Contracted Software At times it may be required that you purchase software to integrate with your product or outsource the development of certain components of your product. This is Purchased or Contracted Software. When you need to integrate third party software to your existing software, this demands the testing of the purchased software with your requirements. Since the two systems are designed and developed differently, the integration takes the top priority during testing. Also, Regression Testing of the integrated software is a must to cross check if the two software’s are working as per the requirements. 4. Types of Software Systems The type of software system refers to the processing that will be performed by that system. This contains the following software system types. 4.1 Batch Systems The Batch Systems are a set of programs that perform certain activities which do not require any input from the user. A practical example is that when you are typing something on a word document, you press the key you require and the same is printed on the monitor. But processing (converting) the user input of the key to the machine understandable language, making the system understand what to be displayed and in return the word document displaying what you have typed is performed by the batch systems. These batch systems contain one or more Application Programming Interface (API) which perform various tasks. 4.2 Event Control Systems Event Control Systems process real time data to provide the user with results for what command (s) he is given. For example, when you type on the word document and press Ctrl + S, this tells the computer to save the document. How this is performed instantaneously? These real time command communications to the computer are provided by the Event Controls that are pre-defined in the system. 4.3 Process Control Systems There are two or more different systems that communicate to provide the end user a specific utility. When two systems communicate, the co-ordination or data transfer becomes vital. Process Control Systems are the one’s which receive data from a different system and instructs the system which sent the data to perform specific tasks based on the reply sent by the system which received the data. 4.4 Procedure Control Systems Procedure Control Systems are the one’s which control the functions of another system. 4.5 Advanced Mathematical Models Systems, which make use of heavy mathematics, fall into the category of Mathematical Models. Usually all the computer software make use of mathematics in some way or the other. But, Advance Mathematical Models can be classified when there is heavy utilization of mathematics for performing certain actions. An example of Advanced Mathematical Model can be simulation systems which uses graphics and controls the positioning of software on the monitor or Decision and Strategy making software’s. 4.6 Message Processing Systems A simple example is the SMS management software used by 4.7 Diagnostic Software Systems The Diagnostic Software System is one that helps in diagnosing the computer hardware components. When you plug in a new device to your computer and start it, you can see the diagnostic software system doing some work. The “New Hardware Found” dialogue can be seen as a result of this system. Today, almost all the Operating System’s come packed with Diagnostic Software Systems. 4.8 Sensor and Signal Processing Systems The message processing systems help in sending and receiving messages. The Sensor and Signal Processing Systems are more complex because these systems make use of mathematics for signal processing. In a signal processing system the computer receives input in the form of signals and then transforms the signals to a user understandable output. 4.9 Simulation Systems A simulation system is a software application, some times used in combination with specialized hardware, which re-creates or simulates the complex behavior of a system in its real environment. It can be defined in many ways: "The process of designing a model of a real system and conducting experiments with this model for the purpose of understanding the behavior of the system and/or evaluating various strategies for the operation of the system"-- Introduction to Simulation Using SIMAN, by C. D. Pegden, R. E. Shannon and R. P. Sadowski, McGraw-Hill, 1990. “A simulation is a software package (sometimes bundled with special hardware input devices) that re-creates or simulates, albeit in a simplified manner, a complex phenomena, environment, or experience, providing the user with the opportunity for some new level of understanding. It is interactive, and usually grounded in some objective reality. A simulation is based on some underlying computational model of the phenomena, environment, or experience that it is simulating. (In fact, some authors use model and modeling as synonyms of simulation.)" --Kurt Schumaker, A Taxonomy of Simulation Software." Learning Technology Review. In simple words simulation is nothing but a representation of a real system. In a programmable environment, simulations are used to study system behavior or test the system in an artificial environment that provides a limited representation of the real environment. Why Simulation Systems Simulation systems are easier, cheaper, and safer to use than real systems, and often the only way to build the real systems. For example, learning to fly a fighter plane using a simulator is much safer and less expensive than learning on a real fighter plane. System simulation mimics the operation of a real system such as the operation in a bank, or the running of the assembly line in a factory etc. Simulation in the early stage of design cycle is important because the cost of mistakes increases dramatically later in the product life cycle. Also, simulation software can analyze the operation of a real system without the involvement of an expert, i.e. it can also be analyzed with a non-expert like a manager. How to Build Simulation Systems In order to create a simulation system we need a realistic model of the system behavior. One way of simulation is to create smaller versions of the real system. The simulation system may use only software or a combination of software and hardware to model the real system. The simulation software often involves the integration of artificial intelligence and other modeling techniques. What applications fall under this category? Simulation is widely used in many fields. Some of the applications are:
What are the Characteristics of Simulation Systems? Simulation Systems can be characterized in numerous ways depending on the characterization criteria applied. Some of them are listed below. Deterministic Simulation Systems Deterministic Simulation Systems have completely predictable outcomes. That is, given a certain input we can predict the exact outcome. Another feature of these systems is idempotency, which means that the results for any given input are always the same. Examples include population prediction models, atmospheric science etc. Stochastic Simulation Systems Stochastic Simulation systems have models with random variables. This means that the exact outcome is not predictable for any given input, resulting in potentially very different outcomes for the same input. Static Simulation Systems Static Simulation systems use statistical models in which time does not play any role. These models include various probabilistic scenarios which are used to calculate the results of any given input. Examples of such systems include financial portfolio valuation models. The most common simulation technique used in these models is the Monte Carlo Simulation. Dynamic Simulation Systems A dynamic simulation system has a model that accommodates for changes in data over time. This means that the input data affecting the results will be entered into the simulation during its entire lifetime than just at the beginning. A simulation system used to predict the growth of the economy may need to incorporate changes in economic data, is a good example of a dynamic simulation system. Discrete Simulation Systems Discrete Simulation Systems use models that have discrete entities with multiple attributes. Each of these entities can be in any state, at any given time, represented by the values of its attributes. . The state of the system is a set of all the states of all its entities. This state changes one discrete step at a time as events happens in the system. Therefore, the actual designing of the simulation involves making choices about which entities to model, what attributes represent the Continuous Simulation Systems If instead of using a model with discrete entities we use data with continuous values, we will end up with continuous simulation. For example instead of trying to simulate battlefield scenarios by using discrete entities such as soldiers and tanks, we can try to model behavior and movements of troops by using differential equations. Social Simulation Systems Social simulation is not a technique by itself but uses the various types of simulation described above. However, because of the specialized application of those techniques for social simulation it deserves a special mention of its own. The field of social simulation involves using simulation to learn about and predict various social phenomenon such as voting patterns, migration patterns, economic decisions made by the general population, etc. One interesting application of social simulation is in a field called artificial life which is used to obtain useful insights into the formation and evolution of life. What can be the possible test approach? A simulation system’s primary responsibility is to replicate the behavior of the real system as accurately as possible. Therefore, a good place to start creating a test plan would be to understand the behavior of the real system. Subjective Testing Subjective testing mainly depends on an expert's opinion. An expert is a person who is proficient and experienced in the system under test. Conducting the test involves test runs of the simulation by the expert and then the expert evaluates and validates the results based on some criteria. One advantage of this approach over objective testing is that it can test those conditions which cannot be tested objectively. For example, an expert can determine whether the joystick handling of the flight feels "right". One disadvantage is that the evaluation of the system is based on the "expert's" opinion, which may differ from expert to expert. Also, if the system is very large then it is bound to have many experts. Each expert may view it differently and can give conflicting opinions. This makes it difficult to determine the validity of the system. Despite all these disadvantages, subjective testing is necessary for testing systems with human interaction. Objective Testing Objective testing is mainly used in systems where the data can be recorded while the simulation is running. This testing technique relies on the application of statistical and automated methods to the data collected. Statistical methods are used to provide an insight into the accuracy of the simulation. These methods include hypothesis testing, data plots, principle component analysis and cluster analysis. Automated testing requires a knowledge base of valid outcomes for various runs of simulation. This knowledge base is created by domain experts of the simulation system being tested. The data collected in various test runs is compared against this knowledge base to automatically validate the system under test. An advantage of this kind of testing is that the system can continually be regression tested as it is being developed. Statistical Methods Statistical methods are used to provide an insight into the accuracy of the simulation. These methods include hypothesis testing, data plots, principle component analysis and cluster analysis. Automated Testing Automated testing requires a knowledge base of valid outcomes for various runs of simulation. This knowledge base is created by domain experts of the simulation system being tested. The data collected in various test runs is compared against this knowledge base to automatically validate the system under test. An advantage of this kind of testing is that the system can continually be regression tested as it is being developed. 4.10 Database Management Systems As the name denotes, the Database Management Systems (DBMS) handles the management of databases. It is basically a collection of programs that enable the storage, modification and extraction from the Database. The DBMS, as they are often referred to as, can be of various different types ranging from small systems that run on PC’s to Mainframe’s. The following can be categorized as example of DBMS:
4.11 Data Acquisition Data Acquisition systems, taken in real time data and store them for future use. A simple example of Data Acquisition system can be a ATC (Air Traffic Control) Software which takes in real time data of the position and speed of the flight and stores it in compressed form for later use. 4.12 Data Presentation Data Presentation software stores data and displays the same to the user when required. An example is a Content Management System. You have a web site and this is in English, you also have your web site in other languages. The user can select the language he wishes to see and the system displays the same web site in the user chosen language. You develop your web site in various languages and store them on the system. The system displays the required language, the user chooses. 4.13 Decision and Planning Systems These systems use Artificial Intelligence techniques to provide decision-making solutions to the user. 4.14 Pattern and Image Processing Systems These systems are used for scanning, storing, modifying and displaying graphic images. The use of such systems is now being increased as research tests are being conducted in visual modeling and their use in our daily lives is increasing. These systems are used for security requests such as diagnosing photograph, thumb impression of the visitor etc. 4.15 Computer System Software Systems These are the normal computer software’s, that can be used for various purposes. 4.16 Software Development Tools These systems ease the process of Software Development. 5. Heuristics of Software Testing TestabilitySoftware testability is how easily, completely and conveniently a computer program can be tested. Software engineers design a computer product, system or program keeping in mind the product testability. Good programmers are willing to do things that will help the testing process and a checklist of possible design points, features and so on can be useful in negotiating with them. Here are the two main heuristics of software testing.
VisibilityVisibility is our ability to observe the states and outputs of the software under test. Features to improve the visibility are
Developers must provide full access (source code, infrastructure, etc) to testers. The Code, change records and design documents should be provided to the testing team. The testing team should read and understand the code.
The events to log include User events, System milestones, Error handling and completed transactions. The logs may be stored in files, ring buffers in memory, and/or serial ports. Things to be logged include description of event, timestamp, subsystem, resource usage and severity of event. Logging should be adjusted by subsystem and type. Log file report internal errors, help in isolating defects, and give useful information about context, tests, customer usage and test coverage. The more readable the Log Reports are, the easier it becomes to identify the defect cause and work towards corrective measures.
Data integrity checking and System level error detection (e.g. Microsoft Appviewer) are useful here. In addition, Assertions and probes with the following features are really helpful
Memory usage should be monitored to find memory leaks. States of running methods, threads or processes should be watched (Profiling interfaces may be used for this.). In addition, the configuration values should be dumped. Resource monitoring is of particular concern in applications where the load on the application in real time is estimated to be considerable. ControlControl refers to our ability to provide inputs and reach states in the software under test. The features to improve controllability are:
Allow data to be inspected, inserted or modified at points in the software. It is especially useful for dataflow applications. In addition, a pipe and filters architecture provides many opportunities for test points.
Custom UI controls often raise serious testability problems with GUI test drivers. Ensuring testability usually requires: § Adding methods to report necessary information
Interfaces may be provided specifically for testing e.g. Excel and Xconq etc. Existing interfaces may be able to support significant testing e.g. InstallSheild, AutoCAD,
Error seeding---instrumenting low level I/O code to simulate errors---makes it much easier to test error handling. It can be handled at both system and application level,
Testers should be notified when installation has completed successfully. They should be able to verify installation, programmatically create sample records and run multiple clients, daemons or servers on a single machine. A BROADER VIEWBelow are given a broader set of characteristics (usually known as James Bach heuristics) that lead to testable software. Categories of Heuristics of software testing
The better it works, the more efficiently it can be tested. The system should have few bugs, no bugs should block the execution of tests and the product should evolve in functional stages (simultaneous development and testing).
What we see is what we test. § Distinct output should be generated for each input § Current and past system states and variables should be visible during testing § All factors affecting the output should be visible. § Incorrect output should be easily identified. § Source code should be easily accessible. § Internal errors should be automatically detected (through self-testing mechanisms) and reported.
The better we control the software, the more the testing process can be automated and optimized. Check that § All outputs can be generated and code can be executed through some combination of input. § Software and hardware states can be controlled directly by the test engineer. § Inputs and output formats are consistent and structured. § Test can be conveniently, specified, automated and reproduced.
By controlling the scope of testing, we can quickly isolate problems and perform effective and efficient testing. The software system should be built from independent modules which can be tested independently.
The less there is to test, the more quickly we can test it. The points to consider in this regard are functional (e.g. minimum set of features), structural (e.g. architecture is modularized) and code (e.g. a coding standard is adopted) simplicity.
The fewer the changes, the fewer are the disruptions to testing. The changes to software should be infrequent, controlled and not invalidating existing tests. The software should be able to recover well from failures.
The more information we will have, the smarter we will test. The testers should be able to understand well the design, changes to the design and the dependencies between internal, external and shared components. Technical documentation should be instantly accessible, accurate, well organized, specific and detailed.
The more we know about the intended use of the software, the better we can organize our testing to find important bugs. The above heuristics can be used by a software engineer to develop a software configuration (i.e. program, data and documentation) that is convenient to test and verify. 6. When Testing should occur? Wrong Assumption Testing is sometimes incorrectly thought as an after-the-fact activity; performed after programming is done for a product. Instead, testing should be performed at every development stage of the product .Test data sets must be derived and their correctness and consistency should be monitored throughout the development process. If we divide the lifecycle of software development into “Requirements Analysis”, “Design”, “Programming/Construction” and “Operation and Maintenance”, then testing should accompany each of the above phases. If testing is isolated as a single phase late in the cycle, errors in the problem statement or design may incur exorbitant costs. Not only must the original error be corrected, but the entire structure built upon it must also be changed. Therefore, testing should not be isolated as an inspection activity. Rather testing should be involved throughout the SDLC in order to bring out a quality product. Testing Activities in Each Phase The following testing activities should be performed during the phases
Now we consider these in detail. Requirements Analysis The following test activities should be performed during this stage. · Invest in analysis at the beginning of the project - Having a clear, concise and formal statement of the requirements facilitates programming, communication, error analysis an d test data generation. The requirements statement should record the following information and decisions: 1. Program function - What the program must do? 2. The form, format, data types and units for input. 3. The form, format, data types and units for output. 4. How exceptions, errors and deviations are to be handled. 5. For scientific computations, the numerical method or at least the required accuracy of the solution. 6. The hardware/software environment required or assumed (e.g. the machine, the operating system, and the implementation language). Deciding the above issues is one of the activities related to testing that should be performed during this stage. · Start developing the test set at the requirements analysis phase - Data should be generated that can be used to determine whether the requirements have been met. To do this, the input domain should be partitioned into classes of values that the program will treat in a similar manner and for each class a representative element should be included in the test data. In addition, following should also be included in the data set: (1) boundary values (2) any non-extreme input values that would require special handling. The output domain should be treated similarly. Invalid input requires the same analysis as valid input. · The correctness, consistency and completeness of the requirements should also be analyzed - Consider whether the correct problem is being solved, check for conflicts and inconsistencies among the requirements and consider the possibility of missing cases. Design The design document aids in programming, communication, and error analysis and test data generation. The requirements statement and the design document should together give the problem and the organization of the solution i.e. what the program will do and how it will be done. The design document should contain: · Principal data structures. · Functions, algorithms, heuristics or special techniques used for processing. · The program organization, how it will be modularized and categorized into external and internal interfaces. · Any additional information. Here the testing activities should consist of:
The first two steps should also be performed by some colleague and not only the designer/developer. Programming/Construction Here the main testing points are:
The metrics used to measure testing thoroughness include statement testing (whether each statement in the program has been executed at least once), branch testing (whether each exit from each branch has been executed at least once) and path testing (whether all logical paths, which may involve repeated execution of various segments, have been executed at least once). Statement testing is the coverage metric most frequently used as it is relatively simple to implement. The amount of testing depends on the cost of an error. Critical programs or functions require more thorough testing than the less significant functions. Operations and maintenance Corrections, modifications and extensions are bound to occur even for small programs and testing is required every time there is a change. Testing during maintenance is termed regression testing. The test set, the test plan, and the test results for the original program should exist. Modifications must be made to accommodate the program changes, and then all portions of the program affected by the modifications must be re-tested. After regression testing is complete, the program and test documentation must be updated to reflect the changes. 7. The Test Development Life Cycle (TDLC) Usually, Testing is considered as a part of the System Development Life Cycle. With our practical experience, we framed this Test Development Life Cycle. The diagram does not depict where and when you write your Test Plan and Strategy documents. But, it is understood that before you begin your testing activities these documents should be ready. Ideally, when the Project Plan and Project Strategy are being made, this is the time when the Test Plan and Test Strategy documents are also made.
8. When should Testing stop? "When to stop testing" is one of the most difficult questions to a test engineer. The following are few of the common Test Stop criteria:
Practically, we feel that the decision of stopping testing is based on the level of the risk acceptable to the management. As testing is a never ending process we can never assume that 100 % testing has been done, we can only minimize the risk of shipping the product to client with X testing done. The risk can be measured by Risk analysis but for small duration / low budget / low resources project, risk can be deduced by simply: -
9. Verification Strategies What is ‘Verification’? Verification is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase.[1] What is the importance of the Verification Phase? Verification process helps in detecting defects early, and preventing their leakage downstream. Thus, the higher cost of later detection and rework is eliminated. 9.1 Review A process or meeting during which a work product, or set of work products, is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. The main goal of reviews is to find defects. Reviews are a good compliment to testing to help assure quality. A few purposes’ of SQA reviews can be as follows:
What are the various types of reviews? Types of reviews include Management Reviews, Technical Reviews, Inspections, Walkthroughs and Audits. Management Reviews Management reviews are performed by those directly responsible for the system in order to monitor progress, determine status of plans and schedules, confirm requirements and their system allocation. Therefore the main objectives of Management Reviews can be categorized as follows:
Support decisions made during such reviews include Corrective actions, Changes in the allocation of resources or changes to the scope of the project In management reviews the following Software products are reviewed: Audit Reports Contingency plans Installation plans Risk management plans Software Q/A The participants of the review play the roles of Decision-Maker, Review Leader, Recorder, Management Staff, and Technical Staff. Technical Reviews
Technical reviews confirm that product Conforms to specifications, adheres to regulations, standards, guidelines, plans, changes are properly implemented, changes affect only those system areas identified by the change specification. The main objectives of Technical Reviews can be categorized as follows: · Ensure that the software confirms to the organization standards. · Ensure that any changes in the development procedures (design, coding, testing) are implemented per the organization pre-defined standards. In technical reviews, the following Software products are reviewed · Software requirements specification · Software design description · Software test documentation · Software user documentation · Installation procedure · Release notes The participants of the review play the roles of Decision-maker, Review leader, Recorder, Technical staff.
What is Requirement Review? A process or meeting during which the requirements for a system, hardware item, or software item are presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include system requirements review, software requirements review. Who is involved in Requirement Review?· Product management leads Requirement Review. Members from every affected department participates in the review Input CriteriaSoftware requirement specification is the essential document for the review. A checklist can be used for the review. Exit CriteriaExit criteria include the filled & completed checklist with the reviewers’ comments & suggestions and the re-verification whether they are incorporated in the documents. What is Design Review?A process or meeting during which a system, hardware, or software design is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include critical design review, preliminary design review, and system design review. Who involve in Design Review?· QA team member leads design review. Members from development team and QA team participate in the review. Input CriteriaDesign document is the essential document for the review. A checklist can be used for the review. Exit CriteriaExit criteria include the filled & completed checklist with the reviewers’ comments & suggestions and the re-verification whether they are incorporated in the documents. What is Code Review?A meeting at which software code is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Who is involved in Code Review?· QA team member (In case the QA Team is only involved in Black Box Testing, then the Development team lead chairs the review team) leads code review. Members from development team and QA team participate in the review. Input CriteriaThe Coding Standards Document and the Source file are the essential documents for the review. A checklist can be used for the review. Exit CriteriaExit criteria include the filled & completed checklist with the reviewers’ comments & suggestions and the re-verification whether they are incorporated in the documents. 9.2 Walkthrough A static analysis technique in which a designer or programmer leads members of the development team and other interested parties through a segment of documentation or code, and the participants ask questions and make comments about possible errors, violation of development standards, and other problems. The objectives of Walkthrough can be summarized as follows: · Detect errors early. · Ensure (re)established standards are followed: · Train and exchange technical information among project teams which participate in the walkthrough. · Increase the quality of the project, thereby improving morale of the team members. The participants in Walkthroughs assume one or more of the following roles: a) Walk-through leader b) Recorder c) Author d) Team member To consider a review as a systematic walk-through, a team of at least two members shall be assembled. Roles may be shared among the team members. The walk-through leader or the author may serve as the recorder. The walk-through leader may be the author. Individuals holding management positions over any member of the walk-through team shall not participate in the walk-through. Input to the walk-through shall include the following: a) A statement of objectives for the walk-through b) The software product being examined c) Standards that are in effect for the acquisition, supply, development, operation, and/or maintenance of the software product Input to the walk-through may also include the following: d) Any regulations, standards, guidelines, plans, and procedures against which the software product is to be inspected e) Anomaly categories The walk-through shall be considered complete when a) The entire software product has been examined b) Recommendations and required actions have been recorded c) The walk-through output has been completed 9.3 Inspection A static analysis technique that relies on visual examination of development products to detect errors, violations of development standards, and other problems. Types include code inspection; design inspection, Architectural inspections, Test ware inspections etc. The participants in Inspections assume one or more of the following roles: a) Inspection leader b) Recorder c) Reader d) Author e) Inspector All participants in the review are inspectors. The author shall not act as inspection leader and should not act as reader or recorder. Other roles may be shared among the team members. Individual participants may act in more than one role. Individuals holding management positions over any member of the inspection team shall not participate in the inspection. Input to the inspection shall include the following: a) A statement of objectives for the inspection b) The software product to be inspected c) Documented inspection procedure d) Inspection reporting forms e) Current anomalies or issues list Input to the inspection may also include the following: f) Inspection checklists g) Any regulations, standards, guidelines, plans, and procedures against which the software product is to be inspected h) Hardware product specifications i) Hardware performance data j) Anomaly categories The individuals may make additional reference material available responsible for the software product when requested by the inspection leader. The purpose of the exit criteria is to bring an unambiguous closure to the inspection meeting. The exit decision shall determine if the software product meets the inspection exit criteria and shall prescribe any appropriate rework and verification. Specifically, the inspection team shall identify the software product disposition as one of the following: a) Accept with no or minor rework. The software product is accepted as is or with only minor rework. (For example, that would require no further verification). b) Accept with rework verification. The software product is to be accepted after the inspection leader or a designated member of the inspection team (other than the author) verifies rework. c) Re-inspect. Schedule a re-inspection to verify rework. At a minimum, a re-inspection shall examine the software product areas changed to resolve anomalies identified in the last inspection, as well as side effects of those changes. 10. Testing Types and Techniques Testing types Testing types refer to different approaches towards testing a computer program, system or product. The two types of testing are black box testing and white box testing, which would both be discussed in detail in this chapter. Another type, termed as gray box testing or hybrid testing is evolving presently and it combines the features of the two types. Testing Techniques Testing techniques refer to different methods of testing particular features a computer program, system or product. Each testing type has its own testing techniques while some techniques combine the feature of both types. Some techniques are
Some of these and many others would be discussed in the later sections of this chapter. Difference between Testing Types and Testing Techniques Testing types deal with what aspect of the computer software would be tested, while testing techniques deal with how a specific part of the software would be tested. That is, testing types mean whether we are testing the function or the structure of the software. In other words, we may test each function of the software to see if it is operational or we may test the internal components of the software to check if its internal workings are according to specification. On the other hand, ‘Testing technique’ means what methods or ways would be applied or calculations would be done to test a particular feature of a software (Sometimes we test the interfaces, sometimes we test the segments, sometimes loops etc.) How to Choose a Black Box or White Box Test? White box testing is concerned only with testing the software product; it cannot guarantee that the complete specification has been implemented. Black box testing is concerned only with testing the specification; it cannot guarantee that all parts of the implementation have been tested. Thus black box testing is testing against the specification and will discover faults of omission, indicating that part of the specification has not been fulfilled. White box testing is testing against the implementation and will discover faults of commission, indicating that part of the implementation is faulty. In order to completely test a software product both black and white box testing are required. White box testing is much more expensive (In terms of resources and time) than black box testing. It requires the source code to be produced before the tests can be planned and is much more laborious in the determination of suitable input data and the determination if the software is or is not correct. It is advised to start test planning with a black box testing approach as soon as the specification is available. White box tests are to be planned as soon as the Low Level Design (LLD) is complete. The Low Level Design will address all the algorithms and coding style. The paths should then be checked against the black box test plan and any additional required test cases should be determined and applied. 10.1 White Box Testing What is WBT? White box testing involves looking at the structure of the code. When you know the internal structure of a product, tests can be conducted to ensure that the internal operations performed according to the specification. And all internal components have been adequately exercised. In other word WBT tends to involve the coverage of the specification in the code. Code coverage is defined in six types as listed below.
What do we do in WBT? In WBT, we use the control structure of the procedural design to derive test cases. Using WBT methods a tester can derive the test cases that · Guarantee that all independent paths within a module have been exercised at least once. · Exercise all logical decisions on their true and false values. · Execute all loops at their boundaries and within their operational bounds · Exercise internal data structures to ensure their validity. White box testing (WBT) is also called Structural or Glass box testing. Why WBT? We do WBT because Black box testing is unlikely to uncover numerous sorts of defects in the program. These defects can be of the following nature: · Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed. Error tend to creep into our work when we design and implement functions, conditions or controls that are out of the program · The logical flow of the program is sometimes counterintuitive, meaning that our unconscious assumptions about flow of control and data may lead to design errors that are uncovered only when path testing starts. · Typographical errors are random, some of which will be uncovered by syntax checking mechanisms but others will go undetected until testing begins. Skills Required Talking theoretically, all we need to do in WBT is to define all logical paths, develop test cases to exercise them and evaluate results i.e. generate test cases to exercise the program logic exhaustively. For this we need to know the program well i.e. We should know the specification and the code to be tested; related documents should be available too us .We must be able to tell the expected status of the program versus the actual status found at any point during the testing process. Limitations Unfortunately in WBT, exhaustive testing of a code presents certain logistical problems. Even for small programs, the number of possible logical paths can be very large. For instance, a 100 line C Language program that contains two nested loops executing 1 to 20 times depending upon some initial input after some basic data declaration. Inside the interior loop four if-then-else constructs are required. Then there are approximately 1014 logical paths that are to be exercised to test the program exhaustively. Which means that a magic test processor developing a single test case, execute it and evaluate results in one millisecond would require 3170 years working continuously for this exhaustive testing which is certainly impractical. Exhaustive WBT is impossible for large software systems. But that doesn’t mean WBT should be considered as impractical. Limited WBT in which a limited no. of important logical paths are selected and exercised and important data structures are probed for validity, is both practical and WBT. It is suggested that white and black box testing techniques can be coupled to provide an approach that that validates the software interface selectively ensuring the correction of internal working of the software. Tools used for White Box testing: Few Test automation tool vendors offer white box testing tools which: 1) Provide run-time error and memory leak detection; 2) Record the exact amount of time the application spends in any given block of code for the purpose of finding inefficient code bottlenecks; and 3) Pinpoint areas of the application that have and have not been executed. 10.1.1 Basis Path Testing Basis path testing is a white box testing technique first proposed by Tom McCabe. The Basis path method enables to derive a logical complexity measure of a procedural design and use this measure as a guide for defining a basis set of execution paths. Test Cases derived to exercise the basis set are guaranteed to execute every statement in the program at least one time during testing. 10.1.2 Flow Graph Notation The flow graph depicts logical control flow using a diagrammatic notation. Each structured construct has a corresponding flow graph symbol. 10.1.3 Cyclomatic Complexity Cyclomatic complexity is a software metric that provides a quantitative measure of the logical complexity of a program. When used in the context of a basis path testing method, the value computed for Cyclomatic complexity defines the number for independent paths in the basis set of a program and provides us an upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once. An independent path is any path through the program that introduces at least one new set of processing statements or a new condition. Computing Cyclomatic Complexity Cyclomatic complexity has a foundation in graph theory and provides us with extremely useful software metric. Complexity is computed in one of the three ways: 1. The number of regions of the flow graph corresponds to the Cyclomatic complexity. 2. Cyclomatic complexity, V(G), for a flow graph, G is defined as V (G) = E-N+2 Where E, is the number of flow graph edges, N is the number of flow graph nodes. 3. Cyclomatic complexity, V (G) for a flow graph, G is also defined as: V (G) = P+1 Where P is the number of predicate nodes contained in the flow graph G. 10.1.4 Graph Matrices The procedure for deriving the flow graph and even determining a set of basis paths is amenable to mechanization. To develop a software tool that assists in basis path testing, a data structure, called a graph matrix can be quite useful. A Graph Matrix is a square matrix whose size is equal to the number of nodes on the flow graph. Each row and column corresponds to an identified node, and matrix entries correspond to connections between nodes. 10.1.5 Control Structure Testing Described below are some of the variations of Control Structure Testing. Condition Testing Condition testing is a test case design method that exercises the logical conditions contained in a program module. Data Flow Testing The data flow testing method selects test paths of a program according to the locations of definitions and uses of variables in the program. 10.1.6 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Requirement No (In RD) | Requirement | Test Case No |
| | | |
What this Traceability Matrix provides you is the coverage of Testing. Keep filling in the Traceability matrix when you complete writing test case’s for each requirement.
12. Validation Phase
The Validation Phase falls into picture after the software is ready or when the code is being written. There are various techniques and testing types that can be appropriately used while performing the testing activities. Let us examine a few of them.
12.1 Unit Testing
This is a typical scenario of Manual Unit Testing activity-
A Unit is allocated to a Programmer for programming. Programmer has to use ‘Functional Specifications’ document as input for his work.
Programmer prepares ‘Program Specifications’ for his Unit from the Functional Specifications. Program Specifications describe the programming approach, coding tips for the Unit’s coding.
Using these ‘Program specifications’ as input, Programmer prepares ‘Unit Test Cases’ document for that particular Unit. A ‘Unit Test Cases Checklist’ may be used to check the completeness of Unit Test Cases document.
‘Program Specifications’ and ‘Unit Test Cases’ are reviewed and approved by Quality Assurance Analyst or by peer programmer.
The programmer implements some functionality for the system to be developed. The same is tested by referring the unit test cases. While testing that functionality if any defects have been found, they are recorded using the defect logging tool whichever is applicable. The programmer fixes the bugs found and tests the same for any errors.
Stubs and Drivers
A software application is made up of a number of ‘Units’, where output of one ‘Unit’ goes as an ‘Input’ of another Unit. e.g. A ‘Sales Order Printing’ program takes a ‘Sales Order’ as an input, which is actually an output of ‘Sales Order Creation’ program.
Due to such interfaces, independent testing of a Unit becomes impossible. But that is what we want to do; we want to test a Unit in isolation! So here we use ‘Stub’ and ‘Driver.
A ‘Driver’ is a piece of software that drives (invokes) the Unit being tested. A driver creates necessary ‘Inputs’ required for the Unit and then invokes the Unit.
A Unit may reference another Unit in its logic. A ‘Stub’ takes place of such subordinate unit during the Unit Testing. A ‘Stub’ is a piece of software that works similar to a unit which is referenced by the Unit being tested, but it is much simpler that the actual unit. A Stub works as a ‘Stand-in’ for the subordinate unit and provides the minimum required behavior for that unit.
Programmer needs to create such ‘Drivers’ and ‘Stubs’ for carrying out Unit Testing.
Both the Driver and the Stub are kept at a minimum level of complexity, so that they do not induce any errors while testing the Unit in question.
Example - For Unit Testing of ‘Sales Order Printing’ program, a ‘Driver’ program will have the code which will create Sales Order records using hardcoded data and then call ‘Sales Order Printing’ program. Suppose this printing program uses another unit which calculates Sales discounts by some complex calculations. Then call to this unit will be replaced by a ‘Stub’, which will simply return fix discount data.
Unit Test Cases
It must be clear by now that preparing Unit Test Cases document (referred to as UTC hereafter) is an important task in Unit Testing activity. Having an UTC, which is complete with every possible test case, leads to complete Unit Testing and thus gives an assurance of defect-free Unit at the end of Unit Testing stage. So let us discuss about how to prepare a UTC.
Think of following aspects while preparing Unit Test Cases –
v Expected Functionality: Write test cases against each functionality that is expected to be provided from the Unit being developed.
e.g. If an SQL script contains commands for creating one table and altering another table then test cases should be written for testing creation of one table and alteration of another.
It is important that User Requirements should be traceable to Functional Specifications, Functional Specifications be traceable to Program Specifications and Program Specifications be traceable to Unit Test Cases. Maintaining such traceability ensures that the application fulfills User Requirements.
v Input values:
o Every input value: Write test cases for each of the inputs accepted by the Unit.
e.g. If a Data Entry Form has 10 fields on it, write test cases for all 10 fields.
o Validation of input: Every input has certain validation rule associated with it. Write test cases to validate this rule. Also, there can be cross-field validations in which one field is enabled depending upon input of another field. Test cases for these should not be missed.
e.g. A combo box or list box has a valid set of values associated with it.
A numeric field may accept only positive values.
An email address field must have ampersand (@) and period (.) in it.
A ‘Sales tax code’ entered by user must belong to the ‘State’ specified by the user.
o Boundary conditions: Inputs often have minimum and maximum possible values. Do not forget to write test cases for them.
e.g. A field that accepts ‘percentage’ on a Data Entry Form should be able to accept inputs only from 1 to 100.
o Limitations of data types: Variables that hold the data have their value limits depending upon their data types. In case of computed fields, it is very important to write cases to arrive at an upper limit value of the variables.
o Computations: If any calculations are involved in the processing, write test cases to check the arithmetic expressions with all possible combinations of values.
v Output values: Write test cases to generate scenarios, which will produce all types of output values that are expected from the Unit.
e.g. A Report can display one set of data if user chooses a particular option and another set of data if user chooses a different option. Write test cases to check each of these outputs. When the output is a result of some calculations being performed or some formulae being used, then approximations play a major role and must be checked.
v Screen / Report Layout: Screen Layout or web page layout and Report layout must be tested against the requirements. It should not happen that the screen or the report looks beautiful and perfect, but user wanted something entirely different! It should ensure that pages and screens are consistent.
v Path coverage: A Unit may have conditional processing which results in various paths the control can traverse through. Test case must be written for each of these paths.
v Assumptions: A Unit may assume certain things for it to function. For example, a Unit may need a database to be open. Then test case must be written to check that the Unit reports error if such assumptions are not met.
v Transactions: In case of database applications, it is important to make sure that transactions are properly designed and no way inconsistent data gets saved in the database.
v Abnormal terminations: Behavior of the Unit in case of abnormal termination should be tested.
v Error messages: Error messages should be short, precise and self-explanatory. They should be properly phrased and should be free of grammatical mistakes.
UTC Document
Given below is a simple format for UTC document.
| Test Case No. | Test Case purpose | Procedure | Expected Result | Actual result |
| ID which can be referred to in other documents like ‘Traceability Matrix’, Root Cause Analysis of Defects etc. | What to test | How to test | What should happen | What actually happened? This column can be omitted when Defect Recording Tool is used. |
Note that as this is a sample, we have not provided columns for Pass/Fail and Remarks.
Example:
Let us say we want to write UTC for a Data Entry Form below:
Given below are some of the Unit Test Cases for the above Form:
| Test Case purpose | Procedure | Expected Result | Actual result | |
| 1 | Item no. to start by ‘A’ or ‘B’. | 1.Create a new record. 2.Type Item no. starting with ‘A’. 3.Type item no. starting with ‘B’. 4.Type item no. starting with any character other than ‘A’ and ‘B’. | 2,3. Should get accepted and control should move to next field. 4. Should not get accepted. An error message should be displayed and control should remain in Item no. field. | |
| 2. | Item Price to be between 1000 to 2000 if Item no. starts with ‘A’. | 1.Create a new record with Item no. starting with ‘A’. 2.Specify price <> 3.Specify price >2000. 4.Specify price = 1000. 5.Specify price = 2000. 6.Specify price between 1000 and 2000. | 2,3.Error should get displayed and control should remain in Price field. 4,5,6.Should get accepted and control should move to next field. | |
| | | | | |
UTC Checklist
UTC checklist may be used while reviewing the UTC prepared by the programmer. As any other checklist, it contains a list of questions, which can be answered as either a ‘Yes’ or a ‘No’. The ‘Aspects’ list given in Section 4.3 above can be referred to while preparing UTC checklist.
e.g. Given below are some of the checkpoints in UTC checklist –
- Are test cases present for all form field validations?
- Are boundary conditions considered?
- Are Error messages properly phrased?
Defect Recording
Defect Recording can be done on the same document of UTC, in the column of ‘Expected Results’. This column can be duplicated for the next iterations of Unit Testing.
Defect Recording can also be done using some tools like Bugzilla, in which defects are stored in the database.
Defect Recording needs to be done with care. It should be able to indicate the problem in clear, unambiguous manner, and reproducing of the defects should be easily possible from the defect information.
Conclusion
Exhaustive Unit Testing filters out the defects at an early stage in the Development Life Cycle. It proves to be cost effective and improves Quality of the Software before the smaller pieces are put together to form an application as a whole. Unit Testing should be done sincerely and meticulously, the efforts are paid well in the long run.
12.2 Integration Testing
Integration testing is a systematic technique for constructing the program structure while at the same time conducting tests to uncover errors associated with interfacing. The objective is to take unit tested components and build a program structure that has been dictated by design.
Usually, the following methods of Integration testing are followed:
1. Top-down Integration approach.
2. Bottom-up Integration approach.
12.2.1 Top-Down Integration
Top-down integration testing is an incremental approach to construction of program structure. Modules are integrated by moving downward through the control hierarchy, beginning with the main control module. Modules subordinate to the main control module are incorporated into the structure in either a depth-first or breadth-first manner.
- The Integration process is performed in a series of five steps:
- The main control module is used as a test driver and stubs are substituted for all components directly subordinate to the main control module.
- Depending on the integration approach selected subordinate stubs are replaced one at a time with actual components.
- Tests are conducted as each component is integrated.
- On completion of each set of tests, stub is replaced with the real component.
- Regression testing may be conducted to ensure that new errors have not been introduced.
12.2.2 Bottom-Up Integration
Bottom-up integration testing begins construction and testing with atomic modules (i.e. components at the lowest levels in the program structure). Because components are integrated from the bottom up, processing required for components subordinate to a given level is always available and the need for stubs is eliminated.
- A Bottom-up integration strategy may be implemented with the following steps:
- Low level components are combined into clusters that perform a specific software sub function.
- A driver is written to coordinate test case input and output.
- The cluster is tested.
Drivers are removed and clusters are combined moving upward in the program structure.
12.3 System Testing
System testing concentrates on testing the complete system with a variety of techniques and methods. System Testing comes into picture after the Unit and Integration Tests.
12.3.1 Compatibility Testing
Compatibility Testing concentrates on testing whether the given application goes well with third party tools, software or hardware platform.
For example, you have developed a web application. The major compatibility issue is, the web site should work well in various browsers. Similarly when you develop applications on one platform, you need to check if the application works on other operating systems as well. This is the main goal of Compatibility Testing.
Before you begin compatibility tests, our sincere suggestion is that you should have a cross reference matrix between various software’s, hardware based on the application requirements. For example, let us suppose you are testing a web application. A sample list can be as follows:
| Hardware | Software | Operating System |
| Pentium – II, 128 MB RAM | IE 4.x, Opera, Netscape | Windows 95 |
| Pentium – III, 256 MB RAM | IE 5.x, Netscape | Windows XP |
| Pentium – IV, 512 MB RAM | Mozilla | Linux |
Compatibility tests are also performed for various client/server based applications where the hardware changes from client to client.
Compatibility Testing is very crucial to organizations developing their own products. The products have to be checked for compliance with the competitors of the third party tools, hardware, or software platform. E.g. A Call center product has been built for a solution with X product but there is a client interested in using it with Y product; then the issue of compatibility arises. It is of importance that the product is compatible with varying platforms. Within the same platform, the organization has to be watchful that with each new release the product has to be tested for compatibility.
A good way to keep up with this would be to have a few resources assigned along with their routine tasks to keep updated about such compatibility issues and plan for testing when and if the need arises.
By the above example it is not intended that companies which are not developing products do not have to cater for this type of testing. There case is equally existent, if an application uses standard software then would it be able to run successfully with the newer versions too? Or if a website is running on IE or Netscape, what will happen when it is opened through Opera or Mozilla. Here again it is best to keep these issues in mind and plan for compatibility testing in parallel to avoid any catastrophic failures and delays.
12.3.2 Recovery Testing
Recovery testing is a system test that focuses the software to fall in a variety of ways and verifies that recovery is properly performed. If it is automatic recovery then re-initialization, check pointing mechanisms, data recovery and restart should be evaluated for correctness. If recovery requires human intervention, the mean-time-to-repair (MTTR) is evaluated to determine whether it is within acceptable limits.
12.3.3 Usability Testing
Usability is the degree to which a user can easily learn and use a product to achieve a goal. Usability testing is the system testing which attempts to find any human-factor problems. A simpler description is testing the software from a users’ point of view. Essentially it means testing software to prove/ensure that it is user-friendly, as distinct from testing the functionality of the software. In practical terms it includes ergonomic considerations, screen design, standardization etc.
The idea behind usability testing is to have actual users perform the tasks for which the product was designed. If they can't do the tasks or if they have difficulty performing the tasks, the UI is not adequate and should be redesigned. It should be remembered that usability testing is just one of the many techniques that serve as a basis for evaluating the UI in a user-centered approach. Other techniques for evaluating a UI include inspection methods such as heuristic evaluations, expert reviews, card-sorting, matching test or Icon intuitiveness evaluation, cognitive walkthroughs. Confusion regarding usage of the term can be avoided if we use ‘usability evaluation’ for the generic term and reserve ‘usability testing’ for the specific evaluation method based on user performance. Heuristic Evaluation and Usability Inspection or cognitive walkthrough does not involve real users.
It often involves building prototypes of parts of the user interface, having representative users perform representative tasks and seeing if the appropriate users can perform the tasks. In other techniques such as the inspection methods, it is not performance, but someone's opinion of how users might perform that is offered as evidence that the UI is acceptable or not. This distinction between performance and opinion about performance is crucial. Opinions are subjective. Whether a sample of users can accomplish what they want or not is objective. Under many circumstances it is more useful to find out if users can do what they want to do rather than asking someone.
Performing the test
- Get a person who fits the user profile. Make sure that you are not getting someone who has worked on it.
- Sit them down in front of a computer, give them the application, and tell them a small scenario, like: “Thank you for volunteering making it easier for users to find what they are looking for. We would like you to answer several questions. There is no right or wrong answers. What we want to learn is why you make the choices you do, what is confusing, why choose one thing and not another, etc. Just talk us through your search and let us know what you are thinking. We have a recorder which is going to capture what you say, so you will have to tell us what you are clicking on as you also tell us what you are thinking. Also think aloud when you are stuck somewhere”
- Now don’t speak anything. Sounds easy, but see if you actually can shut up.
- Watch them use the application. If they ask you something, tell them you're not there. Then shut up again.
- Start noting all the things you will have to change.
- Afterwards ask them what they thought and note them down.
- Once the whole thing is done thank the volunteer.
tools available for Usability Testing
· ErgoLight Usability Software offers comprehensive GUI quality solutions for the professional Windows application developer. ErgoLight offers solutions for developers of Windows applications for testing and evaluating their usability.
· WebMetrics Tool Suite from National Institute of Standards and Technology contains rapid, remote, and automated tools to help in producing usable web sites. The Web Static Analyzer Tool (WebSAT) checks the html of a web page against numerous usability guidelines. The output from WebSAT consists of identification of potential usability problems, which should be investigated further through user testing. The Web Category Analysis Tool (WebCAT) lets the usability engineer quickly construct and conduct a simple category analysis across the web.
· Bobby from Center for Applied Special Technology is a web-based public service offered by CAST that analyzes web pages for their accessibility to people with disabilities as well as their compatibility with various browsers.
· DRUM from Serco Usability Services is a tool, which has been developed by close cooperation between Human Factors professionals and software engineers to provide a broad range of support for video-assisted observational studies.
· Form Testing Suite from Corporate Research and Advanced Development, Digital Equipment Corporation Provides a test suite developed to test various web browsers. The test results section provides a description of the tests.
· The
· Usability Sciences Corporation has a usability lab in
· UserWorks, Inc. (formerly Man-Made Systems) is a consulting firm in the
· Lodestone Research has usability-testing laboratory with state of the art audio and visual recording and testing equipment. All equipment has been designed to be portable so that it can be taken on the road. The lab consists of a test room and an observation/control room that can seat as many as ten observers. A-V equipment includes two (soon to be 3) fully controllable SVHS cameras, capture/feed capabilities for test participant's PC via scan converter and direct split signal (to VGA "slave" monitors in observation room), up to eight video monitors and four VCA monitors for observer viewing, mixing/editing equipment, and "wiretap" capabilities to monitor and record both sides of telephone conversation (e.g., if participant calls customer support).
· Online Computer Library Center, Inc provides insight into the usability test laboratory. It gives an overview of the infrastructure as well as the process being used in the laboratory.
End goals of Usability Testing
To summarize the goals, it can be said that it makes the software more user friendly. The end result will be:
- Better quality software.
- Software is easier to use.
- Software is more readily accepted by users.
- Shortens the learning curve for new users.
12.3.4 Security Testing
Security testing attempts to verify that protection mechanisms built into a system will, in fact, protect it from improper penetration. During Security testing, password cracking, unauthorized entry into the software, network security are all taken into consideration.
12.3.5 Stress Testing
Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume. The following types of tests may be conducted during stress testing;
· Special tests may be designed that generate ten interrupts per second, when one or two is the average rate.
· Input data rates may be increases by an order of magnitude to determine how input functions will respond.
· Test Cases that require maximum memory or other resources.
· Test Cases that may cause excessive hunting for disk-resident data.
· Test Cases that my cause thrashing in a virtual operating system.
12.3.6 Performance Testing
Performance testing of a Web site is basically the process of understanding how the Web application and its operating environment respond at various user load levels. In general, we want to measure the Response Time, Throughput, and Utilization of the Web site while simulating attempts by virtual users to simultaneously access the site. One of the main objectives of performance testing is to maintain a Web site with low response time, high throughput, and low utilization.
Response Time
Response Time is the delay experienced when a request is made to the server and the server's response to the client is received. It is usually measured in units of time, such as seconds or milliseconds. Generally speaking, Response Time increases as the inverse of unutilized capacity. It increases slowly at low levels of user load, but increases rapidly as capacity is utilized. Figure 1 demonstrates such typical characteristics of Response Time versus user load.
Figure1. Typical characteristics of latency versus user load
The sudden increase in response time is often caused by the maximum utilization of one or more system resources. For example, most Web servers can be configured to start up a fixed number of threads to handle concurrent user requests. If the number of concurrent requests is greater than the number of threads available, any incoming requests will be placed in a queue and will wait for their turn to be processed. Any time spent in a queue naturally adds extra wait time to the overall Response Time.
To better understand what Response Time means in a typical Web farm, we can divide response time into many segments and categorize these segments into two major types: network response time and application response time. Network response time refers to the time it takes for data to travel from one server to another. Application response time is the time required for data to be processed within a server. Figure 2 shows the different response time in the entire process of a typical Web request.
Figure 2 shows the different response time in the entire process of a typical Web request.
Total Response Time = (N1 + N2 + N3 + N4) + (A1 + A2 + A3)
Where Nx represents the network Response Time and Ax represents the application Response Time.
In general, the Response Time is mainly constrained by N1 and N4. This Response Time represents the method your clients are using to access the Internet. In the most common scenario, e-commerce clients access the Internet using relatively slow dial-up connections. Once Internet access is achieved, a client's request will spend an indeterminate amount of time in the Internet cloud shown in Figure 2 as requests and responses are funneled from router to router across the Internet.
To reduce these networks Response Time (N1 and N4), one common solution is to move the servers and/or Web contents closer to the clients. This can be achieved by hosting your farm of servers or replicating your Web contents with major Internet hosting providers who have redundant high-speed connections to major public and private Internet exchange points, thus reducing the number of network routing hops between the clients and the servers.
Network Response Times N2 and N3 usually depend on the performance of the switching equipment in the server farm. When traffic to the back-end database grows, consider upgrading the switches and network adapters to boost performance.
Reducing application Response Times (A1, A2, and A3) is an art form unto itself because the complexity of server applications can make analyzing performance data and performance tuning quite challenging. Typically, multiple software components interact on the server to service a given request. Response time can be introduced by any of the components. That said, there are ways you can approach the problem:
· First, your application design should minimize round trips wherever possible. Multiple round trips (client to server or application to database) multiply transmission and resource acquisition Response time. Use a single round trip wherever possible.
· You can optimize many server components to improve performance for your configuration. Database tuning is one of the most important areas on which to focus. Optimize stored procedures and indexes.
· Look for contention among threads or components competing for common resources. There are several methods you can use to identify contention bottlenecks. Depending on the specific problem, eliminating a resource contention bottleneck may involve restructuring your code, applying service packs, or upgrading components on your server. Not all resource contention problems can be completely eliminated, but you should strive to reduce them wherever possible. They can become bottlenecks for the entire system.
· Finally, to increase capacity, you may want to upgrade the server hardware (scaling up), if system resources such as CPU or memory are stretched out and have become the bottleneck. Using multiple servers as a cluster (scaling out) may help to lessen the load on an individual server, thus improving system performance and reducing application latencies.
Throughput
Throughput refers to the number of client requests processed within a certain unit of time. Typically, the unit of measurement is requests per second or pages per second. From a marketing perspective, throughput may also be measured in terms of visitors per day or page views per day, although smaller time units are more useful for performance testing because applications typically see peak loads of several times the average load in a day.
As one of the most useful metrics, the throughput of a Web site is often measured and analyzed at different stages of the design, develop, and deploy cycle. For example, in the process of capacity planning, throughput is one of the key parameters for determining the hardware and system requirements of a Web site. Throughput also plays an important role in identifying performance bottlenecks and improving application and system performance. Whether a Web farm uses a single server or multiple servers, throughput statistics show similar characteristics in reactions to various user load levels. Figure 3 demonstrates such typical characteristics of throughput versus user load.
Figure 3. Typical characteristics of throughput versus user load
As Figure 3 illustrates, the throughput of a typical Web site increases proportionally at the initial stages of increasing load. However, due to limited system resources, throughput cannot be increased indefinitely. It will eventually reach a peak, and the overall performance of the site will start degrading with increased load. Maximum throughput, illustrated by the peak of the graph in Figure 3, is the maximum number of user requests that can be supported concurrently by the site in the given unit of time.
Note that it is sometimes confusing to compare the throughput metrics for your Web site to the published metrics of other sites. The value of maximum throughput varies from site to site. It mainly depends on the complexity of the application. For example, a Web site consisting largely of static HTML pages may be able to serve many more requests per second than a site serving dynamic pages. As with any statistic, throughput metrics can be manipulated by selectively ignoring some of the data. For example, in your measurements, you may have included separate data for all the supporting files on a page, such as graphic files. Another site's published measurements might consider the overall page as one unit. As a result, throughput values are most useful for comparisons within the same site, using a common measuring methodology and set of metrics.
In many ways, throughput and Response time are related, as different approaches to thinking about the same problem. In general, sites with high latency will have low throughput. If you want to improve your throughput, you should analyze the same criteria as you would to reduce latency. Also, measurement of throughput without consideration of latency is misleading because latency often rises under load before throughput peaks. This means that peak throughput may occur at a latency that is unacceptable from an application usability standpoint. This suggests that Performance reports include a cut-off value for Response time, such as:250 requests/second @ 5 seconds maximum Response time
Utilization
Utilization refers to the usage level of different system resources, such as the server's CPU(s), memory, network bandwidth, and so forth. It is usually measured as a percentage of the maximum available level of the specific resource. Utilization versus user load for a Web server typically produces a curve, as shown in Figure 4.
Figure 4. Typical characteristics of utilization versus user load
As Figure 4 illustrates, utilization usually increases proportionally to increasing user load. However, it will top off and remain at a constant when the load continues to build up.
If the specific system resource tops off at 100-percent utilization, it's very likely that this resource has become the performance bottleneck of the site. Upgrading the resource with higher capacity would allow greater throughput and lower latency—thus better performance. If the measured resource does not top off close to 100-percent utilization, it is probably because one or more of the other system resources have already reached their maximum usage levels. They have become the performance bottleneck of the site.
To locate the bottleneck, you may need to go through a long and painstaking process of running performance tests against each of the suspected resources, and then verifying if performance is improved by increasing the capacity of the resource. In many cases, performance of the site will start deteriorating to an unacceptable level well before the major system resources, such as CPU and memory, are maximized. For example, Figure 5 illustrates a case where response time rises sharply to 45 seconds when CPU utilization has reached only 60 percent.
Figure 5. An example of Response Time versus utilization
As Figure 5 demonstrates, monitoring the CPU or memory utilization alone may not always indicate the true capacity level of the server farm with acceptable performance.
While most traditional applications are designed to respond to a single user at any time, most Web applications are expected to support a wide range of concurrent users, from a dozen to a couple thousand or more. As a result, performance testing has become a critical component in the process of deploying a Web application. It has proven to be most useful in (but not limited to) the following areas:
· Capacity planning
· Bug fixing
Capacity Planning
How do you know if your server configuration is sufficient to support two million visitors per day with average response time of less than five seconds? If your company is projecting a business growth of 200 percent over the next two months, how do you know if you need to upgrade your server or add more servers to the Web farm? Can your server and application support a six-fold traffic increase during the Christmas shopping season?
Capacity planning is about being prepared. You need to set the hardware and software requirements of your application so that you'll have sufficient capacity to meet anticipated and unanticipated user load.
One approach in capacity planning is to load-test your application in a testing (staging) server farm. By simulating different load levels on the farm using a Web application performance testing tool such as WAS, you can collect and analyze the test results to better understand the performance characteristics of the application. Performance charts such as those shown in Figures 1, 3, and 4 can then be generated to show the expected Response Time, throughput, and utilization at these load levels.
In addition, you may also want to test the scalability of your application with different hardware configurations. For example, load testing your application on servers with one, two, and four CPUs respectively would help to determine how well the application scales with symmetric multiprocessor (SMP) servers. Likewise, you should load test your application with different numbers of clustered servers to confirm that your application scales well in a cluster environment.
Although performance testing is as important as functional testing, it’s often overlooked .Since the requirements to ensure the performance of the system is not as straightforward as the functionalities of the system, achieving it correctly is more difficult.
The effort of performance testing is addressed in two ways:
- Load testing
- Stress testing
Load testing
Load testing is a much used industry term for the effort of performance testing. Here load means the number of users or the traffic for the system. Load testing is defined as the testing to determine whether the system is capable of handling anticipated number of users or not.
In Load Testing, the virtual users are simulated to exhibit the real user behavior as much as possible. Even the user think time such as how users will take time to think before inputting data will also be emulated. It is carried out to justify whether the system is performing well for the specified limit of load.
For example, Let us say an online-shopping application is anticipating 1000 concurrent user hits at peak period. In addition, the peak period is expected to stay for 12 hrs. Then the system is load tested with 1000 virtual users for 12 hrs. These kinds of tests are carried out in levels: first 1 user, 50 users, and 100 users, 250 users, 500 users and so on till the anticipated limit are reached. The testing effort is closed exactly for 1000 concurrent users.
The objective of load testing is to check whether the system can perform well for specified load. The system may be capable of accommodating more than 1000 concurrent users. But, validating that is not under the scope of load testing. No attempt is made to determine how many more concurrent users the system is capable of servicing. Table 1 illustrates the example specified.
Stress testing
Stress testing is another industry term of performance testing. Though load testing & Stress testing are used synonymously for performance–related efforts, their goal is different.
Unlike load testing where testing is conducted for specified number of users, stress testing is conducted for the number of concurrent users beyond the specified limit. The objective is to identify the maximum number of users the system can handle before breaking down or degrading drastically. Since the aim is to put more stress on system, think time of the user is ignored and the system is exposed to excess load. The goals of load and stress testing are listed in Table 2. Refer to table 3 for the inference drawn through the Performance Testing Efforts.
Let us take the same example of online shopping application to illustrate the objective of stress testing. It determines the maximum number of concurrent users an online system can service which can be beyond 1000 users (specified limit). However, there is a possibility that the maximum load that can be handled by the system may found to be same as the anticipated limit. The Table<##>illustrates the example specified.
Stress testing also determines the behavior of the system as user base increases. It checks whether the system is going to degrade gracefully or crash at a shot when the load goes beyond the specified limit.
Table 1: Load and stress testing of illustrative example
| Types of Testing | Number of Concurrent users | Duration |
| Load Testing | 1 User à 50 Users à100 Users à250 Users à500 Users…………. à1000Users | 12 Hours |
| Stress Testing | 1 User à 50 Users à100 Users à250 Users à500 Users…………. à1000Users àBeyond 1000 Users……….. àMaximum Users | 12 Hours |
Table 2: Goals of load and stress testing
| Types of testing | Goals |
| Load testing |
|
| Stress testing |
|
Table 3: Inference drawn by load and stress testing
| Type of Testing | Inference |
| Load Testing | Whether system Available? If yes, is the available system is stable? |
| Stress Testing | Whether system is Available? If yes, is the available system is stable? If Yes, is it moving towards Unstable state? When the system is going to break down or degrade drastically? |
Conducting performance testing manually is almost impossible. Load and stress tests are carried out with the help of automated tools. Some of the popular tools to automate performance testing are listed in table 4.
Table 4: Load and stress testing tools
| Tools | Vendor |
| LoadRunner | Mercury Interactive Inc |
| Astra load test | Mercury Interactive Inc |
| Silk performer | Segue |
| WebLoad | Radview Software |
| QALoad | CompuWare |
| e-Load | Empirix Software |
| eValid | Software research Inc |
| WebSpray | CAI network |
| TestManager | Rational |
| Web application center test | Microsoft technologies |
| OpenLoad | OpenDemand |
| ANTS | Red Gate Software |
| OpenSTA | Open source |
| Astra Loadtest | Mercury interactive Inc |
| WAPT | Novasoft Inc |
| Sitestress | Webmaster solutions |
| Quatiumpro | Quatium technologies |
| Easy WebLoad | PrimeMail Inc |
Bug Fixing
Some errors may not occur until the application is under high user load. For Example, memory leaks can exacerbate server or application problems sustaining high load. Performance testing helps to detect and fix such problems before launching the application. It is therefore recommended that developers take an active role in performance testing their applications, especially at different major milestones of the development cycle.
12.3.7 Content Management Testing
‘Content Management’ has gained a predominant importance after the Web applications took a major part of our lives. What is Content Management? As the name denotes, it is managing the content. How do they work? Let us take a common example. You are in
Content Management Testing involves:
1. Testing the distribution of the content.
2. Request, Response Time’s.
3. Content display on various browsers and operating systems.
4. Load distribution on the servers.
In fact all the performance related testing should be performed for each version of the web application which uses the content management servers.
12.3.8 Regression Testing
Regression testing as the name suggests is used to test / check the effect of changes made in the code.
Most of the time the testing team is asked to check last minute changes in the code just before making a release to the client, in this situation the testing team needs to check only the affected areas.
So in short for the regression testing the testing team should get the input from the development team about the nature / amount of change in the fix so that testing team can first check the fix and then the side effects of the fix.
In my present organization we too faced the same problem. So we made a regression bucket (this is a simple excel sheet containing the test cases that we need think assure us of bare minimum functionality) this bucket is run every time before the release.
In fact the regression testing is the testing in which maximum automation can be done. The reason being the same set of test cases will be run on different builds multiple times.
But again the extent of automation depends on whether the test cases will remain applicable over the time, In case the automated test cases do not remain applicable for some amount of time then test engineers will end up in wasting time to automate and don’t get enough out of automation.
- What is Regression testing?
Regression Testing is retesting unchanged segments of application. It involves rerunning tests that have been previously executed to ensure that the same results can be achieved currently as were achieved when the segment was last tested.
The selective retesting of a software system that has been modified to ensure that any bugs have been fixed and that no other previously working functions have failed as a result of the reparations and that newly added features have not created problems with previous versions of the software. Also referred to as verification testing, regression testing is initiated after a programmer has attempted to fix a recognized problem or has added source code to a program that may have inadvertently introduced errors. It is a quality control measure to ensure that the newly modified code still complies with its specified requirements and that unmodified code has not been affected by the maintenance activity.
- What do you do during Regression testing?
- Rerunning of previously conducted tests
- Reviewing previously prepared manual procedures
- Comparing the current test results with the previously executed test results
- Rerunning of previously conducted tests
- What are the tools available for Regression testing?
Although the process is simple i.e. the test cases that have been prepared can be used and the expected results are also known, if the process is not automated it can be very time-consuming and tedious operation.
Some of the tools available for regression testing are:
Record and Playback tools – Here the previously executed scripts can be rerun to verify whether the same set of results are obtained. E.g. Rational Robot
- What are the end goals of Regression testing?
- To ensure that the unchanged system segments function properly
- To ensure that the previously prepared manual procedures remain correct after the changes have been made to the application system
- To verify that the data dictionary of data elements that have been changed is correct
- To ensure that the unchanged system segments function properly
Regression testing as the name suggests is used to test / check the effect of changes made in the code.
Most of the time the testing team is asked to check the last minute changes in the code just before making a release to the client, in this situation the testing team needs to check only the affected areas.
So in short for the regression testing the testing team should get the input from the development team about the nature / amount of change in the fix so that testing team can first check the fix and then the affected areas.
In my present organization we too faced the same problem. So we made a regression bucket (this is a simple excel sheet containing the test cases that we need think assure us of bare minimum functionality) this bucket is run every time before the release.
In fact the regression testing is the testing in which maximum automation can be done. The reason being the same set of test cases will be run on different builds multiple times.
But again the extent of automation depends on whether the test cases will remain applicable over the time, In case the automated test cases do not remain applicable for some amount of time then test engineers will end up in wasting time to automate and don’t get enough out of automation.
12.4 Alpha Testing
A software prototype stage when the software is first available for run. Here the software has the core functionalities in it but complete functionality is not aimed at. It would be able to accept inputs and give outputs. Usually the most used functionalities (parts of code) are developed more. The test is conducted at the developer’s site only.
In a software development cycle, depending on the functionalities the number of alpha phases required is laid down in the project plan itself.
During this, the testing is not a through one, since only the prototype of the software is available. Basic installation – uninstallation tests, the completed core functionalities are tested. The functionality complete area of the Alpha stage is got from the project plan document.
Aim
· is to identify any serious errors
· to judge if the indented functionalities are implemented
· to provide to the customer the feel of the software
A through understanding of the product is done now. During this phase, the test plan and test cases for the beta phase (the next stage) is created. The errors reported are documented internally for the testers and developers reference. No issues are usually reported and recorded in any of the defect management/bug trackers
Role of test lead
- Understand the system requirements completely.
- Initiate the preparation of test plan for the beta phase.
Role of the tester
- to provide input while there is still time to make significant changes as the design evolves.
- Report errors to developers
12.5 User Acceptance Testing
User Acceptance testing occurs just before the software is released to the customer. The end-users along with the developers perform the User Acceptance Testing with a certain set of test cases and typical scenarios.
12.6 Installation Testing
Installation testing is often the most under tested area in testing. This type of testing is performed to ensure that all Installed features and options function properly. It is also performed to verify that all necessary components of the application are, indeed, installed.
Installation testing should take care of the following points: -
- To check if while installing product checks for the dependent software / patches say Service pack3.
- The product should check for the version of the same product on the target machine, say the previous version should not be over installed on the newer version.
- Installer should give a default installation path say “C:\programs\.”
- Installer should allow user to install at location other then the default installation path.
- Check if the product can be installed “Over the Network”
- Installation should start automatically when the CD is inserted.
- Installer should give the remove / Repair options.
- When uninstalling, check that all the registry keys, files, Dll, shortcuts, active X components are removed from the system.
- Try to install the software without administrative privileges (login as guest).
- Try installing on different operating system.
Try installing on system having non-compliant configuration such as less memory / RAM / HDD.
12.7 Beta Testing
The Beta testing is conducted at one or more customer sites by the end-user of the software. The beta test is a live application of the software in an environment that cannot be controlled by the developer.
The Software reaches beta stage when most of the functionalities are operating.
The software is tested in customer’s environment, giving user the opportunity to exercise the software, find the errors so that they could be fixed before product release.
Beta testing is a detailed testing and needs to cover all the functionalities of the product and also the dependent functionality testing. It also involves the UI testing and documentation testing. Hence it is essential that this is planned well and the task accomplished. The test plan document has to be prepared before the testing phase is started, which clearly lays down the objectives, scope of test, tasks to be performed and the test matrix which depicts the schedule of testing.
Beta Testing Objectives
· Evaluate software technical content
· Evaluate software ease of use
· Evaluate user documentation draft
· Identify errors
· Report errors/findings
Role of a Test Lead
- Provide Test Instruction Sheet that describes items such as testing objectives, steps to follow, data to enter, functions to invoke.
- Provide feedback forms and comments.
Role of a tester
- Understand the software requirements and the testing objectives.
- Carry out the test cases
Report defects
13. Understanding Exploratory Testing
"Exploratory testing involves simultaneously learning,
planning, running tests, and reporting / troubleshooting
Results."
- Dr. Cem Kaner.
"Exploratory testing is an interactive process of concurrent
product exploration, test design and test execution. To the
extent that the next test we do is influenced by the result
of the last test we did, we are doing exploratory testing.”
- James Bach.
Exploratory testing is defined as simultaneous test design, test execution and bug reporting. In this approach the tester explores the system (finding out what it is and then testing it) without having any prior test cases or test scripts. Because of this reason it also called as ad hoc testing, guerrilla testing or intuitive testing. But there is some difference between them. In operational terms, exploratory testing is an interactive process of concurrent product exploration, test design, and test execution. The outcome of an exploratory testing session is a set of notes about the product, failures found, and a concise record of how the product was tested. When practiced by trained testers, it yields consistently valuable and auditable results. Every tester performs this type of testing at one point or the other. This testing totally depends on the skill and creativity of the tester. Different testers can explore the system in different ways depending on their skills. Thus the tester has a very vital role to play in exploratory testing.
This approach of testing has also been advised by SWEBOK for testing since it might uncover the bugs, which the normal testing might not discover. A systematic approach of exploratory testing can also be used where there is a plan to attack the system under test. This systematic approach of exploring the system is termed Formalized exploratory testing.
Exploratory testing is a powerful approach in the field of testing. Yet this approach has not got the recognition and is often misunderstood and not gained the respect it needs. In many situations it can be more productive than the scripted testing. But the real fact is that all testers do practice this methodology sometime or the other, most often unknowingly!
Exploratory testing believes in concurrent phases of product exploration, test design and test execution. It is categorized under Black-box testing. It is basically a free-style testing approach where you do not begin with the usual procedures of elaborate test plans and test steps. The test plan and strategy is very well in the tester’s mind. The tester asks the right question to the product / application and judges the outcome. During this phase he is actually learning the product as he tests it. It is interactive and creative. A conscious plan by the tester gives good results.
Human beings are unique and think differently, with a new set of ideas emerging. A tester has the basic skills to listen, read, think and report. Exploratory testing is just trying to exploit this and structure it down. The richness of this process is only limited to the breadth and depth of our imagination and the insight into the product under test.
How does it differ from the normal test procedures?
The definition of exploratory testing conveys the difference. In the normal testing style, the test process is planned well in advance before the actual testing begins. Here the test design is separated from the test execution phase. Many a times the test design and test execution is entrusted on different persons.
Exploratory testing should not be confused with the dictionary meaning of “ad-hoc”. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing. The term “exploratory testing”-- by Dr. Cem Kaner, in Testing Computer Software--refers to a sophisticated, systematic, thoughtful approach to ad hoc testing.
What is formalized Exploratory Testing?
A structured and reasoned approach to exploratory testing is termed as Formalized Exploratory Testing. This approach consists of specific tasks, objectives, and deliverables that make it a systematic process.
Using the systematic approach (i.e. the formalize approach) an outline of what to attack first, its scope, the time required to be spent etc is achieved. The approach might be using simple notes to more descriptive charters to some vague scripts. By using the systematic approach the testing can be more organized focusing at the goal to be reached. Thus solving the problem where the pure Exploratory Testing might drift away from the goal.
When we apply Exploratory Planning to Testing, we create Exploratory planning.
The formalized approach used for the Exploratory Testing can vary depending on the various criteria like the resource, time, the knowledge of the application available etc. Depending on these criteria, the approach used to attack the system will also vary. It may involve creating the outlines on the notepad to more sophisticated way by using charters etc. Some of the formal approaches used for Exploratory Testing can be summarized as follows.
ü Identify the application domain.
The exploratory testing can be performed by identifying the application domain. If the tester has good knowledge of domain, then it would be easier to test the system without having any test cases. If the tester were well aware of the domain, it would help analyzing the system faster and better. His knowledge would help in identifying the various workflows that usually exist in that domain. He would also be able to decide what are the different scenarios and which are most critical for that system. Hence he can focus his testing depending on the scenarios required. If a QA lead is trying to assign the tester to a task, it is advisable that the tester identifies the person who has the domain knowledge of that system for ET.
For example, consider software has been built to generate the invoices for its customers depending on the number of the units of power that has been consumed. In such a case exploratory testing can be done by identifying the domain of the application. A tester who has experience of the billing systems for the energy domain would fit better than one who does not have any knowledge. The tester who has knowledge in the application domain knows the terminology used as well the scenarios that would be critical to the system. He would know the ways in which various computations are done. In such a case, tester with good knowledge would be familiar to the terms like to line item, billing rate, billing cycle and the ways in which the computation of invoice would be done. He would explore the system to the best and takes lesser time. If the tester does not have domain knowledge required, then it would take time to understand the various workflows as well the terminology used. He might not be able to focus on critical areas rather focus on the other areas.
ü Identify the purpose.
Another approach to Exploratory Testing is by identifying the purpose of the system i.e. What is that system used for. By identifying the purpose try to analyze to what extent it is used. The effort can be more focused by identifying the purpose.
For example, consider software developed to be used in Medical operations. In such case care should be taken that the software build is 100% defect free. Hence the effort that needs to be focused is more and care should be taken that the various workflows involved are covered.
On the other hand, if the software build is to provide some entertainment then the criticality is lesser. Thus effort that needs to be focused varies. Identifying the purpose of the system or the application to be tested helps to a great extent.
ü Identify the primary and secondary functions.
Primary Function: Any function so important that, in the estimation of a normal user, its inoperability or impairment would render the product unfit for its purpose. A function is primary if you can associate it with the purpose of the product and it is essential to that purpose. Primary functions define the product. For example, the function of adding text to a document in Microsoft Word is certainly so important that the product would be useless without it. Groups of functions, taken together, may constitute a primary function, too. For example, while perhaps no single function on the drawing toolbar of Word would be considered primary, the entire toolbar might be primary. If so, then most of the functions on that toolbar should be operable in order for the product to pass Certification.
Secondary Function or contributing function: Any function that contributes to the utility of the product, but is not a primary function. Even though contributing functions are not primary, their inoperability may be grounds for refusing to grant Certification. For example, users may be technically able to do useful things with a product, even if it has an “Undo” function that never works, but most users will find that intolerable. Such a failure would violate fundamental expectations about how Windows products should work.
Thus by identifying the primary function and secondary functions for the system, testing can be done where more focus and effort can be given to Primary functions compared to the secondary functions.
Example: Consider a web based application developed for online shopping. For such an application we can identify the primary functions and secondary functions and go ahead with Exploratory Testing. The main functionality of that application is that the items selected by the user need to be properly added to the shopping cart and price to be paid is properly calculated. If there is online payment, then security is also an aspect. These can be considered as the primary functions.
Whereas the bulletin board provided or the mail functionality provided are considered as the secondary functions. Thus testing to be performed is more focused at the primary functions rather than on the secondary functions. If the primary functions do not work as required then the main intention of having the application is lost.
ü Identify the workflows.
Identifying the workflows for testing any system without any scripted test cases can be considered as one of the best approaches used. The workflows are nothing but a visual representation of the scenarios as the system would behave for any given input. The workflows can be simple flow charts or Data Flow Diagram’s (DFD) or the something like state diagrams, use cases, models etc. The workflows will also help to identify the scope for that scenario. The workflows would help the tester to keep track of the scenarios for testing. It is suggested that the tester navigates through the application before he starts exploring. It helps the tester in identifying the various possible workflows and issues any found which he is comfortable can be discussed with the concerned team.
Example: Consider a web application used for online shopping. The application has various links on the web page. If tester is trying to test if the items that he is adding to cart are properly being added, then he should know the flow for the same. He should first identify the workflow for such a scenario. He needs to login and then select a category and identify the items and then add the item he would require. Thus without knowing the workflow for such a scenario would not help the tester and in the process loses his time.
In case he is not aware of the system, try to navigate through the application once and get comfortable. Once the application is dully understood, it is easier to test and explore more bugs.
ü Identify the break points.
Break points are the situations where the system starts behaving abnormally. It does not give the output it is supposed to give. So by identifying such situations also testing can be done. Use boundary values or invariance for finding the break points of the application. In most of the cases it is observed that system would work for normal inputs or outputs. Try to give input that might be the ideal situation or the worse situation.
Example: consider an application build to generate the reports for the accounts department of a company depending on the criteria given. In such cases try to select a worse case of report generation for all the employees for their service. The system might not behave normally in the situation.
Try to input a large input file to the application that provides the user to upload and save the data given.
Try to input 500 characters in the text box of the web application.
Thus by trying to identify the extreme conditions or the breakpoints would help the tester to uncover the hidden bugs. Such cases might not be covered in the normal scripted testing. Hence this helps in finding the bugs which might not covered in the normal testing.
ü Check UI against Windows interface etc standards.
The Exploratory Testing can be performed by identifying the User interface standards. There are set standards laid down for the user interfaces that need to be developed. These user standards are nothing but the look and feel aspects of the interfaces the user interacts with. The user should be comfortable with any of the screens that (s)he working. These aspects help the end user to accept the system faster.
Example: For Web application,
o Is the background as per the standards? If the bright background is used, the user might not feel comfortable.
o What is size of the font used?
o Are the buttons of the required size and are they placed in the comfortable location.
o Sometimes the applications are developed to avoid usage of the scroll bar. The content can be seen with out the need to scroll.
By identifying the User standards, define an approach to test because the application developed should be user friendly for the user’s usage. He should feel comfortable while using the system. The more familiar and easier the application for usage, faster the user feels comfortable to the system.
ü Identify expected results.
The tester should know what he is testing for and expected output for the given input. Until and unless the aim of his testing is not known, there is no use of the testing done. Because the tester may not succeed in distinguishing the real error and normal workflow. First he needs to analyze what is the expected output for the scenario he is testing.
Example: Consider software used to provide the user with an interface to search for the employee name in the organization given some of the inputs like the first name or last name or his id etc. For such a scenario, the tester should identify the expected output for any combination of input values. If the input provided does not result in any data and shows a message ”Error not data found”. The tester should not misinterpret this as an error, because this might be as per requirement when no data is found. Instead for a given input, the message shown is “ 404- File not found”, the tester should identify it as an error not a requirement. Thus he should be able to distinguish between an error and normal workflow.
ü Identify the interfaces with other interfaces/external applications.
In the age of component development and maximum reusability, developers try to pick up the already developed components and integrate them. Thus, achieving the desired result in short time. In some cases it would help the tester explore the areas where the components are coupled. The output of one component should be correctly sent to other component. Hence such scenarios or workflows need to be identified and explored more. More focus on some of the shown areas that are more error prone.
Example: consider the online shopping application. The user adds the items to his cart and proceeds to the payments details page. Here the items added, their quantity etc should be properly sent to the next module. If there is any error in any of the data transfer process, the pay details will not be correct and the user will be billed wrong. There by leading to a major error. In such a scenario, more focus is required in the interfaces.
There may be external interfaces, like the application is integrated with another application for the data. In such cases, focus should be more on the interface between the two applications. How data is being passed, is correct data being passed, if there is large data, is transfer of entire data done or is system behaving abnormally when there is large data are few points which should be addressed.
ü Record failures
In exploratory testing, we do the testing without having any documented test cases. If a bug has been found, it is very difficult for us to test it after fix. This is because there are no documented steps to navigate to that particular scenario. Hence we need to keep track of the flow required to reach where a bug has been found. So while testing, it is important that at least the bugs that have been discovered are documented. Hence by recording failures we are able to keep track of work that has been done. This would also help even if the tester who was actually doing ET is not available. Since the document can be referred and list all the bugs that have been reported as well the flows for the same can be identified.
Example: for example consider the online shopping site. A bug has been found while trying to add the items of given category into the cart. If the tester can just document the flow as well as the error that has occurred, it would help the tester himself or any other tester. It can be referred while testing the application after a fix.
ü Document issues and question.
The tester trying to test an application using Exploratory Testing methodology should feel comfortable to test. Hence it is advisable that the tester navigates through the application once and notes any ambiguities or queries he might feel. He can even get the clarification on the workflows he is not comfortable. Hence by documenting all the issues and questions that have been found while scanning or navigating the application can help the tester have testing done without any loss in time.
ü Decompose the main task into smaller tasks .The smaller ones to still smaller activities.
It is always easier to work with the smaller tasks when compared to large tasks. This is very useful in performing Exploratory Testing because lack of test cases might lead us to different routes. By having a smaller task, the scope as well as the boundary are confined which will help the tester to focus on his testing and plan accordingly.
If a big task is taken up for testing, as we explore the system, we might get deviated from our main goal or task. It might be hard to define boundaries if the application is a new one. With smaller tasks, the goal is known and hence the focus and the effort required can be properly planned.
Example: An application that provides email facility. The new users can register and use the application for the email. In such a scenario, the main task itself can be divided into smaller tasks. One task to check if the UI standards are met and it is user friendly. The other task is to test if the new users are able to register with the application and use email facility.
Thus the two tasks are smaller which will the corresponding groups to focus their testing process.
ü Charter- states the goal and the tactics to be used.
Charter Summary:
o “Architecting the Charters” i.e. Test Planning
o Brief information / guidelines on:
o
o What should be tested?
o How to test (approach)?
o What problems to look for?
o Might include guidelines on:
o Tools to use
o Specific Test Techniques or tactics to use
o What risks are involved
o Documents to examine
Desired output from the testing.
A charter can be simple one to more descriptive giving the strategies and outlines for the testing process.
Example: Test the application for report generation.
Or.
Test the application if the report is being generated for the date before 01/01/2000.Use the use cases models for identifying the workflows.
ü Session Based Test Management(SBTM):
Session Based Test Management is a formalized approach that uses the concept of charters and the sessions for performing the ET.
A session is not a test case or bug report. It is the reviewable product produced by chartered and uninterrupted test effort. A session can last from 60 to 90 minutes, but there is no hard and fast rule on the time spent for testing. If a session lasts closer to 45 minutes, we call it a short session. If it lasts closer to two hours, we call it a long session. Each session designed depends on the tester and the charter. After the session is completed, each session is debriefed. The primary objective in the debriefing is to understand and accept the session report. Another objective is to provide feedback and coaching to the tester. The debriefings would help the manager to plan the sessions in future and also to estimate the time required for testing the similar functionality.
The debriefing session is based on agenda called PROOF.
Past: What happened during the session?
Results: What was achieved during the session?
Outlook: What still needs to be done?
Obstacles: What got in the way of good testing?
Feeling: How does the tester feel about all this?
The time spent “on charter” and “on opportunity” is also noted.
A session can be broadly classified into three tasks (namely the TBS metrics).
Session test up: Time required in setting up the application under test.
Test design and execution: Time required scanning the product and test.
Bug investigation and reporting: Time required finding the bugs and reporting to the concerned.
The entire session report consists of these sections:
ü Session charter (includes a mission statement, and areas to be tested)
ü Tester name(s)
ü Date and time started
ü Task breakdown (the TBS metrics)
ü Data files
ü Test notes
ü Issues
ü Bugs
For each session, a session sheet is made. The session sheet consist of the mission of testing, the tester details, duration of testing, the TBS metrics along with the data related to testing like the bugs, notes, issues etc. Data files if any used in the testing would also be enclosed. The data collected during different testing sessions are collected and exported to Excel or some database. All the sessions, the bugs reported etc can be tracked using the unique id associated with each. It is easy for the client as well to keep track. Thus this concept of testers testing in sessions and producing the required output which are trackable is called as Session based test management.
ü Defect Driven Exploratory Testing:
Defect driven exploratory testing is another formalized approach used for ET.
Defect Driven Exploratory Testing (DDET) is a goal-oriented approach focused on the critical areas identified on the Defect analysis study based on Procedural Testing results.
In Procedural testing, the tester executes readily available test cases, which are written based on the requirement specifications. Although the test cases are executed completely, defects were found in the software while doing exploratory testing by just wandering through the product blindly. Just exploring the product without sight was akin to groping in the dark and did not help the testers unearth all the hidden bugs in the software as they were not very sure about the areas that needed to be explored in the software. A reliable basis was needed for exploring the software. Thus Defect driven exploratory testing is an idea of exploring that part of the product based on the results obtained during procedural testing. After analyzing the defects found during the DDET process, it was found that these were the most critical bugs, which were camouflaged in the software and which if present could have made the software ‘Not fit for Use’.
There are some pre requisites for DDET:
o In-depth knowledge of the product.
o Procedural Testing has to be carried out.
o Defect Analysis based on Scripted Tests.
Advantages of DDET:
o Tester has clear clues on the areas to be explored.
Goal oriented approach , hence better results
.
o No wastage of time.
Where does Exploratory Testing Fit:
In general, ET is called for in any situation where it’s not obvious what the next test should be, or when you want to go beyond the obvious tests. More specifically, freestyle Exploratory Testing fits in any of the following situations:
ü You need to provide rapid feedback on a new product or feature.
ü You need to learn the product quickly.
ü You have already tested using scripts, and seek to diversify the testing.
ü You want to find the single most important bug in the shortest time.
ü You want to check the work of another tester by doing a brief independent
Investigation.
ü You want to investigate and isolate a particular defect.
ü You want to investigate the status of a particular risk, in order to evaluate the need for scripted tests in that area.
Pros and Cons:
Pros
ü Does not require extensive documentation.
ü Responsive to changing scenarios.
ü Under tight schedules, testing can be more focused depending on the bug rate or risks.
ü Improved coverage.
Cons
ü Dependent on the tester’s skills.
ü Test tracking not concrete.
More prone to human error.
No contingency plan if the tester is unavailable.
What specifics affect Exploratory Testing?
Here is a list that affects exploratory testing:
· The mission of the particular test session
· The tester skills, talents, preferences
· Available time and other resources
· The status of other testing cycles for the product
· How much the tester knows about the product
The goal of testing needs to be understood first before the work begins. This could be the overall mission of the test project or could be a particular functionality / scenario. The mission is achieved by asking the right questions about the product, designing tests to answer these questions and executing tests to get the answers. Often the tests do not completely answer, in such cases we need to explore. The test procedure is recorded (which could later form part of the scripted testing) and the result status too.
Tester
The tester needs to have a general plan in mind, though may not be very constrained. The tester needs to have the ability to design good test strategy, execute good tests, find important problems and report them. He simply has to think out of the box.
Time
Time available for testing is a critical factor. Time falls short due to the following reasons:
- Many a time in project life cycles, the time and resources required in creating the test strategy, test plan and design, execution and reporting is overlooked. Exploratory testing becomes useful since the test plan, design and execution happen together.
- Also when testing is essential on a short period of notice
- A new feature is implemented
- Change request come in much later stage of the cycle when much of the testing is done with
In such situations exploratory testing comes handy.
Practicing Exploratory Testing
A basic strategy of exploratory testing is to have a general plan of attack, but also allow yourself to deviate from it for short period of time.
In a session of exploratory testing, a set of test ideas, written notes (simple English or scripts) and bug reports are the results. This can be reviewed by the test lead / test manager.
Ø Test Strategy
It is important to identify the scope of the test to be carried. This is dependent on the project approach to testing. The test manager / test lead can decide the scope and convey the same to the test team.
Ø Test design and execution
The tester crafts the test by systematically exploring the product. He defines his approach, analyze the product, and evaluate the risk
Ø Documentation
The written notes / scripts of the tester are reviewed by the test lead / manager. These later form into new test cases or updated test materials.
Where does Exploratory Testing Fit?
Exploratory testing fits almost in any kind of testing projects, projects with rigorous test plans and procedures or in projects where testing is not dictated completely in advance. The situations where exploratory testing could fit in are:
§ Need to provide a rapid feedback on a new feature implementation / product
§ Little product knowledge and need to learn it quickly
§ Product analysis and test planning
§ Done with scripted testing and need to diversify more
§ Improve the quality of existing test scripts
§ Write new scripts
The basic rule is this: exploratory testing is called for any time the next test you should perform is not obvious, or when you want to go beyond the obvious.
A Good Exploratory Tester
Exploratory testing approach relies a lot on the tester himself. The tester actively controls the design of tests as they are performed and uses the information gained to design new and better ideas.
A good exploratory tester should
§ Have the ability to design good tests, execute them and find important problems
§ Should document his ideas and use them in later cycles.
§ Must be able to explain his work
§ Be a careful observer: Exploratory testers are more careful observers than novices and experienced scripted testers. Scripted testers need only observe what the script tells. Exploratory tester must watch for anything unusual or mysterious.
§ Be a critical thinker: They are able to review and explain their logic, looking out for errors in their own thinking.
§ Have diverse ideas so as to make new test cases and improve existing ones.
A good exploratory tester always asks himself, what’s the best test I can perform now? They remain alert for new opportunities.
Advantages
Exploratory testing is advantageous when
- Rapid testing is essential
- Test case development time not available
- Need to cover high risk areas with more inputs
- Need to test software with little knowledge about the specifications
- Develop new test cases or improve the existing
- Drive out monotony of normal step – by - step test execution
Drawbacks
- Skilled tester required
- Difficult to quantize
Balancing Exploratory Testing with Scripted Testing
Exploratory testing relies on the tester and the approach he proceeds with. Pure scripted testing doesn’t undergo much change with time and hence the power fades away. In test scenarios where in repeatability of tests are required, automated scripts having an edge over exploratory approach. Hence it is important to achieve a balance between the two approaches and combine the two to get the best of both.
No comments:
Post a Comment