ISTQB Foundation Level – Complete Reference Guide “From My Eyes”.pdf

ISTQB Foundation Level – Complete Reference Guide “From My Eyes” 24/12/2013

Kapsch CarrierComm Relationship TCS-KAPSCH Mohan Raghunath Sabat Telecom [email protected]

Confidentiality Statement Include the confidentiality statement within the box provided. This has to be legally approved Confidentiality and Non-Disclosure Notice The information contained in this document is confidential and proprietary to TATA Consultancy Services. This information may not be disclosed, duplicated or used for any other purposes. The information contained in this document may not be released in whole or in part outside TCS for any purpose without the express written permission of TATA Consultancy Services.

Tata Code of Conduct We, in our dealings, are self-regulated by a Code of Conduct as enshrined in the Tata Code of Conduct. We request your support in helping us adhere to the Code in letter and spirit. We request that any violation or potential violation of the Code by any person be promptly brought to the notice of the Local Ethics Counselor or the Principal Ethics Counselor or the CEO of TCS. All communication received in this regard will be treated and kept as confidential.

2

Table of Content 1.

Introduction .............................................................................................................................................................. 5 1.1

2.

3.

4.

5.

6.

What is ISTQB? .................................................................................................................................................. 5

Fundamentals of Testing ........................................................................................................................................... 6 2.1

Why is testing necessary? ................................................................................................................................. 6

2.2

Causes of software defects ............................................................................................................................... 6

2.3

What is testing? ................................................................................................................................................ 6

2.4

Objectives of Testing ......................................................................................................................................... 7

2.5

Testing Principles .............................................................................................................................................. 7

2.6

Fundamental Test Process ................................................................................................................................ 7

2.7

How much testing is enough? ........................................................................................................................... 9

Testing throughout the Software life cycle ............................................................................................................. 10 3.1

Software development models ....................................................................................................................... 10

3.2

Test Levels ....................................................................................................................................................... 11

3.3

Test types ........................................................................................................................................................ 12

3.4

Maintenance testing ....................................................................................................................................... 13

Static Techniques .................................................................................................................................................... 14 4.1

Static testing.................................................................................................................................................... 14

4.2

Review Process................................................................................................................................................ 14

4.3

Types of Reviews ............................................................................................................................................. 15

4.4

Static Analysis.................................................................................................................................................. 15

Test Design Techniques........................................................................................................................................... 16 5.1

Test Development Process .............................................................................................................................. 16

5.2

Testing Techniques ......................................................................................................................................... 17

5.3

Specification-based (Black box) technique ..................................................................................................... 18

5.4

Structure-based (White box) technique ......................................................................................................... 19

5.5

Statement coverage and Decision Coverage Problems .................................................................................. 20

5.6

Experience-based technique ........................................................................................................................... 22

Test Management ................................................................................................................................................... 24 6.1

Independent testing ........................................................................................................................................ 24

6.2

Tasks of Test leader and Tester....................................................................................................................... 24 3

7.

8.

6.3

Entry Criteria and Exit Criteria ........................................................................................................................ 24

6.4

Test Estimation................................................................................................................................................ 25

6.5

Test Approach ................................................................................................................................................. 25

6.6

Test Monitoring and control ........................................................................................................................... 25

6.7

Risks in testing................................................................................................................................................. 25

Support for Testing ................................................................................................................................................. 27 7.1

Tools Classification: ......................................................................................................................................... 27

7.2

Tool support for management of testing and tests ........................................................................................ 27

7.3

Tool support for static testing ......................................................................................................................... 28

7.4

Tool support for test specification .................................................................................................................. 29

7.5

Tool support for Test execution and logging .................................................................................................. 29

7.6

Tool support for performance and monitoring .............................................................................................. 30

7.7

Potential benefits of using tools ..................................................................................................................... 30

Reference ................................................................................................................................................................ 31

4

1. Introduction 1.1

What is ISTQB?     

The ISTQB is responsible for the "ISTQB Certified Tester", which is an international qualification scheme and the qualifications in the scheme are based on a syllabus. Indian Testing Board (ITB) is the International Software Testing Qualifications Board (ISTQB) approved National board for India .ITB is responsible for the "ISTQB-Certified-Tester" Certification in India. ISTQB consists of various levels of certification exams – Foundation level and Advanced level. Here we will be dealing with only Foundation level. ISTQB- Foundation Level Certification syllabus consists of 6 chapters, with different levels assigned to each section of the chapters. The exam consists of 40 Questions and passing criteria is at least 65% (i.e. 26 out of 40). The different Levels are as follows: K1 – Remember K2 – Understand K3 – Apply and Analyse

Note: There is a misconception that memorizing the dump questions is enough to pass the exam. This is absolutely wrong because there are no real exam questions in any of the Sample papers. Those papers are only for practice. The ISTQB question papers are taken away after the exam, so none of the sample papers are real exam papers. The questions asked in exams are very tricky.If you try to read some books directly for the exam it would be very boring due to large number of pages. I have prepared this document as I saw, read and cracked the exam, hence the name, “From My eyes”. You can go through this document once and then start reading some good books (I have mentioned one book in the references section) so that you do not take much time understanding those difficult terms and sentences. I have explained the “Sums” asked in paper (5-6 sums are asked every paper) which are difficult to understand from books. Also, you can use this document as a revision material before the exam.

5

2.

Fundamentals of Testing

In this section, we will see the fundamentals of testing: why testing is needed; what is testing; its limitations, objectives and purpose; the principles behind testing; and the process that testers follow. By reading this section you'll gain an understanding of the fundamentals of testing.

2.1

Why is testing necessary?    

2.2

Testing is necessary to check everything and anything that we produce because things can always go wrong- humans make mistakes all the time - it is what we do best. Some of those mistakes are unimportant, but some of them are expensive or dangerous. Software systems are an increasing part of life, from business applications (e.g. banking) to consumer products (e.g. cars). Software that does not work correctly can lead to many problems, including loss of money, time or business reputation, and could even cause injury or death.

Causes of software defects

Human beings tend to commit mistakes (errors), which produces defects (bugs).When this defect/s is executed in software or system it causes a failure. Defects may be caused due to time pressure, complex code, changed technologies, complex infrastructure, etc. Defects in software, systems or documents may result in failures, but not all defects do so.

2.3

What is testing? 

 

Testing is a process rather than a single activity - there are a series of activities involved. A common perception of testing is that it only consists of running tests, i.e. executing the software. This is a part of testing, but not all of the testing activities. Test activities exist before and after test execution. Testing activities include the following: Planning and Control Test Analysis and Design Implementation and Execution Evaluating exit criteria and reporting 6

Test Closure activities These are also the fundamental test processes. NOTE: We will see these activities in coming sections.

2.4

Objectives of Testing   

2.5

To Find Defects. To Prevent Defects. To gain confidence about quality of software and give information for decision making.

Testing Principles

There are seven testing principles as explained below:  











2.6

Testing shows presence of defects: Testing can show that defects are present, but cannot prove that there are no defects. Exhaustive testing is impossible: Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. E.g.: Consider total number of test cases to be tested in a day is 500 and each test case (TC) needs 10 minutes to complete. It is practically impossible to test all these. So we need to prioritize them on basis of risks. Early Testing: To find defects early, testing activities shall be started as early as possible in the software development life cycle. Defect Clustering: Testing effort shall be focused proportionally to the expected and later observed defect density of modules. E.g.: Consider we have 4 modules A, B, C and D. It is not necessary that, all the 4 modules will have 25% defect each. Any module might have large number of defects whereas some might not have any. This is what we mean by defect clustering. So we need to focus on modules with large number of defects. Pesticide paradox: Once a tester finds some defect with the help of a test case, he needs to modify the test case once it has been fixed by the designer. Executing the same test case would not yield anything. Testing is context dependent: Testing is done differently in different contexts. For example, safety-critical software is tested differently from an e-commerce site. Absence-of-errors fallacy: Finding and fixing defects does not help if the system built is unusable and does not fulfil the users’ needs and expectations. For e.g.: Manufacturing and giving a Washing Machine to the customer would not be acceptable, if the customer’s requirement was for a refrigerator.

Fundamental Test Process

This section describes the fundamental test process and activities. These start with test planning and continue through to test closure. We would be discussing the main tasks involved in each part of the test process.  Planning and control 7

Planning: -

Determine the risks and scopes and objectives of testing. Determine the test approach to be used. Schedule all tasks like analysis and design, implementation and execution, and exit criteria.

Control: 

Measure actual progress with the plan. Measure and analyse the results of reviews and testing. Take actions necessary to meet the mission and objective of the project.

Test Analysis and Design General testing objectives are transformed into tangible test conditions and test designs. Review test basis to build test conditions and use these documents to understand what the system should do once built. Identify test conditions based on analysis of test items. Design the tests. Design test environment setup and infrastructure required. This includes testing and support tools.

Test implementation and execution We take test conditions and make them into Test cases (TC) and Test ware and setup the test environment. We transform test conditions into Test cases and procedures. Implementation: Develop and prioritize Test cases. Create test suites for efficient test execution. Test suite is logical collection of Test cases which naturally work together. Execution: Execute test suites and individual Test cases. Log the outcomes and record identities and versions of software. Compare actual v/s expected results. Report discrepancies as incidents. Repeat failed TC for confirmation testing. Do regression testing.



Evaluating Exit criteria and reporting Evaluating exit criteria is the activity where test execution is assessed against the defined objectives. Check test logs against the exit criteria specified in test planning. Assess if more Test cases are needed or if exit criteria needs to be changed. Writing test summary report for stake holders.





Test closure activities Check which plans we actually delivered. Ensure all incident reports have been resolved through defect repair or deferral. Finalize and archive test ware such as scripts, test environment and any other test infrastructure. Handover test ware to maintenance organization. 8

-

2.7

Evaluate how testing went and lessons learned for future releases and projects.

How much testing is enough?

This depends upon the following 2 factors:  

Level of risk. Time and budget of the software.

9

3.

Testing throughout the Software life cycle

This section discusses the most commonly applied software development models, test levels and test types. The four test levels used in this syllabus are:    

3.1

Component(unit) Testing Integration Testing System Testing Acceptance Testing

Software development models Waterfall Model In this model, testing is carried out at the last stage after the complete development life cycle.



User and System requirements >> Global Design >> Detailed Design >> Implementation >> Testing V-Model The V-Model demonstrates the relationships between each phase of the development life cycle and its associated phase of testing i.e. testing happens simultaneously at every stage. So, here we have advantage of finding early defects. Although variants of the V-model exist, a common type of V-model uses four test levels, corresponding to the four development levels. There may be more, fewer or different levels of development and testing depending on the project and the software product. E.g.: There may be component integration testing after Component testing or System Integration testing after System testing.



Iterative-incremental Development Models Iterative-incremental development is the process of establishing requirements, designing, building and testing a system, done as a series of shorter development cycles.



10

e.g.: Agile development process. The resulting system produced by iteration may be tested at several test level as part of its development. Regression testing is important on all iterations after the first one.

3.2

Test Levels

This section looks in detail at the various test levels. The key characteristics for each test level are discussed and defined to be able to more clearly separate the various test levels.  Component Testing Component testing searches for defects in and verifies the functioning of, software modules, programs, objects, classes, etc., that are separately testable. Stubs, drivers and simulators may be used. Test cases are derived from work products such as a specification of the component, the software design or the data model. Component testing usually involves the programmer who wrote the code. Integration Testing Integration testing interfaces between components, interactions with different parts of system. Component integration testing is done after component testing and tests the interactions between software components. System integration testing may be done after system testing and tests the interactions between different systems. Consider we have two components A and B(as shown in diagram below). As per the control flow A comes before than B. Top-down testing - While integrating these components, A is tested first with a simulator used in place of B. The simulator here is called as Stub. Bottom-up testing - While integrating these components is tested first with a simulator used in place of A. The simulator here is called as Driver.



11

-

Functional incremental – Integration and testing is done on basis of functionality as mentioned in functional specification.

System Testing System Testing is concerned with the behaviour of a whole system.



Acceptance Testing Acceptance testing is often the responsibility of the customers or users of a system. Stakeholders may be involved as well. Finding defects is not the main focus of acceptance testing. Goal is to establish confidence in the system. Before a product is put on sale commercially, it has to get feedback from potential or existing customers. Often this type of system undergoes two types of acceptance test. Alpha testing: It is a pre-release testing carried out at the developing organization's site but not by the developers. The testers could be a cross-section of potential users or an independent test team. Beta testing: It is carried out by potential users or customers at their own location. The system is sent to a cross-section of users who install it and use it under real-world working conditions. It is also called as field testing.



3.3

Test types

A test type is focused on a particular test objective. The test types are described below in short. 

Testing of Function: The function that a system or component performs may be documented in the functional specification or any other document. The functions are “what” the system does. Specification-based techniques (black box testing), may be used for functional testing.



Testing of non-functional software characteristics or non-functional testing: Non-functional testing would include performance testing, load testing, usability testing, maintainability testing, reliability testing, portability testing and stress testing. It deals with “how” system works. Non-functional testing may be performed at all test levels.

Testing of Software Structure/Architecture or White-box testing: Structural (white-box) testing may be performed at all test levels. White-box testing helps us to measure the thoroughness of testing through assessment of coverage of a type of structure. This type of testing is done by the developers who have access to code.  Testing related to changes: Testing related to changes includes re-testing and regression testing. The confirmation testing performed to check that the original defect has been successfully removed, once a defect has been fixed is called re-testing. Whenever we make some changes in an already existing code to add some new functionality or change the functionality, we need to test the already tested program to check if any defect has been introduced due to our code. This is called regression testing. 

12

-

3.4

Regression test suites are run many times so is a strong candidate for automation.

Maintenance testing   

Maintenance testing is done on an existing operational system, and is triggered by modifications, migration, or retirement of the software or system. Maintenance testing can be difficult if specifications are out of date or missing, or testers with domain knowledge are not available. Determining how the existing system may be affected by the changes is called Impact Analysis. Maintenance testing uses Impact Analysis to determine the regression test suite.

13

4.

Static Techniques

Static test techniques provide a powerful way to improve the quality and productivity of software development. This section describes static test technique – review process, types of review.

4.1

Static testing  



4.2

Static testing does not involve any execution of code. Static testing relies on manual examination (reviews) or automated analysis (static analysis) of the code or document. Any software work product can be reviewed, including requirements specifications, design specifications, code, test plans, test specifications, test scripts, user guides or web pages. Benefits of static testing: It finds defects early in life cycle. It reduces defects in deliverable. It is a learning experience in standards and technique. Typical defects that are easier to find in reviews than in dynamic testing are: deviations from standards, requirement defects, design defects, insufficient maintainability and incorrect interface specifications.

Review Process    

Review process includes formal and informal technique. Informal review process means going to someone’s desk and reviewing the code or document and giving comments if any. In contrast to informal review process, a formal review process consists of six main steps: Planning > Kick-off > Preparation > Review meeting > Rework > Follow-up The characteristic of each step are as follows: Planning: Defining review Criteria. Allocating roles for the review. Roles can be like reviewer, recorder, moderator, etc. Kick-off: Distributing documents and explaining the objectives, process and documents to the participants. Preparation: Preparing for the review meeting by reviewing the document. Review meeting: Examination, evaluation and recording of results. Making decisions about the defects. Rework: Fixing defects found. Follow-up: Checking that defects have been addressed.

14

4.3

Types of Reviews

A single document may be the subject of more than one review. For example, an informal review may be carried out before a technical review, or an inspection may be carried out on a requirements specification before a walkthrough with customers. The different types of reviews serve different purposes at different stages in the life cycle of a document. The main review types and their characteristics are described below.  Walkthrough: Meeting is led by Author. Sessions can be formal or informal. It is not time boxed. Primary purpose is learning, understanding and finding defects. 

Technical Review: Meeting is led by trained moderator and not author. It may be performed as a peer review without management participation. Pre-meeting preparation by reviewers is needed. Solutions sought after review is completed. Primary purpose: Finding defects, solving technical problems.



Inspection -

4.4

Meeting is led by independent trained moderator and not author. It is most formal review process. Pre-meeting preparation is essential. Entry and exit criteria are defined. It is expensive and time consuming. Primary purpose: Finding defects.

Static Analysis

Static Analysis is analysis of software code without executing them. 



Types of defects found by Static Analysis: Unreachable code Undeclared variable Parameter type mismatch Uncalled function and procedure Possible array bound violation Security violations Inconsistent interface between modules and components Incorrect variable usage Syntax checking Violation of code standard Use of variable without first defining them Variables that are declared but never used. Static Analysis tools are typically used by developers. 15

5.

Test Design Techniques

This section looks at dynamic testing, where the software we are interested in is run by executing tests on the running code.

5.1

Test Development Process

In this section we are looking at three things: test conditions, test cases and test procedures (or scripts). Each is specified in its own document, according to the Test Documentation Standard [IEEE829].

Let us understand the above terms with the help of an example on ATM machine and ATM card. Test Basis It is a document or anything (Source document), looking at which we can derive test information. E.g.: The Manual which is provided to the tester which provides the functionality of the ATM Machine or card can be considered as a test basis. A tester will derive test information based on these documents.



Test Condition We look at the test basis in order to see “what” could be tested. A test condition is something that we could test. For example: Checking functionality of an ATM card. This could be one of the test conditions for which multiple test cases would be derived. Test conditions are documented in the IEEE 829 document called a “Test Design Specification”. Contents of Test Design Specification Template: Test design specification identifier Features to be tested approach Refinements test identification feature Pass/Fail criteria





Test Cases A test case is a set of conditions or variables under which a tester will determine whether an application, software system or one of its features is working as it was originally established for it to do. One or more test cases could be derived from a test condition. For example: Withdrawing money from the machine with your ATM card, Entering PIN number, Checking balance, etc. are a set of test cases derived for above the test condition. Test Cases are documented in the IEEE 829 document called a “Test Case Specification”. Contents of Test Case Specification Template: Test case specification identifier Output specifications Test items Environmental needs 16

-

5.2

Input specifications Special procedural requirements Inter-case dependencies



Test Procedure The test procedure specifies the sequence of actions for the execution of a test. e.g.: The steps to be followed for withdrawing money. 1. Enter your ATM card in the slot. 2. Enter your PIN number 3. Select Cash Withdrawal 4. Select Yes. 5. Collect cash. Contents of Test Procedure Specification Template: Test procedure specification identifier Purpose Special Requirements Procedure Steps



Test Execution Schedule It specifies the time when the Test case needs to be tested.

Testing Techniques

There are two main categories of testing technique: Dynamic and Static Technique. As shown in below diagram, Dynamic Techniques are sub-divided into 3 more categories: Specification-based(Black box) Structure-based(White box) Experience-based Figure below shows the test techniques explained in the syllabus for ISTQB – Foundation level.

17

5.3

Specification-based (Black box) technique

Specification-based techniques are also known as 'black-box' testing techniques because they view the software as a black-box with inputs and outputs, but they have no knowledge of how the system or component is structured inside the box. In essence, the tester is concentrating on what the software does, not how it does it. In this section we will have a look at four specification-based or black-box techniques. Equivalence Partitioning(EP) The idea behind the technique is to divide (i.e. to partition) a set of test conditions into groups or sets that can be considered the same (i.e. the system should handle them equivalently), hence 'equivalence partitioning'. Consider the following example, input and output conditions from a Keyboard. Input Output 10 – 20 A 21 – 30 B 31 – 40 C We can derive test cases to test all the values, since the values are not large. But the same result can also be achieved if we derive one Test case from each class. 10 – 20 is one class. Similarly, 21- 30 and 31 – 40. Hence we can derive a Test case with following values to achieve 100% coverage: Values: 5, 14, 25, 34, 48 (One value from each class). 5 and 48 also need to be considered since both of these will be invalid data.



Boundary Value Analysis (BVA) Boundary value analysis (BVA) is based on testing at the boundaries between partitions. To apply boundary value analysis, we take the minimum and maximum (boundary) values from each of the valid partitions and the first or last value from each of the invalid partition. Consider the same example as described above: For boundary value analysis, we have 9, 10, 20, 21, 30, 31, 40 and 41 as boundary values:



Decision Table Testing Boundary value analysis (BVA) and Equivalence partitioning (EP) cannot be used if we have different combination of results. A decision table is a good way to deal with combination of things. This technique is also called as “cause-effect” table. The decision table contains the triggering conditions, often combinations of true and false for all input conditions, and the resulting actions for each combination of conditions. The coverage standard commonly used with decision table testing is to have at least one test per column in the table. Consider the following decision table example:



18

From the table, we can say the following: If Gender = Male and Employed = True, then his salary is 5000. If Gender = Male and Employed = False, then his salary is 1000. If Gender = Female and Employed = True, then his salary is 4000. If Gender = Female and Employed = False, then his salary is 500. Decision tables are a good way to capture system requirements that contain logical conditions. Advantage: Easy to maintain and update decision table if there is change in requirement. State Transition Testing A system may exhibit a different response depending on current conditions or previous history. A state table shows the relationship between the states and inputs, and can highlight possible transitions that are invalid. Consider the following state transition diagram for ATM card:



From the above diagram, we see the change in transitions depending upon the previous states. If we see the state “1st try“, from this point we have two states, depending on what input was given during 1st try. If 1st try = PIN OK, output = Access to account If 1st try = PIN not OK, output = 2nd try. Use Case Testing Use case testing technique helps us identifying test cases that exercise the whole system on a transaction by transaction basis from start to finish. Use case is description of a particular use of a system by the user of the system (actor). It describes what actor sees and does, rather what input system expects and what system outputs. It often uses language and terms of business rather than technical terms.



5.4

Structure-based (White box) technique

Structure-based techniques serve two purposes: test coverage measurement and structural test case design. Types of White box technique: 

Statement testing and coverage Statement coverage is defined as the percentage of number of statements exercised to the total number of statements.

19



5.5

Decision testing and coverage Decision coverage is defined as the percentage of number of decision outcomes exercised to the total number of decision outcomes.

Statement coverage and Decision Coverage Problems

In order to see how coverage actually works, we will use some code-level examples. These examples are asked in exam as well. So let us see the method to solve them 100% correctly. Example 1: 1. Read A 2. Read B 3. IF A=B 4. Print C 5. End if Note: I have numbered the lines for explanation purpose only. In exams it would not be numbered. Statement Coverage: For Statement coverage we need to cover each and every statement from 1 to 5. Let us consider Test case 1, A=10 and B=10. If we see for this test case, each and every statement from 1 to 5 is covered, that means 100 % Statement coverage. Hence we can say that, minimum of one Test case is needed in this case for 100% Statement Coverage.

In the above example we are considering only True condition and not the false one, because in the example we do not see “ELSE” written anywhere. Only “IF” is given. We will be seeing another example later to have a clearer view. Decision Coverage: For Decision coverage we need to cover each and every decision that comes in between lines 1 to 5. Here, “IF” condition on line number 3 shows two conditions – True and False. So we need to cover both the conditions for 100% decision coverage. Let us consider Test case 1, A=10 and B=10. If we see for this test case, from line number 3 only the TRUE condition is satisfied. Total number of conditions = 2 20

Number of conditions covered = 1 As per definition of Decision coverage, we have only 50% of coverage by using only 1 Test case. Let us consider Test case 2, A=10 and B=11. If we see for this test case, from line number 3 only, the FALSE condition is satisfied. If we add both the, both the decisions are covered. Total number of conditions = 2 Number of conditions covered = 2

As per definition of Decision coverage, we have 100% of coverage by using 2. From the above example we can say that, 100% decision coverage means 100% statement coverage. But viceversa may not be true. Example 2: 1. Read A 2. Read B 3. IF A=B 4. Print C 5. ELSE 6. Print D 7. End if Statement Coverage: For Statement coverage we need to cover each and every statement from 1 to 7. Let us consider Test case 1, A=10 and B=10. If we see for this test case, statements from 5 and 6 are not covered, that means 100 % Statement coverage is not achieved. Test case 2, A=10 and B=11 If we see for this test case, statements 5 and 6 are covered but statement 4 is not covered. But if we add both the above, all statements are covered at least once.

21

Hence we can say that, in this case at least 2 are needed for 100% Statement Coverage. Decision Coverage: For Decision coverage we need to cover all the conditions are present between lines 1 to 7. Let us consider Test case 1, A=10 and B=10. If we see for this test case, only TRUE condition will be executed. Test case 2, A=10 and B=11 If we see for this test case, only FALSE condition will be executed. But if we add both the above, all decisions coming between lines 1 to 7 are covered at least once.

Hence we can say that, in this case at least 2 are needed for 100% Decision Coverage.

5.6

Experience-based technique

Some defects are hard to find using more systematic approaches. For such scenarios we have experience based testing. Experienced-based testing is where tests are derived from the tester’s skill and intuition and their experience with similar applications and technologies. Types of experience-based techniques: Error Guessing and exploratory testing.  Error Guessing: The tester is encouraged to think of situations in which the software may not be able to cope.

22

A structured approach to the error-guessing technique is to list possible defects or failures and to design tests that attempt to produce them. These defect and failure lists can be built based on the tester's own experience. Exploratory testing: As its name implies, exploratory testing is about exploring, finding out about the software, what it does, what it doesn't do, what works and what doesn't work. Exploratory testing is a hands-on approach in which testers are involved in minimum planning and maximum test execution. This is an approach that is most useful when there are no or poor specifications and when time is severely limited. -



23

6.

Test Management

Testing usually accounts for a substantial proportion of the overall project budget. Therefore, we must understand how we should manage the testing we do. In this section, we will cover essential topics for test management.

6.1

Independent testing

A certain degree of independence often makes the tester more effective. Several levels of independence can be defined as shown here from low to high Tests designed by person who wrote the software under test - Lowest level.  Test designed by another person within team – Low level.  Test designed by a person from different organizational group or company – Highest level.

6.2

Tasks of Test leader and Tester Test Leader Sometimes also called test manager or test coordinator. Following are the tasks of a test leader, Planning, monitoring and control of testing activities and tasks. Devising test objectives, test plans, test strategies and organizational test policies in collaboration with other stakeholders. Acquiring necessary resources. Selecting appropriate tools for testing and training of team if required. Monitoring the analysis, design, implementation, and execution of, test procedures and test suites. Ensuring proper configuration management of test ware produced.



Tester Following are the tasks of a tester, Review and contribute to test plans. Identifying test conditions. Creating test designs test procedure specifications and test data. Setting up test environment. Executing and logging the tests. Evaluating results and document the problems found.



6.3

Entry Criteria and Exit Criteria 

Entry Criteria Entry criteria define when to start testing such as at the beginning of a test level or when a set of tests is ready for execution. Typically entry criteria may cover the following: Test environment availability and readiness Test tool readiness in the test environment Testable code availability Test data availability

24



6.4

Exit Criteria Exit criteria define when to stop testing such as at the end of a test level or when a set of tests has achieved specific goal. Thoroughness measures, such as coverage of code, functionality or risk: Estimates of defect density or reliability measures Cost Residual risks, such as defects not fixed or lack of test coverage in certain areas.

Test Estimation

Two types of estimation are followed:  

6.5

Metrics based approach - Estimating the test effort based on metrics of former or similar projects. Expert based approach - Estimating the tasks based on estimates made by the owner of tasks tests is ready for execution.

Test Approach 



6.6

The test approach is the implementation of the test strategy for a specific project. The test approach is defined and refined in the test plans and test designs. It typically includes the decisions made based on the (test) project’s goal and risk assessment. Typical approaches include: Analytical approaches Model based approaches Methodical approaches Process or standard compliant approaches Dynamic approaches Consultative approaches

Test Monitoring and control 

Test Monitoring The purpose of test monitoring is to provide feedback and visibility about test activities. Monitoring percentage of work done and test coverage of requirements, risks or code.



Test Control Test control describes any guiding or corrective actions taken as a result of information and metrics gathered and reported.

Note: Some other topics like test planning, test execution, etc. needed in this Section are already covered in Section – Fundamentals of testing.

6.7

Risks in testing

This section covers a topic that we believe is critical to testing: risk. We'll see that there are risks related to the product and risks related to the project, and look at typical risks in both categories.

25



Project Risk Project risks are the risks that surround the project’s capability to deliver its objectives, like Skill, training and staff shortages Personnel issues Problems in defining the right requirements The extent to which requirements cannot be met given existing constraints Test environment not ready on time Failure of a third party Contractual issues



Product Risk Potential failure areas in the software or system are known as product risks as they are a risk to the quality of the product, like Failure-prone software delivered The potential that the software/hardware could cause harm to an individual or company Poor software characteristics (e.g., functionality, reliability, usability and performance) Poor data integrity and quality (e.g., data migration issues, data conversion problems, data transport problems, violation of data standards) Software that does not perform its intended functions.

26

7.

Support for Testing

In this section we will see that the tools are grouped by the testing activities or areas that are supported by a set of tools. For example, tools that support management activities, tools to support static testing, etc.

7.1

Tools Classification:

Tools that support the management of testing and tests:  Test management tool  Requirements management tool  Incident management tool  Configuration management tool Tools that support static testing:  Review process support tool  Static analysis tool (D)  Modelling tool (D) Tools that support test specification:  Test design tool  Test data preparation tool Tools that support test execution and logging:  Test execution tool  Test harness and Unit test framework tool (D)  Test comparator  Coverage measurement tool (D)  Security tool Tools that support performance and monitoring:  Dynamic analysis tool  Performance-testing, load-testing and stress-testing tool  Monitoring tool Note: Tools mentioned as “D” are design tools.

7.2

Tool support for management of testing and tests

The tools in this broad category provide support for the "management of tests" as well as "managing the testing process". The management of testing applies over the whole of the software development life cycle, so a test management tool could be among the first to be used in a project. A test management tool may also manage the tests, which would begin early in the project and would then continue to be used throughout the project and also after the system had been released. Tools used in this category are mentioned below. 

Test management tools The features provided by test management tools include those listed below Management of tests Execution and scheduling of tests 27

-

Management of testing activities Logging test results Preparing progress reports based on metrics



Requirements management tools The features provided by requirements management tools include those listed below Storing requirement statements Checking consistency of requirements Identifying undefined, missing or 'to be defined later' requirements Prioritizing requirements for testing purposes



Incident management tools Also known as defect tracking tools, defect management tools, bug tracking tools or bug management tools, these tools provide the following features Storing information about the attributes of incidents and also storing attachments Assigning actions Prioritizing incidents Status



Configuration management tools The features provided by configuration management tools include those listed below Storing information about versions and builds of the software and test ware. Keeping track of which versions belong with which configurations Build and release management Base lining (e.g. all the configuration items that make up a specific release)

7.3

Tool support for static testing

The tools described in this section support the testing activities described in Section – Static Techniques. 

Review process support tools The features provided by review process tools include those listed below Monitoring the review status (passed, passed with corrections, requires re-review) Providing traceability between comments, documents reviewed and related documents Keeping track of comments, including defects found, and providing statistical information about them Communicating comments to relevant people



Static Analysis tools These tools are normally used by developers as part of development. Static analysis tools for code can help the developers to understand the structure of the code, and can also be used to enforce coding standards. The most important feature provided by static analysis tools is identifying defects in the code.

28



7.4

Modelling tools Modeling tools help to validate models of the system or software. These tools are normally used by developers. The features provided by Modelling tools include those listed below Identifying inconsistencies and defects within the model Helping to identify and prioritize areas of the model for testing

Tool support for test specification 

Test Design tools These tools are used to generate test inputs or executable tests from requirements, graphical user interfaces, design models (state, data or object) or code.



Test Data Preparation tools These tools are used for enabling data to be selected from an existing database or creating, generating, manipulating and editing for use in tests. They are also used to construct a large number of similar records from a template, to give a large set of records for volume tests.

7.5

Tool support for Test execution and logging 

Test Execution tools The features provided by test execution tools include those listed below Executing tests from stored scripts and optionally data files accessed by the script. Capturing (recording) test inputs while tests are executed manually. Storing an expected result in the form of a screen or object to compare to, the next time the test is run.



Test harness/unit test framework tools These tools are normally used by developers. A test harness provides stubs and drivers, which are small programs that interact with the software under test. Features: Supplying inputs to the software being tested(Drivers) Receiving outputs generated by the software being tested(Stubs) Executing a set of tests within the framework or using the test harness



Test comparators The essence of testing is to check whether the software produces the correct result, and to do that, we must compare what the software produces to what it should produce. A test comparator helps to automate aspects of that comparison.



Coverage measurement tool These tools are normally used by developers. The essence of testing is to check whether the software produces the correct result, and to do that, we must compare what the software produces to what it should produce. A test comparator helps to automate aspects of that comparison. The features provided by coverage measurement tool include those listed below Calculating the percentage of coverage items that were exercised by a suite of tests. 29



7.6

Reporting coverage items that have not been exercised as yet.

Security tool Security testing tools can be used to test security by trying to break into a system, whether or not it is protected by a security tool. The features provided by security tool include those listed below Identifying viruses. Detecting intrusions such as denial of service attacks. Identifying weaknesses in password files and passwords. Security checks during operation. E.g. for checking integrity of files.

Tool support for performance and monitoring

The tools described in this section support testing that can be carried out on a system when it is operational, i.e. while it is running. 

Dynamic Analysis tools Dynamic analysis tools are 'dynamic' because they require the code to be executed. The features provided by Dynamic Analysis tools include those listed below Detecting memory leaks Identifying pointer arithmetic errors such as null pointers



Performance-testing, load-testing and stress-testing tool Features: Generating a load on the system to be tested Measuring the timing of specific transactions as the load on the system varies Measuring average response times; Producing graphs or charts of responses over time.



Monitoring tools Monitoring tools are used to continuously keep track of the status of the system in use, in order to have the earliest warning of problems and to improve service.

7.7

Potential benefits of using tools   

Reduction of repetitive work Greater consistency and repeatability Objective assessment

30

8.

Reference  

Foundations of Software Testing – ISTQB Certification by Dorothy Graham, Erik Veenendaal, Isabel Evans, Rex Black The book is also available for download for free at the below url: http://api.ning.com/files/IPWzwvTv3Hu3RchV7iA9H64BC1i06bZfNKpuGuPSBonklDmaVgsD6VExvHEQWryd2nDINaCTGS6f1M6X6acz5V6JFPdCocd/istqb_foundations_of_software_test ing.pdf

31

Thank You

Contact For more information, contact DEG Core KM Grp

About Tata Consultancy Services (TCS) Tata Consultancy Services is an IT services, consulting and business solutions organization that delivers real results to global business, ensuring a level of certainty no other firm can match. TCS offers a consulting-led, integrated portfolio of IT and IT-enabled infrastructure, engineering and assurance services. This is delivered through its unique Global Network Delivery ModelTM, recognized as the benchmark of excellence in software development. A part of the Tata Group, India’s largest industrial conglomerate, TCS has a global footprint and is listed on the National Stock Exchange and Bombay Stock Exchange in India. For more information, visit us at www.tcs.com.

IT Services Business Solutions Consulting All content / information present here is the exclusive property of Tata Consultancy Services Limited (TCS). The content / information contained here is correct at the time of publishing. No material from here may be copied, modified, reproduced, republished, uploaded, transmitted, posted or distributed in any form without prior written permission from TCS. Unauthorized use of the content / information appearing here may violate copyright, trademark and other applicable laws, and could result in criminal or civil penalties. Copyright © 2011 Tata Consultancy Services Limited