Software testing basics

In short

Software testing is the planned execution and static examination of software to find defects, measure quality against requirements, and give stakeholders evidence about risk before release. It is not proof of correctness: testing can show the presence of defects but never their absence, so the whole discipline is about choosing a small, high-yield subset of the infinite possible inputs. It is examined in every Indian university software-engineering paper (AICTE/CBCS syllabi), in UGC-NET Computer Science, and in ISRO, BARC, DRDO and PSU written tests; note that software engineering was dropped from the GATE CSE syllabus in the 2021 revision, so confirm the current syllabus before preparing it for GATE. In placements it is the default area of questioning for QA/SDET roles at TCS, Infosys, Wipro, Cognizant, Accenture, Capgemini and LTIMindtree, and is asked of every developer as "how would you test this?"

Explained

Basic vocabulary and the testing principles

• Error (mistake) - a human action that produces an incorrect result; defect (fault, bug) - the resulting flaw in code or document; failure - the observable deviation when the defect is executed. A defect in dead code never causes a failure.
• Verification - "are we building the product right?" - conformance to specification, done mostly by reviews, static analysis and unit/integration tests. Validation - "are we building the right product?" - fitness for the user's actual need, done by system testing and UAT.
• Test case - a set of preconditions, inputs, execution steps and expected result; test suite - a grouped set; test script - its executable form; test data - the input values used.
• Test oracle - the source of the expected result (requirement document, a previous version, a reference implementation, a formula, or a knowledgeable user). The "oracle problem" is that without one you can detect crashes only.
• Test bed / test environment - hardware, OS, database, network and build under which tests run; test harness - stubs, drivers and tooling that let a part be executed in isolation.
• Seven principles (ISTQB): (1) testing shows the presence, not the absence, of defects; (2) exhaustive testing is impossible; (3) early testing saves cost (shift-left); (4) defect clustering - a few modules hold most defects (Pareto); (5) pesticide paradox - repeating the same tests stops finding new defects, so tests must be revised; (6) testing is context dependent (a UPI app is not tested like a game); (7) absence-of-errors fallacy - a defect-free product that does not meet user needs is still a failure.
• Cost of defects rises with time of detection - a requirements defect caught in review is far cheaper than the same defect found in production; this is the economic argument for static testing.

Static testing and reviews

• Static testing examines artefacts without executing code: requirements, design, code, test cases themselves.
• Review types, increasing formality: *informal review* (desk check, peer buddy); *walkthrough* (author leads, educates, no formal exit criteria); *technical review* (peers, documented, focused on technical soundness); inspection (Fagan - defined roles of moderator, author, reader, inspector and recorder/scribe; entry/exit criteria, checklists, defect logging, metrics, follow-up).
• Static analysis tools - compilers' warnings, SonarQube, PMD, FindBugs/SpotBugs, Coverity, cppcheck - find unreachable code, uninitialised variables, resource leaks, cyclomatic complexity hot-spots, coding-standard violations (MISRA C in automotive).
• Static vs dynamic: static finds the defect itself at its source and can be applied before code exists; dynamic finds failures and measures behaviour, performance and resource use.

Levels of testing and integration strategies

1. Unit (component) testing - smallest testable unit (function, class, module), usually by the developer, with stubs (called-by dummy) and drivers (calling dummy). Checks internal logic, boundary values, error handling.
2. Integration testing - interfaces and interactions between units.
   • *Big bang* - everything assembled at once; cheap to set up, hard to localise faults.
   • *Top-down* - from the main control module downwards; needs stubs; finds design/control defects early, demonstrates a skeleton product early; lower-level utilities tested late.
   • *Bottom-up* - from utilities upwards; needs drivers; good for critical low-level logic; no working system until late.
   • *Sandwich (hybrid)* - both ends meet at a target layer; common in large systems.
   • *Incremental / continuous integration* - one module added at a time against a working build.
3. System testing - the complete integrated system against the SRS, on a production-like environment: end-to-end functional flows plus non-functional characteristics.
4. Acceptance testing - against user needs and contract. Forms: UAT by business users, contract and regulatory acceptance (RBI, SEBI, IRDAI, NPCI certification for UPI apps), operational acceptance (backup, restore, failover, patching), alpha (in-house, developer site) and beta (real users, their own environment).
5. These levels map to the right-hand arm of the V-model: unit tests to module design, integration to high-level design, system to the SRS, acceptance to user requirements - each test level planned when the corresponding document is written.

Black-box (specification-based) test design

• Equivalence partitioning (EP) - divide each input domain into classes expected to be treated alike; test one value per class, valid and invalid. Reduces test count without losing much coverage.
• Boundary value analysis (BVA) - defects cluster at edges, so test min, min+1, nominal, max−1, max. *Robust* BVA adds min−1 and max+1. Counts: 4n+1 (BVA), 6n+1 (robust), 5ⁿ (worst case, all combinations), 7ⁿ (robust worst case) for n input variables.
• Decision table (cause-effect) testing - list conditions and actions; 2ᵏ rules for k binary conditions; collapse impossible/don't-care rules; one test per surviving rule. Best for complex business rules (loan eligibility, tax slabs, discount policy).
• State transition testing - model states, events, transitions, guards; derive tests to cover all states, all valid transitions (0-switch), transition pairs (1-switch) and invalid events. Natural for ATM, OTP login, order status, payment gateway flows.
• Use case / scenario testing - test the main success flow and each alternative/exception flow end to end; finds integration defects users will actually hit.
• Pairwise / orthogonal array testing (OATS) - most defects arise from one or two parameters interacting, so cover every *pair* of values instead of every combination; huge reduction in configuration testing (browser × OS × language × payment mode).
• Error guessing and exploratory testing - experience-driven; checklists of classic failures (zero, null, blank, negative, very long string, special characters, duplicate submit, back button, session timeout, leap year, 29 February, Unicode/Indian-language text). Ad hoc / monkey testing is unstructured and unrepeatable - a supplement, never a strategy.
• Random and fuzz testing - machine-generated inputs, strong for crashes and security defects.

White-box (structure-based) testing and coverage

• Control flow graph (CFG) - nodes are statements/blocks, edges are control transfers; the basis for structural measures.
• Coverage criteria, broadly weakest to strongest: statement (line) → decision/branch → condition (on its own it does *not* subsume branch coverage) → decision/condition → MC/DC (each condition independently shown to affect the outcome; required by DO-178C Level A avionics, highly recommended by ISO 26262 ASIL D automotive software) → multiple condition → path coverage.
• 100% branch coverage implies 100% statement coverage, not the reverse; 100% path coverage is infeasible when loops exist.
• Basis path testing (McCabe) - compute cyclomatic complexity V(G), then design that many linearly independent paths as the minimum structural test set.
• Loop testing - skip the loop, one pass, two passes, typical, n−1, n and n+1 passes; nested loops tested innermost first.
• Data flow testing - define-use (du) pairs; criteria all-defs, all-uses, all-du-paths; catches uninitialised and dead variables.
• Mutation testing - seed small artificial faults (mutants) and measure how many the test suite kills; measures suite *quality*, not product quality. Equivalent mutants (semantically identical) are excluded.
• Limitation: structural testing can never find a missing requirement - code that was never written has no statements to cover. Hence black-box and white-box are complements, not alternatives.

Types of testing (what quality attribute is targeted)

• Functional - does it do what the specification says.
• Regression - re-running existing tests after a change to ensure nothing previously working has broken; selected by risk and impact analysis; the main candidate for automation. Retesting (confirmation testing) is different: it re-runs only the failed test on the fixed defect.
• Smoke testing - shallow and wide build-verification; if it fails, the build is rejected. Sanity testing - narrow and deep check of the specific area just fixed.
• Performance testing family - *load* (expected concurrency), *stress* (beyond capacity, to find the breaking point and recovery), *spike* (sudden surge - IRCTC Tatkal, exam results day), *endurance/soak* (memory leaks over hours), *volume* (large data), *scalability*.
• Security testing - authentication, authorisation, input validation, injection, session management, encryption; VAPT by CERT-In empanelled auditors is expected for Indian government and BFSI systems (the empanelment list is revised periodically).
• Usability and accessibility - task success, error recovery; Indian government websites are expected to follow GIGW (Guidelines for Indian Government Websites, GIGW 3.0, aligned to WCAG 2.1).
• Compatibility / cross-browser / cross-device, localisation and internationalisation (Devanagari, Tamil and other Indian scripts, ₹ symbol, dd-mm-yyyy, IST), installation, recovery, migration, compliance.
• Quality model: ISO/IEC 25010 defines the product quality characteristics tested - functional suitability, performance efficiency, compatibility, usability/interaction capability, reliability, security, maintainability, portability (the 2023 revision reorganised some characteristics, replacing portability with flexibility and adding safety; check the edition cited by your syllabus).

STLC, documentation, defects and metrics

• STLC phases: requirement analysis → test planning (strategy, estimation, risk) → test case design and RTM (requirements traceability matrix) → test environment setup → test execution and defect reporting → test cycle closure (metrics, lessons, archiving).
• Entry criteria - what must exist before a level starts (frozen SRS, stable build, environment ready); exit criteria - when to stop (coverage target met, all high-severity defects closed, defect-arrival rate falling, time/budget exhausted, risk accepted).
• Test plan contents (IEEE 829-2008) - scope, items to test, features not to be tested, approach, item pass/fail criteria, suspension and resumption criteria, deliverables, environment, responsibilities, staffing, schedule, risks. Other IEEE 829 documents: test design specification, test case specification, test procedure, test log, test incident/anomaly report, test summary report. IEEE 829 has been superseded by ISO/IEC/IEEE 29119 (Part 1 concepts, Part 2 processes, Part 3 documentation, Part 4 techniques, Part 5 keyword-driven testing) - many Indian university question papers still ask IEEE 829 by name.
• Defect report fields: id, title, module, build/version, environment, steps to reproduce, expected vs actual, severity, priority, attachments (log, screenshot), reporter, assignee, status.
• Defect life cycle: New → Assigned → Open → Fixed → Retest → Closed; side states Reopened, Deferred, Rejected, Duplicate, Not a bug/Enhancement.
• Severity vs priority: severity is technical impact (set by the tester), priority is business urgency of the fix (set by the product owner/manager) - they are independent.
• Metrics used in Indian delivery organisations (CMMI-appraised accounts especially): defect density, defect removal efficiency, defect leakage/escape, test coverage, test execution and pass rate, defect ageing, defect severity distribution, cost of quality, mean time between failures. STQC (Standardisation Testing and Quality Certification Directorate, MeitY) operates India's government software test labs and certification schemes; ISTQB (through the Indian Testing Board) is the common tester certification.
• Test data and the law: the Digital Personal Data Protection Act, 2023 makes using real customer personal data in test environments a compliance risk; masked, anonymised or synthetic test data is now standard practice in BFSI and healthcare projects.

Key points

• Testing - executing or examining software to find defects and measure quality; it cannot prove correctness.
• Error → defect → failure - human mistake, flaw in the artefact, observed misbehaviour.
• Verification vs validation - building it right (specification) vs building the right thing (user need).
• Exhaustive testing is impossible - design techniques exist to pick a high-yield subset.
• Stub vs driver - a stub is called by the unit under test; a driver calls it.
• Top-down needs stubs, bottom-up needs drivers; big bang makes fault localisation hard.
• EP - one value per class of equivalent inputs; BVA - values at and around the edges (4n+1 / 6n+1).
• Decision table - 2ᵏ rules for k binary conditions, collapsed using don't-care entries.
• State transition testing - covers states, valid transitions, transition pairs and invalid events.
• Pairwise/OATS - covers all parameter pairs instead of all combinations; large reduction in configuration tests.
• Branch coverage subsumes statement coverage; MC/DC is required by DO-178C Level A and highly recommended by ISO 26262 ASIL D.
• Cyclomatic complexity V(G) = E − N + 2P = predicate nodes + 1 = number of regions = basis-path test count.
• Mutation score measures the test suite, not the software.
• Smoke is wide and shallow, sanity is narrow and deep; retesting checks a fix, regression checks everything else.
• Severity is technical, priority is business - a cosmetic typo on a bank's home page is low severity, high priority.
• Defect clustering and the pesticide paradox - focus effort where defects are, but keep refreshing the tests.
• Standards - IEEE 829 (test documents, superseded by ISO/IEC/IEEE 29119), ISO/IEC 25010 (quality model), ISTQB syllabus, STQC in India.
• Exit criteria, not "testing is finished" - testing stops on coverage, defect trend, risk and cost, which are agreed in advance.

Formulas and examples

Example 1 - Cyclomatic complexity, basis paths and coverage.

int grade(int m){
  if (m < 0 || m > 100) return -1;   /* d1 */
  if (m >= 90) return 1;             /* d2 */
  else if (m >= 75) return 2;        /* d3 */
  else if (m >= 40) return 3;        /* d4 */
  return 4;
}

• CFG: N = 11 nodes (10 statement blocks + exit), E = 14 edges, P = 1.
• V(G) = 14 − 11 + 2 = 5. Check: predicate nodes = 4 (d1–d4) → 4 + 1 = 5. Both agree, so 5 basis paths / 5 minimum tests: m = −5 → −1; m = 95 → 1; m = 80 → 2; m = 50 → 3; m = 20 → 4.
• If the compound condition m<0 || m>100 is counted as two decisions (condition-level), V(G) = 6 and a sixth test m = 105 is needed - state the convention used in the exam answer.
• Coverage of a weak suite {95, 20}: statements executed = {read, d1, d2, return 1, d3, d4, return 4} = 7 of 10 → 70% statement coverage. Branch outcomes = 4 decisions × 2 = 8; exercised = d1-F, d2-T, d2-F, d3-F, d4-F = 5 of 8 → 62.5% branch coverage. Lesson: statement coverage flatters a suite.

Example 2 - EP and BVA for a job-portal field. "Age" accepts integers 18 to 60 inclusive.

• EP classes: invalid < 18; valid 18–60; invalid > 60; invalid non-numeric/blank/negative → 1 valid + 3 invalid class representatives.
• BVA (4n + 1, n = 1) = 5 tests: 18, 19, 39, 59, 60. Robust BVA (6n + 1) = 7 tests: adds 17 and 61.
• Now add a second field "experience 0–40 years": BVA = 4(2) + 1 = 9 tests; robust = 6(2) + 1 = 13; worst case = 5² = 25; robust worst case = 7² = 49. Exhaustive integer combinations would be 43 × 41 = 1,763 - the reduction is the whole point.
• Configuration testing by pairwise: 4 parameters × 3 values each = 3⁴ = 81 combinations; an L9 orthogonal array covers every pair in 9 tests - an 88.9% reduction.

Example 3 - Decision table for login. Conditions: C1 valid user ID, C2 valid password, C3 account locked → 2³ = 8 rules, collapsed to 4 test cases (the don't-care entries cover all 8).

Example 4 - Defect metrics for a release. Module size 25,000 LOC = 25 KLOC. Testing found 150 defects; users reported 18 in the first three months. 1,200 test cases planned, 1,080 executed, 950 passed.

• Defect density = 150 / 25 = 6.0 defects per KLOC.
• DRE = 150 / (150 + 18) × 100 = 150/168 = 89.3%.
• Defect leakage = 18 / 168 × 100 = 10.7%.
• Test execution = 1,080 / 1,200 = 90%; pass rate = 950 / 1,080 = 88.0%; 130 test cases failed.
• Reading: DRE below about 90% is weak for a BFSI release; 120 unexecuted test cases must be listed as residual risk in the test summary report.

Example 5 - Reliability and availability of a tested server application. Observed 2,000 hours of operation, 8 failures, average repair time 1.5 hours.

• MTTF = 2,000 / 8 = 250 hours; λ = 1/250 = 0.004 failures per hour; MTTR = 1.5 h.
• Availability = 250 / (250 + 1.5) = 250/251.5 = 0.99404 → 99.40%.
• Annual downtime = (1 − 0.99404) × 8,760 ≈ 52.2 hours - short of a "three nines" (99.9%, ≈8.8 h) SLA.
• Reliability over a 24-hour day: R(24) = e^{−0.004 × 24} = e^−0.096 = 0.908, i.e. about a 90.8% chance of no failure in a day.

Where it is used

• IT services and GCCs: dedicated QA/testing practices at TCS, Infosys, Wipro, Cognizant, Accenture, Capgemini, LTIMindtree and HCLTech - test analyst, test lead, performance tester, SDET roles, most on CMMI-appraised accounts with metric-based reporting.
• BFSI and payments: UAT sign-off and regulator/NPCI certification for UPI, IMPS and net-banking releases; load and spike testing of result-day and Tatkal-type peaks (IRCTC, exam boards, Aadhaar-linked services).
• Product companies and start-ups: Zoho, Freshworks, Razorpay, Zerodha, Swiggy and similar run exploratory testing, release smoke suites and performance budgets, with testers embedded in feature teams.
• Safety-critical and embedded engineering: Bosch, Tata Elxsi, KPIT, HAL, ISRO suppliers - MC/DC coverage, requirements traceability and static analysis to ISO 26262 (automotive) and DO-178C (avionics).
• Government and e-governance: STQC (MeitY) test and certification labs, GIGW accessibility conformance, CERT-In empanelled VAPT before go-live.
• Independent testing and compliance roles: test managers writing ISO/IEC/IEEE 29119 documentation, audit of test evidence, and masking of personal data in test environments under the DPDP Act, 2023.

Interview questions

• Q: What is the difference between verification and validation? A: Verification checks the product against its specification (reviews, unit tests) - building it right; validation checks it against the user's real need (system, UAT) - building the right product.
• Q: Differentiate error, defect and failure. A: An error is the human mistake, the defect is the resulting flaw in the code or document, and a failure is the observable wrong behaviour when that defect is executed.
• Q: Can testing prove software is correct? A: No. The input space is effectively infinite, so testing can only show the presence of defects; confidence comes from risk-based coverage, not from proof.
• Q: What are stubs and drivers, and which integration strategy needs which? A: A stub is a dummy for a module that the unit under test calls; a driver is a dummy that calls it. Top-down integration needs stubs, bottom-up needs drivers.
• Q: Severity or priority - a spelling mistake in the company name on the landing page? A: Low severity (nothing breaks) but high priority, because it is customer-facing and damages credibility, so it is fixed in the next build.
• Q: Smoke versus sanity testing? A: Smoke is a broad, shallow build-verification run to decide whether the build is testable at all; sanity is a narrow, deeper check of the specific area that was just changed or fixed.
• Q: Regression testing versus retesting? A: Retesting re-runs the failed test on the fixed defect to confirm the fix; regression re-runs other existing tests to confirm the fix broke nothing else.
• Q: Does 100% statement coverage mean the code is well tested? A: No. Branch, condition and path coverage can still be low, boundary values may be untested, and missing requirements have no code to cover.
• Q: How do you compute cyclomatic complexity and what do you do with it? A: V(G) = E − N + 2P, or decision nodes + 1; it gives the number of linearly independent basis paths, hence the minimum structural test count, and flags over-complex functions for refactoring.
• Q: How would you test an age field that accepts 18 to 60? A: Equivalence classes (below 18, 18–60, above 60, non-numeric) plus boundary values 17, 18, 19, 59, 60, 61, and negative tests for blanks, decimals, very long input and special characters.
• Q: What is the pesticide paradox and how do you counter it? A: Repeatedly running the same tests stops revealing new defects, so test cases must be reviewed, extended and varied, with exploratory and risk-based testing added.
• Q: When do you stop testing? A: When the exit criteria agreed in the test plan are met - coverage targets, no open high-severity defects, falling defect-arrival rate - and the remaining risk is formally accepted by the stakeholders.
• Q: Why is production data not simply copied into the test environment? A: It exposes personal data, which is a compliance risk under the DPDP Act, 2023; masked, anonymised or synthetic data preserving the same distributions is used instead.

Common mistakes

• Writing a test case without a definite expected result, which makes the pass/fail judgement subjective and the test unrepeatable.
• Treating severity and priority as the same field, or letting the tester decide fix priority.
• Claiming high quality on the strength of statement coverage alone, and quoting 100% path coverage as achievable in code with loops.
• Using the wrong node/edge count in V(G), or forgetting that a compound condition A || B adds two decisions under condition-level counting.
• Swapping stub and driver, or saying top-down integration needs drivers.
• Testing only valid inputs - no negative, boundary, blank, duplicate-submit or session-timeout cases - and ignoring non-functional attributes entirely.
• Calling exhaustive testing merely "expensive" rather than impossible, and reporting defect counts without the denominator (size, test cases executed of planned) so the metric cannot be interpreted.

Related topics

Test automation and TDD · SDLC and Agile · Requirements engineering and UML · Clean code and code reviews · Software project management and estimation · Low-level design (object-oriented design) interviews