Software quality assurance: metrics, reviews and standards

In short

Software quality assurance (SQA) is the planned, systematic set of activities that gives confidence a product and its process conform to requirements — as distinct from quality control, which inspects the product that was built. It rests on three legs: metrics (measure quality objectively), reviews (find defects statically, before execution), and standards and models (ISO/IEC, CMMI, Six Sigma) that define what "good" means. Indian IT services firms — TCS, Infosys, Wipro, HCLTech, Cognizant, LTIMindtree, Tech Mahindra — run large dedicated SQA/quality functions and are among the world's heaviest users of CMMI and ISO certification, so this topic appears in almost every campus placement technical round for QA, developer and business-analyst roles. In exams it is standard in AICTE/CBCS "Software Engineering" and "Software Quality Assurance and Testing" papers, and GATE CS asks the computable parts: cyclomatic complexity, function points, defect density and reliability.

Explained

Quality, QA and QC: definitions and the V&V frame

• Software quality (IEEE): the degree to which a system, component or process meets specified requirements *and* customer/user needs or expectations. Two faces: conformance to requirements (Crosby) and fitness for use (Juran).
• Quality assurance (QA): process-oriented, preventive, applies across the life cycle — defining standards, audits, training, process definition, metrics programmes. Answers "are we building it the right way?"
• Quality control (QC): product-oriented, detective — reviews, inspections, testing of a specific deliverable. Answers "is this artefact right?"
• Testing is a subset of QC; SQA ⊃ QC ⊃ testing.
• Verification: does the product conform to its specification (reviews, inspections, static analysis, unit testing)? Validation: does it meet the user's actual need (system, acceptance, alpha/beta testing)?
• Quality cost (CoQ) has four buckets: prevention (training, process, tools), appraisal (reviews, testing, audits), internal failure (rework before release), external failure (field defects, support, penalties, reputation). Cost of fixing a defect rises roughly an order of magnitude per phase it escapes (Boehm's 1:10:100 pattern) — the economic argument for reviews.
• SQA Plan (IEEE 730): the controlling document — purpose, reference documents, management and organisation, documentation to be produced, standards and conventions, reviews and audits, test, problem reporting and corrective action, tools, supplier control, records retention, training, risk management.

Quality models and attributes

• McCall's model (1977): 11 factors in 3 groups — *product operation* (correctness, reliability, efficiency, integrity, usability); *product revision* (maintainability, testability, flexibility); *product transition* (portability, reusability, interoperability). Factors → criteria → metrics (the "FCM" hierarchy).
• Boehm's model (1978): high-level uses (as-is utility, maintainability, portability) → intermediate constructs → primitive measurable characteristics; explicitly adds hardware/economy considerations.
• Dromey's model: links product properties (correctness, internal, contextual, descriptive) of each component to quality attributes — useful because it is implementation-traceable.
• ISO/IEC 25010 (SQuaRE family, product quality model) — the current examinable standard, superseding ISO/IEC 9126: eight characteristics — *functional suitability, performance efficiency, compatibility, usability (interaction capability), reliability, security, maintainability, flexibility/portability*. Note the 2023 revision renamed and regrouped some characteristics (e.g. interaction capability, flexibility, safety added), so quote the characteristic rather than a fixed list of eight if the edition is not stated. A separate quality-in-use model covers effectiveness, efficiency, satisfaction, freedom from risk, context coverage.
• Garvin's five views of quality: transcendental, user, manufacturing, product, value-based — explains why "quality" disputes are often view mismatches.

Software metrics: classification and the core set

• By what is measured: product metrics (size, complexity, defects in the artefact), process metrics (defect removal efficiency, review rate, cycle time), project/resource metrics (effort, cost, schedule, productivity).
• By derivation: direct/primitive (LOC, effort in person-months, defect count, duration) vs indirect/computed (productivity, quality, defect density, efficiency).
• By scale (measurement theory): nominal, ordinal, interval, ratio, absolute — a metric must not be used in arithmetic its scale does not permit (e.g. averaging ordinal severity codes is invalid).
• Size metrics:
  • LOC / KLOC — easy to count, but language-dependent, encourages verbose code, unavailable at estimation time.
  • Function Point (FP), Albrecht/IFPUG — size from functionality: five function types (EI external inputs, EO external outputs, EQ external inquiries, ILF internal logical files, EIF external interface files), each rated simple/average/complex, giving UFP; adjusted by 14 general system characteristics (GSC), each 0-5, into the VAF. Mark II FP (Symons) and COSMIC-FFP (ISO/IEC 19761) handle real-time/embedded better.
  • Use-case points, story points — Agile-era size proxies; story points are ordinal and team-relative, never comparable across teams.
• Complexity metrics:
  • McCabe's cyclomatic complexity V(G) — number of linearly independent paths; also the minimum number of tests for branch (decision) coverage in basis-path testing.
  • Halstead's software science — from operators and operands: vocabulary, length, volume, difficulty, effort, estimated bugs.
  • Information flow (Henry-Kafura) — fan-in, fan-out based module complexity.
  • Object-oriented: CK metrics suite — WMC (weighted methods per class), DIT (depth of inheritance tree), NOC (number of children), CBO (coupling between objects), RFC (response for a class), LCOM (lack of cohesion of methods). High DIT/CBO predicts fault-proneness.
  • Maintainability Index, cohesion/coupling measures, code churn, technical debt ratio (SonarQube-style).
• Defect and reliability metrics: defect density, defect age/latency, defect removal efficiency (DRE), defect leakage/escape rate, defect severity-priority distribution, MTBF = MTTF + MTTR, availability, failure intensity, reliability growth models (Jelinski-Moranda, Goel-Okumoto NHPP, Musa basic/logarithmic), Rayleigh/Putnam defect-discovery profile.
• Test metrics: test coverage (statement, branch, path, condition, MC/DC for safety-critical), test case effectiveness, test execution productivity, pass/fail/blocked rate, requirements traceability coverage, automation coverage.
• GQM (Goal-Question-Metric) paradigm — define the goal, derive questions, then choose metrics; prevents "measure what is easy".
• Limitations: metrics can be gamed (LOC, bug counts), aggregate numbers hide distribution, no single metric captures quality, and measuring people rather than processes destroys data integrity.

Reviews: types, the inspection process and metrics

• Why static review works: it finds defects (including missing requirements, which testing can never find), works on any artefact — SRS, design, code, test cases, user manual — and works before code executes. Typical review efficiency is a large fraction of total defects removed, at far lower cost than post-release fixes.
• Formality spectrum (IEEE 1028 defines the formal kinds):
  1. Desk check / peer deskcheck — informal, one reader, no records.
  2. Pair programming / over-the-shoulder — continuous, informal review.
  3. Walkthrough — author-led; the author presents and walks the artefact through scenarios; aims at understanding, training and consensus; no formal entry/exit criteria.
  4. Technical review — peer group of qualified technical staff evaluates suitability and conformance to standards; may be led by a trained moderator, documented defect list.
  5. Inspection (Fagan) — most formal: trained moderator, defined roles, checklists, entry/exit criteria, logged defects, measured rates, mandatory follow-up.
  6. Audit — independent evaluation (often by SQA group or external body) of conformance to standards, contracts and regulations; produces findings and non-conformance reports.
  7. Management review — progress, risk and resource adequacy against plan.
• Fagan inspection steps: Planning → Overview (kick-off) → Preparation (individual study, the step where most defects are actually found) → Inspection meeting (logging only, no solutioning) → Rework → Follow-up, with causal analysis feeding process improvement.
• Roles: Author/Producer (does not defend, answers questions), Moderator (plans, runs, enforces rules, confirms exit), Reader/Presenter (paraphrases the artefact), Inspectors/Reviewers, Recorder/Scribe. Managers do not attend inspections of their own staff's work.
• Golden rules: review the product, not the producer; log defects, do not fix them in the meeting; limit sessions to about two hours and keep checking rates moderate (commonly quoted around 100-200 LOC or a few pages per hour — tune to local data); use checklists; never use review data for appraisal.
• Review metrics: preparation effort, meeting effort, inspection rate (LOC or pages per hour), defect detection rate (defects per hour), defect density (defects per KLOC or per page), review effectiveness/yield, rework effort, cost per defect found.
• Entry criteria (artefact complete, compiles, static analysis clean, checklist available) and exit criteria (all logged defects dispositioned, rework verified, moderator sign-off).
• Modern practice: tool-assisted asynchronous review — pull/merge requests on GitHub, GitLab, Bitbucket or Gerrit with mandatory approvals and branch protection; static analysis gates (SonarQube quality gate, PMD, Checkstyle, SpotBugs, ESLint, Coverity) and DAST/SAST in CI pipelines; coverage thresholds in the build.

Standards, models and certification

• ISO 9001:2026 — the current edition of the generic quality-management-system standard (it replaced ISO 9001:2015); process approach, risk-based thinking, Plan-Do-Check-Act, management responsibility, internal audit, continual improvement. ISO/IEC/IEEE 90003 gives guidance on applying ISO 9001 to software. Certification is voluntary (commercially often contractually demanded).
• ISO/IEC/IEEE 12207 — software life-cycle processes; ISO/IEC/IEEE 15288 — system life-cycle processes; ISO/IEC/IEEE 29119 — software testing (concepts, processes, documentation, techniques); ISO/IEC 15504 / ISO/IEC 33001 series (SPICE) — process assessment; ISO/IEC 25010 and 25040 — product quality model and evaluation; ISO/IEC 27001 — information-security management (very common alongside quality certification in Indian IT); ISO/IEC 20000-1 — IT service management.
• IEEE standards commonly examined: IEEE 730 (SQA plans), IEEE 829 / now 29119-3 (test documentation), IEEE 1028 (reviews and audits), IEEE 1012 (verification and validation), IEEE 1061 (software quality metrics methodology), IEEE 830 (SRS; superseded by ISO/IEC/IEEE 29148 requirements engineering).
• CMMI (Capability Maturity Model Integration), now administered by ISACA. Staged representation — five maturity levels: 1 Initial (ad hoc), 2 Managed (project-level discipline: requirements management, planning, monitoring, configuration management, measurement, process/product QA), 3 Defined (organisation-wide standard process, tailoring, training, verification and validation, decision analysis), 4 Quantitatively Managed (statistical process control, quantitative goals), 5 Optimizing (continuous and innovative process improvement, causal analysis and resolution). Continuous representation rates individual practice areas on capability levels instead.
• Appraisal: a formal benchmark appraisal by a certified lead appraiser gives a maturity-level rating valid for a limited period (currently three years); it is an appraisal rating, not a certification, and is voluntary.
• Six Sigma for software: DMAIC (Define-Measure-Analyse-Improve-Control) for existing processes, DMADV/DFSS for new ones; 3.4 defects per million opportunities at six-sigma capability; tools — Pareto, fishbone (Ishikawa), control charts, FMEA, 5-Why, histogram, scatter, check sheet, stratification.
• Other improvement frameworks: TQM, Deming's PDCA and 14 points, Kaizen, Zero-defects (Crosby), Poka-yoke; ITIL for service quality; TMMi and TPI for test-process maturity.
• Indian context: certification/appraisal is done by accredited bodies; NABCB (under the Quality Council of India) accredits certification bodies in India, and BIS is the national standards body — BIS adopts many ISO/IEC software-engineering standards as IS/ISO/IEC equivalents, so cite the ISO/IEC number unless the exact IS number and year are known. STQC (Standardisation Testing and Quality Certification Directorate, MeitY) tests and certifies IT products, e-governance applications and security (Common Criteria/ISO 15408 through its labs). CERT-In directions and the DPDP Act, 2023 shape security and privacy quality requirements; under the DPDP Rules, 2025 the Data Fiduciary duties (notice, consent, breach intimation, erasure) and the omission of IT Act s.43A apply only from 13 May 2027 — until then IT Act s.43A and the SPDI Rules govern "reasonable security practices", which is what audit checklists currently cite. For safety-critical software, DO-178C (airborne), IEC 61508 / ISO 26262-style functional safety (automotive: ISO 26262 is commonly written ISO 26262; the automotive functional safety standard is ISO 26262 — quote the subject if unsure of the part number) and IEC 62304 (medical device software) apply; medical devices in India are regulated under the Medical Devices Rules, 2017 by CDSCO.

Key points

• SQA - process-focused, preventive, whole-life-cycle; SQC - product-focused, detective (reviews and testing).
• Verification - conformance to spec ("right way"); validation - fitness for need ("right product").
• CoQ - prevention + appraisal + internal failure + external failure; cost of a defect multiplies roughly ten-fold per phase escaped.
• McCall - 11 factors in operation/revision/transition; Boehm - uses → constructs → primitives; ISO/IEC 25010 - current product quality model (replaced 9126).
• Direct vs indirect metrics - counted (LOC, defects, effort) vs computed (density, productivity, DRE).
• Function points - size from EI, EO, EQ, ILF, EIF; UFP × VAF = FP; language-independent, available early.
• Cyclomatic complexity V(G) = E − N + 2P - independent paths and the minimum branch-coverage test count; keep modules under about 10.
• Halstead - volume V = N log₂ n; difficulty and effort derived from operator/operand counts.
• CK metrics - WMC, DIT, NOC, CBO, RFC, LCOM for object-oriented design quality.
• Defect density - defects per KLOC or per FP; the standard cross-project quality yardstick.
• DRE - defects removed in-phase ÷ (in-phase + escaped); ideal 1.0; the headline SQA process metric.
• Review hierarchy - deskcheck < walkthrough (author-led) < technical review < inspection (moderator, checklist, metrics) < audit (independent).
• Fagan steps - planning, overview, preparation, inspection, rework, follow-up; preparation finds most defects; the meeting logs, never fixes.
• IEEE 1028 - reviews and audits; IEEE 730 - SQA plan; IEEE 1061 - metrics methodology; ISO/IEC/IEEE 12207 - life-cycle processes; 29119 - testing.
• CMMI levels - 1 Initial, 2 Managed, 3 Defined, 4 Quantitatively Managed, 5 Optimizing; a benchmark appraisal gives a rating, not a certificate.
• ISO 9001:2026 - current QMS edition; 90003 applies it to software; certification is voluntary and granted by accredited bodies (NABCB accredits in India).
• Six Sigma - DMAIC/DMADV; 3.4 DPMO; STQC (MeitY) tests and certifies IT products in India.
• GQM - set goal, ask questions, then pick metrics; never measure individuals for appraisal.

Formulas and examples

Example 1 — Cyclomatic complexity and basis-path tests. A control flow graph for a module has N = 12 nodes and E = 16 edges, one connected component.

• V(G) = E − N + 2P = 16 − 12 + 2(1) = 6.
• Cross-check by predicates: the module has 5 binary decisions → V(G) = 5 + 1 = 6. Consistent.
• So 6 independent paths, and a minimum of 6 test cases for basis-path (branch) coverage; complexity 6 is in the low-risk band (≤10).
• If one if (a && b) were counted as one predicate instead of two conditions, V(G) by the predicate rule would come to 5 — compound conditions must be counted per simple condition.

Example 2 — Function points, productivity and cost (₹). A college-ERP module: EI = 22 (average, weight 4), EO = 16 (average, 5), EQ = 10 (simple, 3), ILF = 6 (average, 10), EIF = 3 (average, 7). The 14 GSCs sum to Σ Fi = 42.

• UFP = (22×4) + (16×5) + (10×3) + (6×10) + (3×7) = 88 + 80 + 30 + 60 + 21 = 279.
• VAF = 0.65 + 0.01 × 42 = 1.07.
• FP = 279 × 1.07 = 298.53 ≈ 299 FP.
• Effort actually spent = 30 person-months → productivity = 298.53 ÷ 30 = 9.95 FP per person-month.
• At a loaded cost of ₹1,20,000 per person-month, total = ₹36,00,000 → cost per FP = 36,00,000 ÷ 298.53 = ₹12,059 per FP.
• 40 defects found in all → defect density = 40 ÷ 298.53 = 0.134 defects per FP.

Example 3 — Defect density and DRE across phases. A 45,000-LOC (45 KLOC) release. Defects found: requirements review 30, design review 48, code review 90, unit testing 72, system testing 60, UAT 25; field defects in the first 3 months = 15.

• Defects found before delivery, E = 30 + 48 + 90 + 72 + 60 + 25 = 325; escaped D = 15.
• Overall DRE = 325 ÷ (325 + 15) × 100 = 325 ÷ 340 × 100 = 95.59%.
• Total defect density (all phases) = 340 ÷ 45 = 7.56 defects per KLOC; delivered (field) defect density = 15 ÷ 45 = 0.33 per KLOC.
• Static reviews alone found 30 + 48 + 90 = 168 of 340 = 49.4% of all defects, before a single test ran.
• Phase DRE of code review, treating all 172 later-found defects as its escapes: 90 ÷ (90 + 172) × 100 = 34.4% — showing why DRE must always be quoted with its scope.

Example 4 — Review economics and inspection rate. An SRS of 60 pages is inspected by 4 inspectors: preparation 2 h each, meeting 3 h with all 4 plus a moderator (5 people), rework 10 h by the author.

• Effort = (4 × 2) + (5 × 3) + 10 = 8 + 15 + 10 = 33 person-hours.
• 48 defects logged → defect detection rate = 48 ÷ 33 = 1.45 defects per person-hour; defect density = 48 ÷ 60 = 0.8 defects per page.
• Inspection rate for the meeting = 60 pages ÷ 3 h = 20 pages per hour — far above the usual guideline of a few pages per hour, so the session was almost certainly too shallow; split into several sessions.
• Economics: at ₹600 per person-hour, review cost = 33 × 600 = ₹19,800, i.e. ₹412.50 per defect. If each of these defects escaping to production cost ₹10,000 to fix, avoided cost = 48 × 10,000 = ₹4,80,000 → ROI ≈ 23× (4,80,000 ÷ 19,800 = 24.2, net 23.2 times the review spend).

Example 5 — Reliability, availability and DPMO. A payment gateway logs 6 failures in 3,000 operating hours, with an average repair time of 1.5 h.

• MTTF = 3,000 ÷ 6 = 500 h; MTBF = 500 + 1.5 = 501.5 h; λ = 1/500 = 0.002 failures per hour.
• Availability = 500 ÷ 501.5 × 100 = 99.70% — below a "three nines" (99.9%) SLA, so a penalty clause would trigger.
• Reliability for a 24-hour window: R(24) = e^{−0.002 × 24} = e^−0.048 = 0.953, i.e. about 95.3% chance of no failure in a day.
• Quality in DPMO terms: 1,20,000 transactions processed, 1 opportunity each, 420 failed → DPMO = (420 ÷ 1,20,000) × 1,000,000 = 3,500 DPMO (about 4.2 sigma) against 3.4 DPMO at six sigma.

Where it is used

• Indian IT services and GCCs: dedicated QA/quality-process groups at TCS, Infosys, Wipro, HCLTech, Cognizant, LTIMindtree, Tech Mahindra, Capgemini India and captive centres run ISO 9001 and ISO/IEC 27001 QMS plus CMMI-appraised delivery processes; metrics dashboards (defect density, DRE, schedule and effort variance, review yield) are contractual in managed-services deals.
• Product engineering and GCCs: mandatory pull-request review with two approvers, SonarQube quality gates, coverage thresholds and DORA metrics in CI/CD at Flipkart, Zoho, Freshworks, Zerodha, PhonePe and banking product firms.
• Government and e-governance: STQC (MeitY) testing, certification and security audit of e-governance applications, UIDAI/Aadhaar-linked systems, digital-payment and smart-city software; CERT-In empanelled auditors for security audits before go-live.
• Regulated and safety-critical work: ISRO, DRDO, HAL and avionics suppliers (DO-178C), automotive software at Bosch, Continental, KPIT, Tata Elxsi (automotive functional safety plus Automotive SPICE), medical-device software (IEC 62304, CDSCO/Medical Devices Rules, 2017), and BFSI software under RBI IT-governance and cyber-security framework expectations.
• Job roles: QA engineer, SDET, quality analyst, SQA/process consultant, internal auditor, Six Sigma Green/Black Belt, software metrics or EPG (engineering process group) analyst, release manager.
• Exams and certifications: university SE/SQA papers, GATE CS (complexity, FP, reliability), ISTQB Foundation and Advanced, CMMI appraisal team member, ISO 9001 lead auditor, Six Sigma belts.

Interview questions

• Q: What is the difference between QA and QC? A: QA is process-oriented and preventive — defining and auditing the process so defects are not injected; QC is product-oriented and detective — reviewing and testing a specific deliverable to find defects already present.
• Q: Verification or validation — which does a code review perform? A: Verification: it checks the code against its specification and standards without executing it. Validation needs execution against real user needs, typically in system or acceptance testing.
• Q: Why review when we can test? A: Reviews find defects far earlier and cheaper, find classes of defects testing cannot (missing or ambiguous requirements, unmaintainable design, missing error handling), and apply to non-executable artefacts like the SRS and test plan.
• Q: Define cyclomatic complexity and state one practical use. A: V(G) = E − N + 2P, the number of linearly independent paths through a module; it gives the minimum number of test cases for branch coverage and flags modules above about 10 for refactoring.
• Q: How do you compute DRE and what does a low value mean? A: DRE = defects found before release ÷ (before + after) release. A low value means defects are escaping to customers, so the in-process review and test filters are weak or coverage is thin.
• Q: Why are function points preferred to LOC for estimation? A: FP measures delivered functionality, so it is available from the requirements before any code exists and is independent of language, coding style and developer verbosity, which makes cross-project comparison valid.
• Q: What are the roles in a Fagan inspection and which step finds most defects? A: Moderator, author, reader, inspectors and recorder; individual preparation finds most defects, while the meeting mainly consolidates and logs them.
• Q: Why must defects not be fixed during a review meeting? A: Solutioning consumes the meeting, derails coverage of the rest of the artefact and lets the strongest voice decide; defects are logged, dispositioned by the author in rework, and verified in follow-up.
• Q: What is the difference between ISO 9001 certification and a CMMI level? A: ISO 9001 (current edition ISO 9001:2026) certifies a quality management system against a pass/fail standard through an accredited certification body; CMMI gives a maturity-level rating from 1 to 5 through a benchmark appraisal by a certified lead appraiser, valid for a limited period — a rating, not a certificate. Both are voluntary.
• Q: 100% code coverage was achieved — is the software defect-free? A: No. Coverage shows which code ran, not that outputs were asserted correctly, nor that missing code, wrong requirements, concurrency, performance or security defects are absent.
• Q: Which metrics would you put on a release-readiness dashboard? A: Open defects by severity, defect density, defect arrival and closure trend, DRE to date, test and requirement coverage, failed-test ratio, static-analysis blocker count and build stability — with a trend, not a single snapshot.
• Q: How does SQA work in Agile, where there is no separate QA phase? A: Quality is built in — definition of done, pair programming and pull-request review, CI with automated unit, API and regression suites, static-analysis quality gates, test-first development, and retrospectives acting as causal analysis; the metrics shift to escaped defects, lead time, build-failure rate and technical debt.
• Q: Why should review data never be used in appraisals? A: Engineers then hide defects and avoid raising them, which destroys both the data and the review's purpose; reviews measure the artefact and the process, never the person.

Common mistakes

• Treating SQA as "testing" or as a team that only runs test cases; forgetting audits, process definition, metrics and training.
• Swapping verification and validation, or claiming a walkthrough is more formal than an inspection.
• Arithmetic slips in V(G): using E − N + 1, omitting 2P, or counting a compound condition a && b as one predicate instead of two.
• Forgetting the VAF step in function points (reporting UFP as FP), or applying FP weights without stating simple/average/complex.
• Quoting DRE or defect density without the denominator and scope — "95% DRE" is meaningless without the phase and the escape window; comparing defect density across different languages or counting rules.
• Assuming MTBF = MTTF (MTBF = MTTF + MTTR), or confusing reliability R(t) with availability.
• Claiming certification or appraisal is legally mandatory — ISO 9001 and CMMI are voluntary (often contractually required); and citing superseded references (ISO 9126 instead of ISO/IEC 25010, IEEE 829 instead of ISO/IEC/IEEE 29119-3, ISO 9001:2015 instead of ISO 9001:2026).
• Fixing defects in the review meeting, inviting line managers to inspect their reports' work, or reviewing 60 pages in one session.

Related topics

Software testing: levels and techniques · Test tools: Selenium, JMeter and Postman · SDLC and Agile · Software project management and estimation · Software configuration management and DevOps · Software reliability and maintenance