Skip navigation.

Evaluating Test Suites and Adequacy Criteria using Simulation-Based Models of Distributed Systems

Publication Type:

Journal Article


IEEE Transactions on Software Engineering, IEEE Computer Society, Volume 34, Issue 4 (2008)


Distributed systems; discrete-event simulation; test adequacy criteria; fault-based analysis


Test adequacy criteria provide the engineer with
guidance on how to populate test suites. While adequacy criteria
have long been a focus of research, existing testing methods do not
address many of the fundamental characteristics of distributed
systems, such as distribution topology, communication failure,
and timing. Furthermore, they do not provide the engineer with
a means to evaluate the relative effectiveness of different criteria,
nor the relative effectiveness of adequate test suites satisfying
a given criterion. This paper makes three contributions to the
development and use of test adequacy criteria for distributed
systems: (1) a testing method based on discrete-event simulations;
(2) a fault-based analysis technique for evaluating test suites and
adequacy criteria; and (3) a series of case studies that validate the
method and technique. The testing method uses a discrete-event
simulation as an operational specification of a system, in which
the behavioral effects of distribution are explicitly represented.
Adequacy criteria and test cases are then defined in terms of this
simulation-based specification. The fault-based analysis involves
mutation of the simulation-based specification to provide a foil
against which test suites and the criteria that formed them can
be evaluated. Three distributed systems were used to validate
the method and technique, including DNS, the Domain Name


© 2008 IEEE. Personal use of this material is permitted. However, permission to
reprint/republish this material for advertising or promotional purposes or for creating new
collective works for resale or redistribution to servers or lists, or to reuse any copyrighted
component of this work in other works must be obtained from the IEEE.

RutherfordCarzanigaWolfTSE2008.pdf2.85 MB