se logo Chair of Software Engineering
eth logo
   

 

RESEARCH PROJECT DESCRIPTION: Contract-based testing

Automatic test case generation for contract-equipped object-oriented software

 

Overview | Rationale | Basic concepts | Results so far | Download | Project members | Funding | Publications | Contact

Overview

Assertions, as described in the Design by Contract(TM) software development methodology, contain the specification of the system. Thus, they provide an automated oracle for the testing process. Based on this insight, we are developing a tool for fully automatic testing of object-oriented code.

Rationale

Properly testing medium to large-scale applications requires huge effort. Practice shows that resources are often under-allocated and, furthermore, testing tools are inadequate. Hence, the idea of automatic testing. If a developer could write his code in the morning, then, before leaving for lunch, press a button, and, upon returning from lunch, have a report saying "Your method m1 of class C1 contains a bug. Tests passed for all other methods of your class.", we think it would be safe to assume that released software products would be far less buggy. Testing would only require the "time and effort" of a machine, and, ideally, would itself be much less error-prone.

Basic concepts

The goal of our work is to fully automate the entire testing process. The difficulty lies in automating:

  • Generation of input values
  • The oracle

For both of these tasks, the knowledge of a human tester is usually required. However, when the specification of a system is part of its source code, an automated oracle becomes freely available. This idea is the basis of Design by Contract(TM) and thus our approach takes full advantage of existing contracts in Eiffel code. However, using contracts as an automated, already available oracle raises some problems:

  • Contracts very rarely (if ever) contain the full specification of the software under test. To give only an example, the postcondition of a routine should state both what it changes and what it doesn't change (for instance, a print method should state that it changes the appearance of the screen by adding some text to it, but it does not increase the value of a counter, connect to a database, etc.). When dealing with systems that operate in open environments, such a task is impossible with the mechanisms currently offerred by Design by Contract(TM) as it is implemented in Eiffel. This situation is known as the "frame problem".
  • Contracts may not only be incomplete, but also incorrect: they might not represent the real specification of the software, because of mistakes made by developers when translating the natural language specification into programming language constructs, or because the requirements document itself is incorrect or incomplete.

Automatic generation of input data also presents some difficulties:

  • The precondition of the routine under test must be fulfilled, otherwise the test case is invalid. Strong preconditions might never be satisfied (in a restricted time frame) by a random input value generator.
  • Generated values, in addition to satisfying the precondition of the tested routine, must fulfill some test adequacy criteria (such as coverage of code or specification, etc).

There are several further issues to consider when the testing process becomes fully automatic:

  • What degree of control can the user have over it? How can we parameterize it so that the user can adapt it to his needs?
  • Automatic integration of test results
  • Fault isolation
  • Finding the minimal failure-reproducing examples

What we have achieved so far

AutoTest Screenshot
(Click on image to see example results.)

The main contribution of our work is a fully automatic testing process implemented in a tool called AutoTest. A great amount of related work exists on this topic. However, most approaches focus on automating only certain steps, not the entire process. Many approaches that claim to be completely automatic turn out to require manual intervention or leave so many cases uncovered, that they become unusable for practical purposes. Furthermore, one of the main advantages in our work is that we can test our approach on full fledged, industrial scale applications; we do not need to create toy examples (which might be biased too), as is the case for a vast field of related work that addresses Java code equipped with JML contracts.

For further details please refer to the AutoTest documentation.

We have used our tool to test production-quality libraries. Here are just a few examples representing some classes (taken from the EiffelBase 5.6 library) that we tested and how many distinct bugs we found in each of them, counting also bugs present in suppliers of the classes which do not allow the tested classes to work correctly:

  • STRING - 12
  • BOUNDED_STACK - 11
  • HASH_TABLE - 23
  • ARRAYED_SET - 27
  • LINKED_LIST - 25

Download

Current project members

Bertrand Meyer, Andreas Leitner, Ilinca Ciupa

Funding

ETH funding

Publications

  • Ciupa, I., Pretschner, A., Leitner, A., Oriol, M., Meyer, B., "On the Predictability of Random Tests for Object-Oriented Software", accepted for publication in Proceedings of ICST'08: IEEE International Conference on Software Testing, Verification and Validation 2008, (Lillehammer, Norway), April 2008
  • Ciupa, I., Leitner, A., Oriol, M., Meyer, B., "Experimental Assessment of Random Testing for Object-Oriented Software", Proceedings of ISSTA'07: International Symposium on Software Testing and Analysis 2007, (London, UK), July 2007
  • Leitner, A., Eugster, P., Oriol, M., Ciupa, I., "Reflecting on an Existing Programming Language", to appear in Proceedings of TOOLS Europe 2007 - Objects, Models, Components, Patterns, (Zurich, Switzerland), June 2007
  • Meyer, B., Ciupa, I., Leitner, A., Liu, L., "Automatic Testing of Object-Oriented Software", in Proceedings of SOFSEM 2007, Current Trends in Theory and Practice of Computer Science, (Harrachov, Czech Republic), January, 2007 (also published as ETH Technical Report 538 11/2006)
  • Leitner, A., Ciupa, I., Meyer, B., Howard, M., "Reconciling Manual and Automated Testing: the AutoTest Experience", in Proceedings of the 40th Hawaii International Conference on System Sciences (Waikoloa, Big Island, Hawaii), January, 2007
  • Ciupa, I., Leitner, A., "Automatic Testing Based on Design by Contract", in Proceedings of Net.ObjectDays 2005: 6th Annual International Conference on Object-Oriented and Internet-based Technologies, Concepts, and Applications for a Networked World (Erfurt, Germany) , September, 2005.
    • Paper presented in the Developer Session of the International Workshop on Software Quality (SOQUA 2005)
  • (The draft paper describing a tool called Test Wizard from which AutoTest inherits some of its features) Karine Arnout, Xavier Rousselot, and Bertrand Meyer, "Test Wizard: Automatic test case generation based on Design by Contract(TM)". Available at: http://se.inf.ethz.ch/people/arnout/arnout_rousselot_meyer_test_wizard.pdf.

Contact

Andreas Leitner, Ilinca Ciupa

 

 

Last update: 25-Mar-2008 by Ilinca Ciupa