This is the mail archive of the mailing list for the Mauve project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Improving mauve (Was: Re: Tests, Documentation)

Hallo Sascha,

I'm taking the discussion off the Classpath mailing list, and moving it to mauve, where it belongs ;)

Sascha Brawer wrote:

test-writing has to be made to feel at least as
engaging as code-writing, if it's to be successful.

I imagine that this will be hard: For most people (including myself),
it's so much more fun to write code that actually does something.

In the long run, test writing needs to become so straightforward that it can be largely replaced by automatically generated tests.

Example: you can often find interesting types of bugs in methods by throwing somehow preconditioned input patterns at them, and checking if the output/object state still adheres to some post-condition.

The hard way to do this is to do it manually, for every bug pattern we observe, in each class it could occur. Which is what mauve currently makes us do basically, and I don't know if JUnit is much better in that respect.

The nice way to do it would be to have helper methods/classes that

a) allow us to generate interesting (i.e. possibly failure inducing) input patters.
b) it should be possible to optionally select only the input that matches a pre-condition, since some tests for methods may only make sense if the input instances (fixtures) are in some specific state [1].
c) have utility methods for running a test case over all fixtures, a subset, all pairs, etc.
d) have utility methods for checking post-conditions.

basically, I want to move away from seeing tests like this in every class:

  FieldPosition fp2 = new FieldPosition(21);
  harness.check(fp.equals(fp2) == true, "equals (true)");

to tests like this:

harness.testAllPairs(getFieldPositionFixtures(), EqualsTestCase);

or in another class, that also implements equals:

harness.testAllPairs(getSomeOtherFixtures(), EqualsTestCase);

where the post-condition logic of the equals API (or bug pattern, or whatever you need to test) is encoded in the test case, and what's being tested is if the 'interesting' input triggers a bug (i.e. an unexpected deviation) or not. The nice thing is that interesting input can be generated quite easily, in general. Even more so, if you can use reflection. [2]

I think we need to separate test input generation from test logic to make tests easier to write. My typical test for a new class would look something like this:

void testClass(TestClass cl) {

void testConstructors(TestClass cl) {
  for each constructor
    try creating an instance with a set of parameters
    test if the result meets post-condition of that constructor
      [ like should throw an exception if input is null ]

void testMethods(TestClass cl) {
   for each method
     get a set of fixtures that can be used to test the method
     invoke method on fixtures
     test if the result it meets post-condition of that method

I hope that the work in writing the test case can move away from grinding out similar test cases, to specifying interesting inputs, pre- and post-conditions for the methods in something like TestClass.

It's all a work in progress, as I'm trying to figure out nice, simple ways to do this. I've converted the FieldPosition tests into a more modular form (attached) [3]. I'll use that to try out my ideas in this regard. Critique of what I have so far, and the directions I have in mind would be very welcome, as most of you guys have been using mauve for years without that much of problem ;)

dalibor topic

[1] If I want to test for example how behaves on a closed stream, I want to test just with the BufferedReader test fixtures that are closed. Testing with others wouldn't be that useful. ;) That's the simple part.

The read-should-throw-an-exception-if-reader-is-closed is specified by the API of Reader.close(). So what I really want is a generic test for Reader.close(), that allows me to pass different types of readers that overwrite read() so that I can test if they still comply to the API in the test case for close() for the classes extending Reader. Otherwise I'd still have to copy the test code for each reader I write the test code for.

[2] It shouldn't be too hard to write some fixtureGenerator(Constructor cons, Object[] types) method for example, that automatically populates a fixture set given a constructor, and an array of primitive types the constructor receives, by creating all possible 'interesting' instances of fixtures.

[3] As FieldPosition has different (accessible) members depending on the API version, I separated 1.1 specific fixtures from those that would only be un-equal under 1.2. That resulted in a separation of tests into a generic part, and an API release specific part that calls the generic part with API release specific fixtures.

Attachment: fpos.tar.bz2
Description: BZip2 compressed data

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]