Re: A critique of test-first...

From: Laurent Bossavit (laurent_at_dontspambossavit.com)
Date: 11/15/04

  • Next message: Arthur J. O'Dwyer: "Re: Need idears to solve an algorithmic problem" 
    Date: Mon, 15 Nov 2004 23:23:53 +0100

    > Even if we break it down to the minimal amount of analysis per transform kind,
    > it will still amount to 1kloc+. (BTW: this is based on recent experience,
    > not hypothetical).

    I suspect we differ on what we call a "unit" test, or a "real" test.

    Here is a thought experiment. Suppose you suspect that, somewhere in
    these 1000 lines of optimizer code for a given transformation, there is
    one nasty bug.

    I would suppose that you will instrument the code for debugging - put a
    printf in there, or activate one that you've already put there for that
    purpose.

    The next step might be running the compiler on a test source file that
    is carefully designed to exercise, among other paths, one that causes
    the printf statement to be activated.

    Now we have a string that has been output to stdout. If the debug
    instrumentation is at all helpful, we have a precise idea what that
    string should be. If the string actually output differs from that, we
    have a "smoking gun" that the bug is indeed, in the main, what and where
    we suspected.

    We could make this more efficient by writing a small function that
    examines stdout for us, and compares the particular line in the output
    that we're interested in with the value we expect. Couldn't we ?

    Maybe we don't need to exercise the whole compiler, either. The analysis
    is, I suppose, running over part of the partially compiled code - mostly
    assembly with unresolved jump locations, or somesuch. Maybe we can store
    the binary representation of that instead of the source file. And do we
    really need more than a dozen or a few dozen bytes of it in order to
    expose the bug ? I may be wrong (I've only written one compiler in my
    life, with fairly simple peephole optimizations), but I'll assume that's
    about right.

    Here's the one part I'm not 100% sure of. Is the 1000 lines of analysis
    code one big function, or is it broken down further into functions ? In
    the latter case, presumably we can massage the input data further still,
    until we know exactly what *would* be passed into the function that has
    the debug printf, if we were encountering the bug in real use. Instead
    of a printf, perhaps we can devise other ways to get the output back -
    say in a return value, or a parameter passed by reference.

    So we can write a test function that:
    - constructs the input data we need
    - passes it into the relevant function of the analysis module
    - receives an encoded "status" string as one result of the function
    - compares the string it received with an "expected" string

    Well, surprise - that test function is an XP-style unit test. Ex
    hypothesi (unless I've made a mistake in the above) it tests something
    that is considerably less than 1000 lines of code. (I haven't put a
    number on that size, but it's "whatever the function length you consider
    reasonable"). Is it a "real" test ? Well, again ex hypothesi, I wrote it
    because of an actual bug, so presumably it tests something relevant.

    Instead of writing these tests when a bug surfaces, when arguably it is
    too late, XP suggests that they should be written before the code. This
    helps developers focus on testability in their design (they can't get
    away with a 1000-line function), avoids confirmation bias, and helps
    with debugging in much the same way that instrumentation does, when a
    rare bug does surface. Though actually the beginning of this paragraph
    should read "In addition to", not "Instead of" - in XP it's considered a
    best practice to write a new test for any bug found.

    Laurent

  • Next message: Arthur J. O'Dwyer: "Re: Need idears to solve an algorithmic problem"

    Relevant Pages

    • Getting Method objects without reflection
      ... This is a shameless plug to try to get votes for a recent bug I've ... -it's inelegant to use Strings to perform at runtime what the compiler ... Instead of specifying String names ...
      (comp.lang.java.programmer)
    • Re: Benefit of not defining the order of execution
      ... The compiler did not allocate a buffer for each return string. ... Bug was acknowledged but not fixed, ...
      (comp.lang.c)
    • Re: Most Interesting Bug Track Down
      ... This could be initialised with a string as follows: ... I had to read through the code in detail before the bug ... but causing subtle problems elsewhere. ... turned out that a compiler upgrade was the culprit. ...
      (comp.lang.c)
    • Re: replace substring
      ... to a char * string, but you have done nothing to make sure its ... and changing the cast (which is all the compiler will complain about) will still leave it broken. ... a post correcting errors is meant to have at least one error! ... With the options I normally use the compiler I normally use would have reported my bug too, so that is not such a big benefit. ...
      (comp.lang.c)
    • Re: Text box throws an Error 7 Out of Memory with less than 27K in it!
      ... I'd have to agree with Bob here, ... >> those exact numbers and then concatenates that many more, ... I used to work supercomputer compiler support at CDC. ... > My favorite such bug at CDC was a benchmark that ran instantaneously. ...
      (microsoft.public.vb.enterprise)