Re: Seriously struggling with C

Ian Collins wrote:
Flash Gordon wrote:

The sense isn't false if the tests are good and written first. In my opinion, test added after the code is written are second rate.


On one of the projects I worked on early in my career (written in Pascal) I don't think any of us ever used the debugger and we did not write the tests until after we wrote the code, and even then the tests were system level. However, when we did write the tests we went a long way out of our way to try to think of every single way we could break the system. Running out software with some of the kit switched off, unplugging cables whilst it was running, swapping 525 line cards in to what was meant to be a 625 line system (we had a 525 line variant with the same code base) etc. During that testing we found a significant number of bugs. During the remaining 10 years of my time in the company, through many versions of the SW, including getting fresh graduates to no experience or domain knowledge to do changes, the customers found very few bugs. However, rerunning these manual system level tests after changes *did* find problems, and each time the customer found a bug we extended our tests to catch the bug.

What you are describing are what I'd call acceptance tests, working on the code form the outside confirming that the system behaves as expected by the customer.

No, we were not trying to confirm it behaved as expected. Quite the reverse, we were being as devious as possible in trying to prove it did *not* work. We succeeded in that. Our customer acceptance tests, on the other hand, where designed to demonstrate it worked as expected and took far less time.

> Unit tests written as part of the TDD process test
individual components of the system, down to individual function level form the inside. They test the logic according to the programmer's understanding.

We did tests at the system level designed to exercise the specific functions, and sometimes specific if statements or exception trap.

I'd always recommend both types of testing.

As would I.

So tests written after the code can be *very* effective, but you have to actively *try* to break the code in your testing rather than trying to prove that it is correct.

Indeed. But not unit tests written after the code.

In this instance they proved highly effective. In terms of customer reported bugs and customer perception of the quality of the code, it is probably about the most successful production project I have come across. 50000 lines of code and I think under 10 customer reported bugs in 10 years. Even if I am a factor of 10 out that is still only 1 bug per 500 lines of code over a 10 year period, or 1 but per 5000 lines per year.

I've seen far more problems with testing written either by an independent team or where the tests have been designed before the coding where people have been trying to prove the code correct than I have with tests written after the fact with people actively trying to prove the code is wrong. Of course, the ideal would probably be to write the test first but to write them as an active attempt to prove the software *wrong*.

When doing TDD, every unit test fails until the code to make it pass is written.

That could be said of any test. If the test checks if 5 is returned when the input is 3, then until you have put something inside the function body of course it will fail.

When we were testing this code we tried to get the code to fail by having devices absent that should always be there, intermittent communications over what we knew were definitely reliable links, trying to force it to do division by zero (faking it so that it missed the reference frequency in a frequency response test) and so on. This is something you can apply to unit testing, system level testing, or any other form of testing, but it is also something that in my experience many people do *not* do whatever form of testing they are doing.

I'm not disputing the benefits of TDD, nor saying that today I would do things the same way we did in 1990 in the Test Engineering Department (making test equipment) where I used to work. I'm saying that:
1) Methods other than TDD can be successful in the right situation
2) Whatever testing you are doing the tests should be designed to prove
in every conceivable way *and* in inconceivable ways that the code is
*wrong*.

Part of 2 is testing boundary conditions, part is forgetting what the requirements on external systems are and what is possible for them (You know that a user can't press a key 1000 time a second, don't you. Forget that because you have forgotten that a HW fault could have the same effect as a user pressing the key 1000 time a second) and part is working on the assumption that you know damn well there is a bug somewhere that you couldn't possibly conceive, so you need to test the inconceivable.

BTW, I've seen a HW fault causing the same effect as a user pressing a key at a stupidly high rate. The logic circuit basically became an oscillator for as long as a key was held down, so I know damn well that what most would consider inconceivable is not only possible theoretically, but sometimes actually happens in real life.

I also spent time as I say working in the Test Engineering Department, and since we were building test equipment (which also had to test itself) we developed the attitude of assuming that the SW has to survive and continue working properly as much as possible even if fundamental parts of the system are failing in ways you can't conceive of, a philosophy that I find very useful.
--
Flash Gordon
Living in interesting times.
Web site - http://home.flash-gordon.me.uk/
comp.lang.c posting guidlines and intro - http://clc-wiki.net/wiki/Intro_to_clc
.

Relevant Pages

  • Re: Code correctness, and testing strategies
    ... Platforms for which unit tests are hard to setup/run. ... There is no good way of reusing your prototype code in TDD. ... Unit tests won't help with bugs they ... It makes refactoring much easier. ...
    (comp.lang.python)
  • Re: Code correctness, and testing strategies
    ... production, but version 2 will definitely go into production, so I ... let's go back and write unit tests for all the existing code". ... As for your other points - agreed, bugs getting to the customer is not ...
    (comp.lang.python)
  • Re: Number of bugs - statistics
    ... which is not important for most enterprise server users ... Customer uses the device as part of some machines he sells. ... Neither the manufactorer nor the customer do want to _ever_ become ... proven and prioritized bugs maybe better to say. ...
    (Linux-Kernel)
  • Re: Microsoft Release VS2003 SP1
    ... many Updates did Delphi 7, 2005, 2006 have when it was still relevant? ... I believe fixing bugs early on should be ... considered better customer support, instead of fixing bugs to products ... mean you should toss us aside just because we didn't buy the latest ...
    (borland.public.delphi.non-technical)
  • Re: Delphi 8 common quality
    ... bugs reports that are found there. ... > relations with your customer? ... because of the price of Borland's tools. ... > customer (within bounds of reason) is good for the business. ...
    (borland.public.delphi.non-technical)