Re: Test Driven Development

From: H. S. Lahman (h.lahman_at_verizon.net)
Date: 12/09/03 

Date: Tue, 09 Dec 2003 20:35:57 GMT

Responding to Erwin...

>>We seem to have different definitions about what 'testable' means. In
>>this context I believe it means that one can make a measurement to
>>unambiguously determine whether the implementation has correctly
>>satisfied the requirement.
>>
>>When defining those scenarios, how does one know that the only useful
>>test scenario (i.e., where pass/fail is meaningful insofar as whether
>>the requirement is satisfied) is 750/250? That is, the test result for
>>any other mix is ambiguous. There is nothing in the statement of the
>>requirement to indicate that.
>>
>>IOW, the requirement identifies a statistical measurement without
>>defining how it relates to correctness of the implementation. My point
>>is that for a statistical requirement like the example to be testable,
>>it must also define the sampling criteria so that a pass/fail result for
>>a given test case unambiguously determines whether the requirement is
>>satisfied (pass) or not (fail).
>>
>
>
> That's an old argument that I can remember having in the mid-70s. I
> suppose you can substitute 'demonstrable' for 'testable'. We would
> measure the software performance at the task level and in the real-time
> operating system and then do a very sophisticated queueing analysis (See
> Chapter 3 of Volume 2 of Kleinrock for the theory) to demonstrate that
> the system could handle all realistic loads. (The invention of the
> spread*** was a godsend for this--up until then, we had used
> specialized programs to do the same thing.) This model was
> experimentally calibrated to system performance in integration tests to
> demonstrate its validity.

I am familiar with queuing analysis (or at least I was 20 years ago) and
I believe I understand the technique you used. The queuing analysis
will only be as good as the definitions of the distribution of the
transaction types and the distribution of transaction interarrivals.
Those distributions were not specified in the requirement.

One can make pretty good assumptions about them (e.g., interarrivals are
Poison) but they are just that: assumptions. If the requirements are
going to specify statistical measures, they should also specify the
sampling context for those measures.

Alternatively, one can derive the distributions empirically by sampling
the real distributions through prototypes or existing systems (when
feasible). But that just moves the needed specification from the
distributions themselves to the parameters of the empirical sampling
(e.g., confidence of 1 sigma? 0.1 sigma?).

You seem to have combined these two techniques by making <probably quite
standard> assumptions about the distributions for the tests and then
validating those assumptions after the fact through confidence testing
with actual sampling of the system operation. While this does a good
job of isolating the ambiguity to a second order effect that is also
standardized, there is still ambiguity -- confidence limit on the
distribution validation.

I still argue that the confidence limit for the after-the-fact
validation is, itself, an assumption that introduces ambiguity into the
test results. IOW, the customer needed to sign off on it as a
requirement context to bless the test results. [Though I admit it is
probably the sort of thing that, in practice, only gets reviewed after
the shuttle crashes. B-)]

*************
There is nothing wrong with me that could
not be cured by a capful of Drano.

H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions -- Put MDA to Work
http://www.pathfindermda.com
(888)-OOA-PATH