Re: Test Driven Development

From: H. S. Lahman (h.lahman_at_verizon.net)
Date: 12/08/03 

Date: Mon, 08 Dec 2003 21:07:30 GMT

Responding to Erwin...

>>>>[Consider the requirements that (a) the average time to process 10000
>>>>transactions should be no more 10 seconds and (b) 10000 transactions
>>>>should never take more than 30 seconds to execute. If the application
>>>>processes a wide variety of transactions with potentially varying
>>>>execution times, then neither of these requirements is testable as is.
>>>
>>>
>>>They are testable or at least demonstrable. You would _prefer_ to have a
>>>rate monotonic scheduler in your operating system, and you need
>>>competence at queuing theory. With known input scenarios and known
>>>transaction timing data, the necessary analysis can be done on a
>>>spread***. I used to make a living doing it.
>>
>>They are /measurable/, but that is not the same as testable. The
>>requirements are incomplete as far as testability is concerned because
>>there is no specification of the sampling for the measurements.
>
>
> The requirements are not only measurable; they're testable if you do it
> right.
>
>
>>For example, consider two transactions: A takes 0.9 ms to execute while
>>B takes 1.1 ms to execute. The average requirement, (a), is meaningless
>>unless one has a distribution on the expected frequency of occurrence of
>>A and B in a random sample of 10K transactions. The tests will fail
>>whenever one supplies more Bs than As in the test sample. But what if
>>the problem space constrains things such that there are always exactly
>>three As for every B? The test samples with more Bs are then measurable
>>but irrelevant; in fact, the /only/ test sample that is relevant to
>>testing the requirement is one with exactly 750 As and 250 Bs (ignoring
>>order).
>
>
> That's why Boehm emphasizes early definition of test scenarios.

We seem to have different definitions about what 'testable' means. In
this context I believe it means that one can make a measurement to
unambiguously determine whether the implementation has correctly
satisfied the requirement.

When defining those scenarios, how does one know that the only useful
test scenario (i.e., where pass/fail is meaningful insofar as whether
the requirement is satisfied) is 750/250? That is, the test result for
any other mix is ambiguous. There is nothing in the statement of the
requirement to indicate that.

IOW, the requirement identifies a statistical measurement without
defining how it relates to correctness of the implementation. My point
is that for a statistical requirement like the example to be testable,
it must also define the sampling criteria so that a pass/fail result for
a given test case unambiguously determines whether the requirement is
satisfied (pass) or not (fail).

*************
There is nothing wrong with me that could
not be cured by a capful of Drano.

H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions -- Put MDA to Work
http://www.pathfindermda.com
(888)-OOA-PATH