Re: Test first as specification

On Fri, 12 Aug 2005 12:06:10 +0200, Laurent Bossavit wrote:

>>> I'm not sure I understand the spec. Does it specify a language for which
>>> the only valid input string is an infinite string of a's ?
>>
>> You're right. It wasn't my intent. Let's add one more production: S->a. Now
>> the set of productions will be {S->a, S->aS }
>
> I'm sure there is a lesson there. You proposed a specification. It was,
> from a purely formal standpoint, a fine specification. But *thinking*
> about how I would represent it in the form of tests was enough for me to
> spot a flaw.

Yes. You have analyzed the specification. You'd be unable to do it, if
there were none.

>>> ACCEPT: a, aa, aaa, aaaa, aaaaa
>>> REJECT: b, c, z, ab, aaab
>>>
>>> Can you infer the specification from that ? I'm sure you can.
>>
>> How could I? Without some additional knowledge about the language we
>> cannot.
>
> What additional knowledge do you need ?

The context you mention below.

> It's perfectly clear that the tests point to a language that accepts
> strings of a's of any length, and rejects everything else. (It's clear
> because of the context of our discussion, not entirely because of the
> test set. So what: every software development effort occurs in a context
> which narrows down the possibilities for interpretation.)

Then we are in agreement, because specifications is a way to define the
context.

>> [ BTW, IQ tests always puzzle me: 1, 234, 5, 678 what is the next
>> number of the sequence? The right answer is ANY! (:-))]
>
> Oh. How would you defend "1066" as the right answer ?
>
> The problem may be that you're thinking of "right/wrong" as a binary
> distinction. That is fine in domains where program correctness is a
> binary matter. In other domains, correctness may be a matter of degree.
> For instance, we should be able to rank the following answers by order
> of "reasonableness":
>
> 1, 234, 5, 678, ...
> 1
> 234
> 91011
> 9
> 9AB

The problem is not the target set. If you replace Boolean with Rank you
gain nothing by doing that. Because the problem is in the mapping
answer->something, which is still undefined. It is the context, which
defines that mapping. Take the context "what people would think about it",
and the mapping will be some random fuzzy variable of preferences
distributed in some particular way.

[ BTW, human given ranks are unordered, there were studies concerning that.
In general it is always a good idea to avoid referring to "common sense"
where possible. Because human brain is a system, much more complex to
describe, predict, etc than arithmetic. It is simpler to define Pi, than to
ask a crowd to write it down and then to average the answers... (:-))]

>> Actually, you have pinned the point. To test or define something we have to
>> know what we are doing. That means ANALYSIS (capitalized.) When Robert
>> tells us about tests as specifications he cheats. Because he already did
>> that analysis, got the specifications in his head, constrained the design
>> to the limits where anything produced were testable.
>
> I'm not sure what you mean. "Cheats" ? We're not playing a game; we want
> an effective approach to engineer solutions. We don't have to be
> "sporting" about it, and make it any more difficult, or aesthetically
> pleasing, than it needs to be.

Shouldn't an engineer solution be predictable?

> I'm vaguely hearing a concern: that an approach based on tests will
> unduly mix "definition of the problem" and "assumptions regarding the
> solution", what we call analysis and design. Am I interpreting
> correctly?

It is too specific. Actually to approach it, some very preliminary
questions are necessary:

1. What is the problem?
2. What is the solution space?
3. Which solutions can be identified by a conceivable number of tests?

My concern is that: when 1 is rich, 3 is incomplete. When 3 is complete, 2
is insufficient.

And, even this is too hasty, if we'd seriously try to consider
specifications = tests. But I have an impression that you don't insist on
it.

>> For the language above, let's make a parser that recognizes only one letter.
>> Then only one test will be needed. It is cheating: because one not-tests
>> specification was replaced with another, again not-tests (= parser having
>> this property.)]
>
> But neither Bob nor I are proposing that only one test is needed for the
> trivial language above. The test set I proposed include ten tests. So,
> whatever my approach consists of, it's not quite what you are arguing
> against here.

The point is that one test is no more radical than any finite number of
tests. As you said, it depends on the context. In the context of "any chain
of the same character", one test is enough. Your ten tests also require a
context.

> You'll agree with me that any implementation which *fails* one of my ten
> tests is not a correct implementation of what I have informally
> specified as "strings of a's of any length". Yes ?

Sure.

> Now the question is, what can we believe about an implementation that
> *passes* all ten tests ? Would you agree that a belief that "The
> implementation is correct if it passes all ten tests", while not a
> hundred percent secure, is at least satisficing, i.e. "good enough"?

See above. This assumes something about the way human beings are reasoning
as the context. [ Do we really know? ] Considering my example with 1, 234,
5, 678, the answer you probably hold for evident (9), might puzzle an
intelligent Babylonian, not accustomed to decimal numbers.

> (I'm asking if you agree in principle, not necessarily about this
> particular set of ten tests; we might quibble about issues like
> implementations with an arbitrary limit on the size of the string,
> requiring a "really long string" test. That doesn't change the
> argument.)

This supports *my* argument. An important part of the specification is in
fact the context you are arguing to ("good enough" etc.) You cannot propose
tests to define what people hold for "good enough", right? This is the
cheating I meant.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de
.

Relevant Pages

  • Re: SQL
    ... Just a naked string. ... also that in this case it is not the context which determines the types, ... I.e. it is dynamic typing, ... PHP is an example of a dynamically-typed language (and frankly it is ...
    (comp.object)
  • Re: Test first as specification
    ... >> the only valid input string is an infinite string of a's? ... Without some additional knowledge about the language we ... correctness may be a matter of degree. ... The test set I proposed include ten tests. ...
    (comp.object)
  • Re: Static vs. Dynamic typing (big advantage or not)---WAS: c.programming: OOP and memory management
    ... >>of some specific type (int, float, String, Java.stuff) and use ... > when the type is used out of context. ... > 2) the type usage/relationship is valid for a sizable part of the code. ... If you're right then a language such as C is nearly perfect, ...
    (comp.programming)
  • Re: Static vs. Dynamic typing (big advantage or not)---WAS: c.programming: OOP and memory management
    ... >>of some specific type (int, float, String, Java.stuff) and use ... > when the type is used out of context. ... > 2) the type usage/relationship is valid for a sizable part of the code. ... If you're right then a language such as C is nearly perfect, ...
    (comp.object)
  • Re: Operator overloading in C
    ... All development of C as an independent language has ... making any changes or improvements to the standard ... The lack of a counted string data structure, ... Pointers can't be used for arg1 or arg2. ...
    (comp.std.c)