Re: How do you make decisions that optimize software quality?

From: Juhan Leemet (juhan_at_logicognosis.com)
Date: 08/14/04

  • Next message: Roberto Nunnari: "Nunni FSM Generator v. 1.1 released!" 
    Date: Fri, 13 Aug 2004 23:19:05 -0200

    On Thu, 12 Aug 2004 15:41:14 -0400, John Roth wrote:
    > "Juhan Leemet" <juhan@logicognosis.com> wrote in message
    > news:pan.2004.08.12.18.20.57.825960@logicognosis.com...
    >> On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote:
    >> > "Andrew" <analystresearch2002@yahoo.com> wrote in message
    >> > news:43cf64b0.0408111141.4cc1a6dc@posting.google.com...
    >> > 2. Every line of code the developers write must be integrated that
    >> > night. The integration and build system must run all of the passing
    >> > acceptance tests to date, and all of the unit tests must run at
    >> > 100%...

    >> That sounds pretty arbitrary to me: esp. "every night"...
    >
    > Actually, any period will not do. There is a growing body of evidence
    > that the amount of trouble you have in integration is super-linear in
    > the amount of time code has stayed out without being integrated. In
    > other words, "integration Hell" is an easily predictable consequence of
    > leaving integration until the end of the project.

    OK, I mis-spoke. It's obviously not "any period" because the bean-counters
    would then have us only do one build at the very end = cheapest. OTOH too
    frequent can burn up resources running on a treadmill without much result.

    Maybe I'm thinking of larger teams which have to be structured in groups.
    In that case, I would prefer to have better defined deliverables,
    periodically submitted into the system build. I don't believe that every
    design (esp. redesign) can be completed within a day. Is that always true?
    Personally, I think for a schedule that runs out years, one could have
    full system integrations done, say, weekly. The groups themselves might do
    their own "integrations" against a stable baseline (working system) more
    frequently.

    As I write this, I realize that there is no point in continuing this. If
    you're talking "XP speak" and I'm thinking concepts and terminology over
    some decades, we're probably not communicating well at all.

    > On the other side, daily is "white book" XP...
    > The experiance in XP shops is that they almost never have integration
    > problems, even though everyone is allowed (indeed encouraged) to change
    > anything, anywhere in the code base that they need to for the current
    > story.

    I guess I'm uncomfortable with the notion that "everyone is encouraged to
    change anything". Sounds too much like "programming with your spurs on". I
    happen to think there should be some thought given to interfaces and they
    should be protected "somewhat". If they can be defined (best) then they
    should not be changed arbitrarily, but only after consultation and
    thought. I like the idea that everyone is "deputized" to be on the lookout
    for quality issues. None of that "not my problem" kind of attitude.
    Everyone has to be engaged to deliver good product in a timely manner.

    >> "what is the remedy?"... Probably a "jeopardy alert" to the PM...
    >
    > If you're doing daily (or more frequent) integration / builds, then "it
    > doesn't work" isn't a crisis - except for the dummy that broke the
    > build. (The Microsoft principle - one of the few real crimes that their
    > developers can commit is to break the build.)

    That makes me really uneasy. Having worked with some M$ documentation on
    their development tools and finding that they were only very loosely
    cohesive (and incredibly mutable, between revisions, see the OS/2 SDKs).

    You're talking out of both sides of your mouth here: on the one hand
    "doesn't work isn't a crisis", but at the same time "real crime is
    breaking the build". Which is it? One could go either way. I've worked
    with "bully boy" management that "kills the messenger" (the M$ way?) and
    that has literally turned out to be a bankrupt policy!

    If the schedule depended on some functionality being available on a
    certain date, and the build is busted, then obviously that functionality
    is NOT available, and that deadline has been missed. Now whether that is
    the "real deadling" or not... ask the executives and bean counters?

    > As I said up front in my original response, I'm answering this question
    > from an XP perspective. I'm not trying to be even-handed since I firmly
    > believe that "standard software engineering" methodologies have
    > demonstrable problems, both in effectiveness and efficiency.

    I haven't worked with XP (that I know? maybe some similar principles?). I
    suppose I should investigate it to some depth. I get a little put off by
    the "golly gee" kind of evangelism that I seem to hear. OTOH, if those
    techniques are good and they really work well, then we should all use them.

    Personally, I believe a lot of the problems of "standard software
    engineering" methodologies are more related to their implementations. I
    remember it related that Gane & Sarson did 2 pilot projects of their
    (first iteration?) SSADM. One was a success (those that grasped the
    concepts and their purpose) and one was a total failure ("hostile" group?
    just went through the motions, and generated useless "paperwork"). I'll
    bet there are some duds out there that can make a total hash of XP, too.

    >> Also, a general comment on "bugs" and "issues". Some organizations
    >> and/or customers are shocked at the numbers of bugs that are actually
    >> discovered and/or recorded... injection rate for software development
    >> was something like 50 errors per KLOC...

    If you never look at the bugs, it doesn't mean they are not being
    generated. How many are really being fixed? I'd prefer to have some idea
    of how many there are: say starting from a full system integration, not
    the itty bitty ones that arise from the low level "fizz" of programming
    and subsystem integration. How does XP deal with bugs/problems?

    > See the comment in my original post. It's demonstrably possible to get
    > an almost defect-free product because some teams are actually doing it.
    > Customer visible defects are *expensive*! One of the rallying cries in
    > manufacturing in the last several decades has been "Quality is free",
    > meaning that increasing your quality has paybacks in terms of support
    > and warrenty service, etc., that more than pay for the effort.

    I've worked with a small team that delivered near perfect (3 "service
    calls": 1 RTFM, 1 modification feasibilty question, and only 1 spelling
    mistake bug, working at 400% load for years). The travesty was that
    "management" did not recognize what they had and "refused" to reward this
    form of behaviour. The project was canned. I believe the IP could have
    been sold outright for something like $200K but that seemed to be too much
    trouble?!? Others since have commented (incredulously) "that was too
    good", perhaps hinting that they would have tried to screw our budget?

    The point being that I think I understand what it takes. I've also worked
    with teams that have recorded 1400 and 11000 bugs/issues during
    integration. I can't remember how many of those "leaked out" into the
    customer site/environment, maybe a third? half? There was a lot of yelling
    and screaming. These same groups refused to believe there would be bugs.

    >> I have found it useful to "manage expectations" of (naive?) project and
    >> corporate management to mention that they should expect some at least
    >> 5/KLOC bugs/issues. (No way! is the usual response)...
    >
    > There is a naive approach that says: we can do anything we want, and
    > it'll work out in the end. That demonstrably leads to crap.

    I get the impression that the XP process does not record any problems.
    Must start at some point? How do you know that you're not "going round in
    circles"? Are "busted build" problems recorded? Only customer bugs?

    >> Like that old joke:
    >> "You want good, cheap, fast? Pick any 2 and call me back!"
    >
    > The only way you get quality is to pay thorough, detailed attention to
    > quality in every aspect of the process...

    When was that not true? I would suggest that those people who couldn't
    make "waterfall" methodology work were just doing it really badly. Now
    maybe XP is better (but not quite the same thing as a methodology)?
    Whatever process you choose, you have to understand the purpose of every
    thing you do and all "deliverables", otherwise you're just going through
    the motions, and you'll get nowhere (good). The earliest proponents that I
    met that wanted to do "spiral development" seemed to suggest that they
    would just spin their wheels and we should be happy with whatever they
    happen to produce. Bloody hell! Not on my dime. Call your shots.

    > The notion that you will inevitably produce a defective product is one
    > of the most pernicious beliefs in the industry today. It gets repeated
    > ad nausium by the pundits...

    I hope you were not interpreting my comments as an example? I think it is
    important to understand what is going on in the process. If humans inject
    some 50 bugs per KLOC then you need some methods and processes to take
    them all out of there. I was reporting some "wishful thinking" kind of
    "planning" that imagined "how bad could it be?" some 100 bugs? No sweat!
    In actual fact, there could be tons of bugs. If you are not paying
    proper attention to quality throughout your process, and you're not
    finding and fixing those bugs, you'll get "buried" (in ***?).

    I have a feeling that we're probably in "violent agreement", but maybe not
    totally viz. methods. I don't believe "one size fits all": XP is the
    "silver bullet"? I'm mindful of Brooks and others who warn against any
    complacency, to think that you can find one thing that'll do it all.

    >> Sometimes organizations set themselves up for failure by arbitrarily
    >> and "willfully" driving the software development process out of its
    >> "compliance limits" (i.e. until something breaks). There's no free
    >> lunch.
    >
    > I'm not sure what you're saying here, other than that every process has
    > known limitations. You cannot arbitrarilly specify scope, time, budget
    > and quality, and expect that, by some miracle, it will be met. It
    > doesn't work in building bridges, and it doesn't work in building
    > software. That doesn't stop MBA trained managers from trying.

    Again we're agreed. The "bully boy" executive mentality thinks that
    software is infinitely mutable and therefore can be beaten out of people.

    Can you suggest a good overview of XP, to put it into perspective for me?
    I'm not sure I have time to read several books and do some projects before
    I can make up my mind about it. I'm a little bit uneasy. Seems too facile. 

    -- 
Juhan Leemet
Logicognosis, Inc.
  • Next message: Roberto Nunnari: "Nunni FSM Generator v. 1.1 released!"