Re: How do you make decisions that optimize software quality?

From: John Roth (newsgroups_at_jhrothjr.com)
Date: 08/14/04 

Date: Sat, 14 Aug 2004 10:11:10 -0400

"Juhan Leemet" <juhan@logicognosis.com> wrote in message
news:pan.2004.08.14.01.19.02.689313@logicognosis.com...
> On Thu, 12 Aug 2004 15:41:14 -0400, John Roth wrote:
> > "Juhan Leemet" <juhan@logicognosis.com> wrote in message
> > news:pan.2004.08.12.18.20.57.825960@logicognosis.com...
> >> On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote:
> >> > "Andrew" <analystresearch2002@yahoo.com> wrote in message
> >> > news:43cf64b0.0408111141.4cc1a6dc@posting.google.com...
> >> > 2. Every line of code the developers write must be integrated that
> >> > night. The integration and build system must run all of the passing
> >> > acceptance tests to date, and all of the unit tests must run at
> >> > 100%...
>
> >> That sounds pretty arbitrary to me: esp. "every night"...
> >
> > Actually, any period will not do. There is a growing body of evidence
> > that the amount of trouble you have in integration is super-linear in
> > the amount of time code has stayed out without being integrated. In
> > other words, "integration Hell" is an easily predictable consequence of
> > leaving integration until the end of the project.
>
> OK, I mis-spoke. It's obviously not "any period" because the bean-counters
> would then have us only do one build at the very end = cheapest. OTOH too
> frequent can burn up resources running on a treadmill without much result.

One integration at the end generally leads to "integration hell."
The fact is that you shouldn't even think of optimizing parts of
a process until the entire process is working to your satisfaction.
The bean counter that comes in and makes changes to pieces
of the process without looking at the entire process is a menace,
and should be fired promptly.

Well, current XP practice is to integrate every twenty to thirty
minutes! What can I say - it works and if you tend it properly
it doesn't use that much resource. Remember that this is in
an environment where you're probably running several hundred
unit tests after *every* edit, and you've spent a significant amount
of effort to make sure that the collection of unit tests can run in
under a minute. It's not the same process.

> Maybe I'm thinking of larger teams which have to be structured in groups.
> In that case, I would prefer to have better defined deliverables,
> periodically submitted into the system build.

Delayed integration is poison. If you're working on separate
components, you still need to deal with the fact that your
interface will change just as everything else will.

> I don't believe that every
> design (esp. redesign) can be completed within a day. Is that always true?

Major redesigns take time. It takes some skill to fit them into a
flow and still keep delivering value while implementing the redesign.

> Personally, I think for a schedule that runs out years, one could have
> full system integrations done, say, weekly. The groups themselves might do
> their own "integrations" against a stable baseline (working system) more
> frequently.
>
> As I write this, I realize that there is no point in continuing this. If
> you're talking "XP speak" and I'm thinking concepts and terminology over
> some decades, we're probably not communicating well at all.

I think we're communicating quite well. You've realized that I have
a very different model of how effective software development works
than you do; that's the first step to starting to understand the differences
and how that affects details.

> > On the other side, daily is "white book" XP...
> > The experiance in XP shops is that they almost never have integration
> > problems, even though everyone is allowed (indeed encouraged) to change
> > anything, anywhere in the code base that they need to for the current
> > story.
>
> I guess I'm uncomfortable with the notion that "everyone is encouraged to
> change anything". Sounds too much like "programming with your spurs on".

Yeah, it does sound somewhat like "cowboy coding" doesn't it? The
control is that changes have to be done in very small increments - five
lines in an edit before running your test suite is average. And you have
to make that change work before moving on. You can't leave broken
unit tests behind you, and you can't remove existing tests because "I
can't make that work right now." All of those things are symptoms of
cowboy coders, and in a well functioning XP team, they are very offensive.

> I happen to think there should be some thought given to interfaces and
they
> should be protected "somewhat". If they can be defined (best) then they
> should not be changed arbitrarily, but only after consultation and
> thought.

Programming without thinking about design is a disaster. So is
programming without consultation - that's why pair programming
is a core practice. If you don't know an area, pair with someone
who does.

> I like the idea that everyone is "deputized" to be on the lookout
> for quality issues. None of that "not my problem" kind of attitude.
> Everyone has to be engaged to deliver good product in a timely manner.

Exactly. Can't be emphasized too strongly.

> >> "what is the remedy?"... Probably a "jeopardy alert" to the PM...
> >
> > If you're doing daily (or more frequent) integration / builds, then "it
> > doesn't work" isn't a crisis - except for the dummy that broke the
> > build. (The Microsoft principle - one of the few real crimes that their
> > developers can commit is to break the build.)
>
> That makes me really uneasy. Having worked with some M$ documentation on
> their development tools and finding that they were only very loosely
> cohesive (and incredibly mutable, between revisions, see the OS/2 SDKs).

Well, I didn't say that the entire M$ process was worthy
of emulation - just that their practice of doing integration,
build, test and package daily was probably the only reason
they manage to get anything out the door.

> You're talking out of both sides of your mouth here: on the one hand
> "doesn't work isn't a crisis", but at the same time "real crime is
> breaking the build". Which is it? One could go either way. I've worked
> with "bully boy" management that "kills the messenger" (the M$ way?) and
> that has literally turned out to be a bankrupt policy!

Bad management is bad management. What more can be said,
unless you want to categorize the ways that management can
be bad.

If you're doing frequent integrations, the failure of a single integration
isn't a crisis because it can be fixed quickly. There isn't that much that
went into it, so tracking down the problem is usually quite easy. The
less frequent the integrations, the more goes into them, and the
more problems surface. Since there were more changes, each of
the problems is correspondingly harder to track down. In other
words, the difficulty in integration is proportional to the square
of the time between integrations (measured in terms of changed
(new, deleted) loc.

> If the schedule depended on some functionality being available on a
> certain date, and the build is busted, then obviously that functionality
> is NOT available, and that deadline has been missed. Now whether that is
> the "real deadling" or not... ask the executives and bean counters?

That's only true if 1) you're doing "big bang" integration at the
end of the project, or 2) it's the last integration before the target
date. Again, if you're doing very frequent small integrations, it's
not that hard to find and fix it.

Another issue here is 'end of project variance." The size of problems
and the amount of time to fix them is not proportional to the amount
of time remaining - you can have a large problem right at the end.

This is also where the general Agile practice of scheduling the
highest business value functionality first comes into play. If you
run into a problem right before the hard ship date, by definition
it's the least valuable functionality that was scheduled for that
release, so it's a viable choice to fall back to the last complete
build, test and package that actually worked. That is, of course,
a business decision, and it's critically dependent on the output
of each iteration really, truely being production quality without
needing additional after the fact testing and bug fixing.

>
> > As I said up front in my original response, I'm answering this question
> > from an XP perspective. I'm not trying to be even-handed since I firmly
> > believe that "standard software engineering" methodologies have
> > demonstrable problems, both in effectiveness and efficiency.
>
> I haven't worked with XP (that I know? maybe some similar principles?). I
> suppose I should investigate it to some depth. I get a little put off by
> the "golly gee" kind of evangelism that I seem to hear. OTOH, if those
> techniques are good and they really work well, then we should all use
them.

A lot of people seem to be put off by the level of evangalism.
The other problem is that it's different enough from standard
software development methodology that people have a real
hard time getting theirm minds around it at first look.

> Personally, I believe a lot of the problems of "standard software
> engineering" methodologies are more related to their implementations. I
> remember it related that Gane & Sarson did 2 pilot projects of their
> (first iteration?) SSADM. One was a success (those that grasped the
> concepts and their purpose) and one was a total failure ("hostile" group?
> just went through the motions, and generated useless "paperwork"). I'll
> bet there are some duds out there that can make a total hash of XP, too.

Oh, absolutely.

> >> Also, a general comment on "bugs" and "issues". Some organizations
> >> and/or customers are shocked at the numbers of bugs that are actually
> >> discovered and/or recorded... injection rate for software development
> >> was something like 50 errors per KLOC...
>
> If you never look at the bugs, it doesn't mean they are not being
> generated. How many are really being fixed? I'd prefer to have some idea
> of how many there are: say starting from a full system integration, not
> the itty bitty ones that arise from the low level "fizz" of programming
> and subsystem integration. How does XP deal with bugs/problems?

It may surprise you that the "hard" answer is that you shouldn't have
bugs on the level you're describing. That's not the same as don't, of
course. The thing to remember is that an XP project is integrating
continuously, and running the full test suite every night. All unit tests
have to run 100%, and all previously passing acceptance tests have
to continue to pass. If not, fixing it is the first order of business the
next
morning. If you want to be real hard-ass about it, you'll also run
coverage monitors to insure that you have 100% statement and
branch coverage over the unit tests. (This isn't real difficult: TDD
normally delivers statement coverage in the high 90s, and branch
coverage in the mid 90s, so getting to 100% is easy, at least if you
keep at it. It becomes easier over time as the developers learn what
they need to pay attention to). This catches the "cowboy coder" who
might like to remove a unit test because he can't make it pass.

If you can get there (and it isn't easy) then all you've got are
field reported bugs, and you schedule those the same way
you do stories. Having to have a separate bug data base is a
process smell.

> > See the comment in my original post. It's demonstrably possible to get
> > an almost defect-free product because some teams are actually doing it.
> > Customer visible defects are *expensive*! One of the rallying cries in
> > manufacturing in the last several decades has been "Quality is free",
> > meaning that increasing your quality has paybacks in terms of support
> > and warrenty service, etc., that more than pay for the effort.
>
> I've worked with a small team that delivered near perfect (3 "service
> calls": 1 RTFM, 1 modification feasibilty question, and only 1 spelling
> mistake bug, working at 400% load for years). The travesty was that
> "management" did not recognize what they had and "refused" to reward this
> form of behaviour. The project was canned. I believe the IP could have
> been sold outright for something like $200K but that seemed to be too much
> trouble?!? Others since have commented (incredulously) "that was too
> good", perhaps hinting that they would have tried to screw our budget?

Bad management is bad management.

> The point being that I think I understand what it takes. I've also worked
> with teams that have recorded 1400 and 11000 bugs/issues during
> integration. I can't remember how many of those "leaked out" into the
> customer site/environment, maybe a third? half? There was a lot of yelling
> and screaming. These same groups refused to believe there would be bugs.

Yup. As I've mentioned previously, the amount of difficulty you have
in integration is proportional to the square of the number of LOC you're
trying to integrate. I won't go so far as to say you absolutly have to do
continuous integration XP-style, but anyone who designs a project
with "big bang" integration should be sacked.

> >> I have found it useful to "manage expectations" of (naive?) project and
> >> corporate management to mention that they should expect some at least
> >> 5/KLOC bugs/issues. (No way! is the usual response)...
> >
> > There is a naive approach that says: we can do anything we want, and
> > it'll work out in the end. That demonstrably leads to crap.
>
> I get the impression that the XP process does not record any problems.
> Must start at some point? How do you know that you're not "going round in
> circles"? Are "busted build" problems recorded? Only customer bugs?

Well, one of the fundamental XP processes is the executable
acceptance test. This must be in existance before working on
the story (test first!). Teams work to make the acceptance tests
pass (the acceptance test is the requirment). Defects that leak
through that indicate that the acceptance tests are insufficient,
which comes down to how well the requirements analysts are
cooperating with the QA analysts.

XP also believes in appropriate measurements. If you're not
getting a significant number of defects, then process measurements
are a waste of time. If you are, then the process needs to be
inspected with some diligence and reworked until you are
no longer getting defects.

> >> Like that old joke:
> >> "You want good, cheap, fast? Pick any 2 and call me back!"
> >
> > The only way you get quality is to pay thorough, detailed attention to
> > quality in every aspect of the process...
>
> When was that not true? I would suggest that those people who couldn't
> make "waterfall" methodology work were just doing it really badly.

We need to distinguish between a strict phasist waterfall and the way
standard software engineering practice does it. The first is a recognized
prescription for disaster except in very restricted circumstances.

Project managers who can make the second work on a regular basis
have my deepest respect. They are rare, but they do exist.

> Now maybe XP is better (but not quite the same thing as a methodology)?

Well, it's different. It puts a very high value on absolutely minimizing
work in process, all the way from requirements through final packaging.
The less WIP, the less possibility you have for stuff you think is complete
but has lurking problems.

> Whatever process you choose, you have to understand the purpose of every
> thing you do and all "deliverables", otherwise you're just going through
> the motions, and you'll get nowhere (good).

Yep. Some of them aren't exactly obvious in XP, though, which is
a problem.

> The earliest proponents that I
> met that wanted to do "spiral development" seemed to suggest that they
> would just spin their wheels and we should be happy with whatever they
> happen to produce. Bloody hell! Not on my dime. Call your shots.

Yep. While it's possible to deliver less than you promised in XP,
it's the second last resort (the last resort is extending the deadline.)
Either is a symptom of something really wrong with the process
(or a developer suddenly being off sick, of course!)

> > The notion that you will inevitably produce a defective product is one
> > of the most pernicious beliefs in the industry today. It gets repeated
> > ad nausium by the pundits...
>
> I hope you were not interpreting my comments as an example? I think it is
> important to understand what is going on in the process. If humans inject
> some 50 bugs per KLOC then you need some methods and processes to take
> them all out of there. I was reporting some "wishful thinking" kind of
> "planning" that imagined "how bad could it be?" some 100 bugs? No sweat!
> In actual fact, there could be tons of bugs. If you are not paying
> proper attention to quality throughout your process, and you're not
> finding and fixing those bugs, you'll get "buried" (in ***?).

If your process produces defects, that's a comment on your process,
not on the nature of reality. One of the few places where I think
that manufacturing production experiance is applicable to software
development is this precise attitude: defects are unacceptable, and
you need to fix the process until it does not produce them. You
should not need end of the line testing except to certify the process.

I have nothing but sympathy for managers that refuse to believe that
your process is going to produce 50 defects per KLOC. What I
find hard to believe is that they simply don't come out and say that
is absolutely unacceptable, and fix your process so it doesn't do
that.

There was a recent talk at Atlanta SPIN (Dec 2003 meeting)

http://www.atlantaspin.org/Archives.html

by Iraj Hirmanpour from the
S.E.I. He had a couple of real eye-opening slides on the difference
in effectiveness between CMM Level 5 organizations and organizations
that had implemented PSP and TSP in the context of an otherwise
standard software engineering process.

I'd have to go back to the presentation, but I believe that the
difference was around two orders of magnitude fewer defects
for the PSP and TSP shops. And that is from the S.E.I, not from
some wild-eyed off-brand process fanatic.

> I have a feeling that we're probably in "violent agreement", but maybe not
> totally viz. methods. I don't believe "one size fits all": XP is the
> "silver bullet"? I'm mindful of Brooks and others who warn against any
> complacency, to think that you can find one thing that'll do it all.

XP certainly isn't a silver bullet. There is no such thing.

> >> Sometimes organizations set themselves up for failure by arbitrarily
> >> and "willfully" driving the software development process out of its
> >> "compliance limits" (i.e. until something breaks). There's no free
> >> lunch.
> >
> > I'm not sure what you're saying here, other than that every process has
> > known limitations. You cannot arbitrarilly specify scope, time, budget
> > and quality, and expect that, by some miracle, it will be met. It
> > doesn't work in building bridges, and it doesn't work in building
> > software. That doesn't stop MBA trained managers from trying.
>
> Again we're agreed. The "bully boy" executive mentality thinks that
> software is infinitely mutable and therefore can be beaten out of people.
>
> Can you suggest a good overview of XP, to put it into perspective for me?
> I'm not sure I have time to read several books and do some projects before
> I can make up my mind about it. I'm a little bit uneasy. Seems too facile.

Unfortunately, someone else is going to have to do that. I'm one of the
loudmouths on the XP mailing list, and it's really been a while since I
evaluated any of the recent books. The White Book (XP Explained)
is four years old now, and some parts are quite outdated. One book
I'd heartily recommend (although it's not on XP per se) is Lean
Software Development by Mary and Tom Poppendieck. That
gives a lot of process tools without recommending a specific process.

One of the things to watch out for in any XP presentation is that it
is going to take a significant learning curve. Test Driven Development,
emergent design and refactoring are not things that people learn overnight.
You have to find a personal (or team) comfort point between explicit
design and emergent design, and that is going to take time to learn.
You also need to find a balance between depending on the unit
tests and code for design documentation, and where you absolutely
do need some ancilliary documentation.

Executable Acceptance Tests not only take some learning, but there
are specific architectural motifs you have to use for user interface
testing, and that doesn't come overnight either.

John Roth
>
> --
> Juhan Leemet
> Logicognosis, Inc.
>

Loading