Re: How do you make decisions that optimize software quality?

From: Juhan Leemet (juhan_at_logicognosis.com)
Date: 08/12/04 

Date: Thu, 12 Aug 2004 16:20:59 -0200

On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote:
> "Andrew" <analystresearch2002@yahoo.com> wrote in message
> news:43cf64b0.0408111141.4cc1a6dc@posting.google.com...
>> ...mandate that you must deliver software at the absolute lowest
>> possible total cost of development and deployment...

Somewhat open ended and demanding question?

> There are several basic processes needed.
>
> 1. Executable (meaning repeatable) acceptance tests. These tests must be
> accessable to non-technical people on the customer side...

I would go further and say that these tests should be "defined" (perhaps
with assistance) by the customer and/or his representatives. In some/many
contractual arrangments that is mandated. Always a good idea.

> 2. Every line of code the developers write must be integrated that
> night. The integration and build system must run all of the passing
> acceptance tests to date, and all of the unit tests must run at 100%.
> The build must finish with a properly packaged deployable. The build
> system may run analysis code like code coverage metrics; if so, unit
> test statement and branch coverage must be at 100%, and acceptance tests
> coverage and branch metrics must be fairly high.

That sounds pretty arbitrary to me: esp. "every night". That's the popular
line, but any period would do, depending on the shop and the process, etc.

viz. the requirements of the built system: as a lawyer used to tell me
"what is the remedy?" Clearly it should be the goal of a built system
to pass all/most tests, etc., but suppose it does not work? (at all?) What
do you do about it? Probably a "jeopardy alert" to the project management
viz. schedule and cost overruns. What else? What can you do, but "reject"
the build contents and redo the build and test at a later time.

> 3. QA must be done statistically...

This will probably be a quality measured "after the fact"? Naively one
might think one can set a target like 99.9% (or some such?). A mature shop
with an establised and "calibrated" process might be able to "call their
shots" like that. More likely they know what their average process results
are, and they won't stray significantly. Isn't this related to CMM stuff?
If you're starting out with a new group/process it's going to be tough.
Maybe set a goal of no severe bugs found within a particulat time interval?

> The interesting thing about this list is that it's not completely
> specific to XP. XP mandates...

I would hesitate to presuppose any implementation method or methodology,
while trying to set objectives and targets. If they are chosen after goals
are set, and they achieve the goals, then that's great!

Also, a general comment on "bugs" and "issues". Some organizations and/or
customers are shocked at the numbers of bugs that are actually discovered
and/or recorded. I would suggest that even pretty good shops have some few
per KLOC (not popular metric, but that IS what people write!). Boehm
suggested IIRC that the measured injection rate for software development
was something like 50 errors per KLOC? Good developers and integrators
get most of them out, but some remain. That was some 20 years ago, but
like the saying goes "times may have changed, but people haven't". Does
anyone have any good numbers, from more recent studies? Wooly anyway?

I have found it useful to "manage expectations" of (naive?) project and
corporate management to mention that they should expect some at least
5/KLOC bugs/issues. (No way! is the usual response) If you don't have
tools and processes in place to manage that number (i.e. 1MLOC -> 5000+
"bugs" or "issues") you will get buried! Dunno what an average cost is,
but for smallish projects I have used about $500/bug for budgeting in the
past (i.e. 1MLOC -> $2.5M+ for bug/issue fixing). If you're not planning
or setting aside something similar (or more), there are likely going to be
some bitter tears (yours and/or customers)!

Like that old joke:
"You want good, cheap, fast? Pick any 2 and call me back!"

Sometimes organizations set themselves up for failure by arbitrarily and
"willfully" driving the software development process out of its
"compliance limits" (i.e. until something breaks). There's no free lunch. 

-- 
Juhan Leemet
Logicognosis, Inc.

Relevant Pages

  • Re: How do you make decisions that optimize software quality?
    ... >> ...mandate that you must deliver software at the absolute lowest ... >> possible total cost of development and deployment... ... Clearly it should be the goal of a built system ... If they are chosen after goals ...
    (comp.lang.java.softwaretools)
  • Re: Data driven people arguments
    ... you want to achieve is to minimize the cost of some future change. ... colleague is concerned with the cost of redeploying and the cost ... possiblity of redeploying with a 5 minutes downtime (because it gives ... You'll probably find that you have lots of common goals, ...
    (comp.object)
  • Re: Agile developement ... more than just extreme programming ???
    ... >> will have different cost outcomes depending on the order. ... > Achieving the other two goals enables this goal. ... Shayne Wissler ...
    (comp.object)
  • Re: FAA ADS-B Propaganda Video
    ... Are you referring to the mandated ADS-B-out, or the Garmin 796 ... Yes, there's a cost. ... This NPRM only will mandate ADS-B Out. ... That would be like FORCING you to pay $10,000 to same me some ...
    (rec.aviation.piloting)
  • Re: Air Force Signs Off on SRB-CEV
    ... >>>VSE has the goals it has. ... lunar missions planned for year by the annual cost of the standing ... I think "affordable" means what the public is willing to pay for it, ... But then, so was the Shuttle. ...
    (sci.space.policy)