Re: How do you make decisions that optimize software quality?

From: Juhan Leemet (juhan_at_logicognosis.com)
Date: 08/12/04

  • Next message: Krishna Tyner: "Big Sun News from LinuxWorld 2004" 
    Date: Thu, 12 Aug 2004 16:20:59 -0200

    On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote:
    > "Andrew" <analystresearch2002@yahoo.com> wrote in message
    > news:43cf64b0.0408111141.4cc1a6dc@posting.google.com...
    >> ...mandate that you must deliver software at the absolute lowest
    >> possible total cost of development and deployment...

    Somewhat open ended and demanding question?

    > There are several basic processes needed.
    >
    > 1. Executable (meaning repeatable) acceptance tests. These tests must be
    > accessable to non-technical people on the customer side...

    I would go further and say that these tests should be "defined" (perhaps
    with assistance) by the customer and/or his representatives. In some/many
    contractual arrangments that is mandated. Always a good idea.

    > 2. Every line of code the developers write must be integrated that
    > night. The integration and build system must run all of the passing
    > acceptance tests to date, and all of the unit tests must run at 100%.
    > The build must finish with a properly packaged deployable. The build
    > system may run analysis code like code coverage metrics; if so, unit
    > test statement and branch coverage must be at 100%, and acceptance tests
    > coverage and branch metrics must be fairly high.

    That sounds pretty arbitrary to me: esp. "every night". That's the popular
    line, but any period would do, depending on the shop and the process, etc.

    viz. the requirements of the built system: as a lawyer used to tell me
    "what is the remedy?" Clearly it should be the goal of a built system
    to pass all/most tests, etc., but suppose it does not work? (at all?) What
    do you do about it? Probably a "jeopardy alert" to the project management
    viz. schedule and cost overruns. What else? What can you do, but "reject"
    the build contents and redo the build and test at a later time.

    > 3. QA must be done statistically...

    This will probably be a quality measured "after the fact"? Naively one
    might think one can set a target like 99.9% (or some such?). A mature shop
    with an establised and "calibrated" process might be able to "call their
    shots" like that. More likely they know what their average process results
    are, and they won't stray significantly. Isn't this related to CMM stuff?
    If you're starting out with a new group/process it's going to be tough.
    Maybe set a goal of no severe bugs found within a particulat time interval?

    > The interesting thing about this list is that it's not completely
    > specific to XP. XP mandates...

    I would hesitate to presuppose any implementation method or methodology,
    while trying to set objectives and targets. If they are chosen after goals
    are set, and they achieve the goals, then that's great!

    Also, a general comment on "bugs" and "issues". Some organizations and/or
    customers are shocked at the numbers of bugs that are actually discovered
    and/or recorded. I would suggest that even pretty good shops have some few
    per KLOC (not popular metric, but that IS what people write!). Boehm
    suggested IIRC that the measured injection rate for software development
    was something like 50 errors per KLOC? Good developers and integrators
    get most of them out, but some remain. That was some 20 years ago, but
    like the saying goes "times may have changed, but people haven't". Does
    anyone have any good numbers, from more recent studies? Wooly anyway?

    I have found it useful to "manage expectations" of (naive?) project and
    corporate management to mention that they should expect some at least
    5/KLOC bugs/issues. (No way! is the usual response) If you don't have
    tools and processes in place to manage that number (i.e. 1MLOC -> 5000+
    "bugs" or "issues") you will get buried! Dunno what an average cost is,
    but for smallish projects I have used about $500/bug for budgeting in the
    past (i.e. 1MLOC -> $2.5M+ for bug/issue fixing). If you're not planning
    or setting aside something similar (or more), there are likely going to be
    some bitter tears (yours and/or customers)!

    Like that old joke:
    "You want good, cheap, fast? Pick any 2 and call me back!"

    Sometimes organizations set themselves up for failure by arbitrarily and
    "willfully" driving the software development process out of its
    "compliance limits" (i.e. until something breaks). There's no free lunch. 

    -- 
Juhan Leemet
Logicognosis, Inc.
  • Next message: Krishna Tyner: "Big Sun News from LinuxWorld 2004"

    Relevant Pages

    • Re: How do you make decisions that optimize software quality?
      ... >> ...mandate that you must deliver software at the absolute lowest ... >> possible total cost of development and deployment... ... Clearly it should be the goal of a built system ... If they are chosen after goals ...
      (comp.object)
    • Re: Data driven people arguments
      ... you want to achieve is to minimize the cost of some future change. ... colleague is concerned with the cost of redeploying and the cost ... possiblity of redeploying with a 5 minutes downtime (because it gives ... You'll probably find that you have lots of common goals, ...
      (comp.object)
    • Re: Days not weeks
      ... A programs success is determined by whether it meets its goals. ... is not a goal then cost is not a success criteria. ...
      (rec.sport.golf)
    • Re: Days not weeks
      ... A programs success is determined by whether it meets its goals. ... is not a goal then cost is not a success criteria. ...
      (rec.sport.golf)
    • Re: FAA ADS-B Propaganda Video
      ... Are you referring to the mandated ADS-B-out, or the Garmin 796 ... Yes, there's a cost. ... This NPRM only will mandate ADS-B Out. ... That would be like FORCING you to pay $10,000 to same me some ...
      (rec.aviation.piloting)