Re: Migrating a system incrementally. Was: Re: Static vs. Dynamic typing (big advantage or not)

From: Nick Landsberg (SPAMhukolauTRAP_at_SPAMworldnetTRAP.att.net)
Date: 07/20/04 

Date: Tue, 20 Jul 2004 16:12:04 GMT

Phlip wrote:

> Nick Landsberg wrote:
>
>
>>Our customers are large (sometimes huge)
>>corporations. They depend on some of the
>>systems we provide to, among other things,
>>keep their business running and maintain
>>cash flow. (Example: a system which debits
>>your prepaid account when you make a cellular
>>phone call. Monetary losses if such a system
>>is unavailable for an /hour/ have been estimated
>>in the hundreds of thousands of dollars.)
>>
>>These same /customers/ insist on *fixed* delivery
>>cycles, e.g. a major delivery every 6 months,
>
>
> We discuss two separate things here - release and deploy. One means the
> programmers firm up a version and push it out of the lab. The other means
> someone puts it on its users' desktops.

Yes, we are talking two seperate things here, and we're
probably talking past each other. We have no "desktop".
(Other than for administrative accesses which are
less than 0.001% of our load.)
All accesses to and from this system are by other
systems or by the phone network.

I make no claim to understanding the desktop environment.
I was just describing our environment.

Note: In my experience, most systems are built to
either:
a) increase productivity (i.e. cut costs)
b) increase revenue (which includes "prevent revenue loss")
c) increase safety (e.g. E911, railroad signal monitoring,
air traffic control, etc.)
d) other (including such things as complex mathematical
computations, aerodynamic simulations, etc.)

(But my experience has not been comprehensive, so I
welcome others to add categories.)

The majority of what most folks are familiar with is in
category (a). I happen to come from a background of
mostly category (b) and a smattering of (c). Therefor
my views are slanted by my experiences. As yours
are slanted by your experiences (which, I believe
from your posts, are mostly in category (a) ).

>
> The ability to release frequently is priceless. Each release should be
> capable of demonstrating a user productivity boost.

No desktop, therefore no user productivity boost. Rather an
enhanced revenue stream for our customer. We're addressing
a different set of issues, you and I. Talking past each
other again.

>
> The ability to deploy frequently is also priceless, but we can do without
> it. The goal is to make deployment entirely a business-side decision. They
> should never endure a blackout before becoming empower to release the most
> recent new features.
>
> An "Agile" process can:
>
> - accept feature requests in any order
> - release any integration to Quality Control and beyond
> - minimize the time between specifying a feature and using it.
>
> That doesn't say "does" it says "can". The ability to do those things is
> more important than doing them. Most feature requests are in a predictable
> order - don't scramble them. Agile teams release once a week, but they
> _could_ release any integration. Your lab's Quality Control then rates
> whether to release to your customer's lab's Quality Control.

Customer acceptance tests specify at least a 72 hour
stability run at full load (say 500-1000 transactions per
second for round numbers). In some cases they required
a 14 day stability run. Thus our QC folks also do
a 72 hour automated stability run.

Since our lab cannot possibly afford to buy
all the equipment which interfaces
to our box. A major part of the work is to write
simulators which mimic the rest of the customer
network. Again, no "desktop" is where our
worlds differ.

>
>
>>and an update every 3 months. After delivery,
>>they will *not* deploy it immediately, but
>>install it in their lab and typically
>>run acceptance tests and stability tests
>>for at least a month. Then, they go through
>>the pain of slowly converting the 10 million
>>record customer database from the old schema
>>to the new schema at a rate of about two million
>>customers a week.
>
>
> If you can't automate doing all that in your lab, you have not done your
> job. You have sloughed this work off on them.

No, we automate all this stuff in our lab. And test it several
times. The customers still specifically insist on testing
the conversion process, testing the backout process
and running their own acceptance tests and stability
tests. Actually, automating all this conversion stuff
takes takes more work than actually doing the feature work,
but we're getting better at it as the years pass.
We're down to <5 console commands to do it where
many years ago it took ~50. (The customers were
unhappy with 50 and rigthly so.)

>
> The ability to release frequently is priceless, so you must force down its
> cost - on principle. _Not_ so you can force users to upgrade every week.
>
> Your application's release process, once a week, should demonstrate the
> ability to upgrade its database, then run the kind of version comparison
> tests you describe.
>
> You are up against the common and feared "Database Refactoring Speedbump".
> Because databases, out of the box, cannot reliably refactor, nobody writes
> scripts that ensure each refactor in the lab brings all its production data
> along with it.
>
> I have not studied this, but I suspect teams put each refactor into a queue,
> and run the queue at deployment time, incrementally migrating the data over.

Incremental refactoring of a database schema runs into a major
"speed bump" when you have several 100 GB to convert from old
schema to new schema. While individual developers may play with
the new schema in their toy databases, unloading the old
database to flat-files, loading in a new schema, reloading the
data (including the necessary code to re-arrange the files
to be consistent with the new schema), also takes significant
time (on the order of hours, at least, for the master database).

Backout from an ill-advised change is much more than hitting "undo."
A script, possibly taking several hours to run, would
have to be written for backout.

If there are multiple "incremental" schema changes, each of
which took several hours, then our customers would not
accept that. They would want a comprehensive automated
script which did all the changes in one swell foop.
Therefore, yet another script would have to be written
to incorporate all the schema changes between "initial"
state and "final" state (and the necessary backout
script too).

I'm concerned about the scaling factor and whether the
time spent in the incremental conversions may negate the
benefits of Agile methods in this case.

As I said, we live in different worlds, and my world
may be the exception which proves the rule.

>
> I am _not_ saying that writing these scripts will magically give you the
> ability to deploy weekly. That's a nice goal. Your crew should automate
> database refactors to make your own process more agile, and give more
> control to your business side.

See above.

>
>
>>During this time, they
>>have *both* the old and new systems running.
>>If a major bug is found during any part of this
>>process, they revert to the old system,
>>and back-out any customers they have already
>>converted to the new system. Any *bug fix* delivery
>>causes the cycle to start all over again.
>>After 10 weeks or so, they are just about
>>ready for the 3 month update, and the cycle
>>starts all over again anyway.
>
>
> All that stuff is your application's responsibility. The more your own team
> and your own tests support these behaviors and automated

See above. The time it takes to develop and execute
the automation code is greater than the time it takes
to develop the feature code.

I'm not saying that Agile methods are wrong. I'm saying
that someone should investigate and study how they
would be applicable to this environment and all the
second and third order effects this environment
imposes on the development process.

>
> I'm not saying you don't - I'm saying the division of labor inhibits
> feedback, at some point, from the deployment process back into the source
> code. If you could immediately tell, in the lab, that a given change
> impaired the deployment process, you could adjust faster.
>
> A car has breaks to stop, enabling it to go fast. With more of this feedback
> occuring inside your lab, you could go faster.
>
>
>>That's my environment.
>>
>>Most folks, it seems from this thread, do
>>not work in such an environment and this
>>is probably an extreme case. But that's
>>one example of where the customer does
>>*not* want to see "continuous progress"
>>(unless you count once every 3 months
>>as "continuous").
>
>
> The "Agile" literature has been unanimous about "Frequent Releases". Think
> of cpan.org - the world's biggest backup server. Long lists of tiny ticks on
> various modules. >That's< agile.
> 

-- 
"It is impossible to make anything foolproof
because fools are so ingenious"
  - A. Bloch

Relevant Pages

  • Re: same application on multiple schemas
    ... The application will be instantiated for multiple users. ... application must have its own "database", or schema, Oracle speaking. ... providing and appplication for multiple customers on a single database. ... And suppose one of those customers used the system far more heavily than ...
    (comp.databases.oracle.server)
  • Re: same application on multiple schemas
    ... The application will be instantiated for multiple users. ... application must have its own "database", or schema, Oracle speaking. ... providing and appplication for multiple customers on a single database. ... And suppose one of those customers used the system far more heavily than ...
    (comp.databases.oracle.server)
  • Re: Migrating a system incrementally. Was: Re: Static vs. Dynamic typing (big advantage or not)
    ... >>Our customers are large ... >>to the new schema at a rate of about two million ... we automate all this stuff in our lab. ... > ability to upgrade its database, then run the kind of version comparison ...
    (comp.programming)
  • Re: same application on multiple schemas
    ... application must have its own "database", or schema, Oracle speaking. ... Suppose you were providing and appplication for multiple customers on a single database. ...
    (comp.databases.oracle.server)
  • Re: Insert Stored Procedure where the table is provided as input argument
    ... web app so the utility will be just dealing with tables management. ... Surely you would NEVER allow end-users to change the schema. ... will never do business with them again. ... That is among the customers ...
    (microsoft.public.sqlserver.programming)