Re: Applications using over 2 gigabytes of addressing

From: Harley (dennis.harleyNoSpam_at_worldnet.att.net)
Date: 11/27/03 

Date: Thu, 27 Nov 2003 13:54:15 GMT

Pete,

Not to worry.

The application volume is very low, in mainframe standards.
The program doesn't consume enormous amounts of storage, just more than it
did before.

The main problem with the current system is that the people who wrote it
were VSAM Random Access freaks. The update process is incremental, and they
didn't incorporate adequate backups because it would further degrade the
system.

What I am proposing is a variation of the Extract, Sort, Print model.
Extract the data, pass it through the process, and Update the VSAM files at
the end.
(The VSAM files are not available on-line during the process).
It will run faster, and be easier to test.
As it is now, you need about 8 VSAM files, and they have to be "in-sync", to
test any part of the system flow. Unit testing is a horror.

"Peter E.C. Dashwood" <dashwood@enternet.co.nz> wrote in message
news:3fc15d54_6@news.athenanews.com...
| Dennis,
|
| before you get too excited (and without wishing to rain on your parade)
| there are some serious factors you need to consider.
|
| See below...
|
| "Harley" <dennis.harleyNoSpam@worldnet.att.net> wrote in message
| news:Tm2wb.315845$0v4.18414718@bgtnsc04-news.ops.worldnet.att.net...
| >
| > "Clark F. Morris, Jr." <cfmtech@istar.ca> wrote in message
| > news:3F94484C.5BFFD357@istar.ca...
| > | As a z/OS (IBM mainframe) COBOL programmer, I am trying to get a
handle
| > | on the uses for more than 2 gigabytes of addressability for an
| > | application. Since both the i-series/AS400 and many Unix systems have
| > | 64 bit addressing, I was wondering if there were any applications on
| > | these systems that take advantage of the large address range that have
| > | COBOL main programs and/or subroutines. If so, I would appreciate it
if
| > | the driver for the large addressing range could be discussed and how
use
| > | of such a range made the application development easier.
| > |
| > | Clark Morris, CFM Technical Programming Services, Inc.
| > |
| >
| > TABLES.
| >
| > Lately, I've been doing some new development, (extensions for an
existing
| > system).
| > The main program loads data into tables, and processes them.
| > The program is a "pig" (it isn't optimized, uses exchange sorts.)
| > The process runs for under a minute wall time.
| >
| > The current, production, system relies heavily on VSAM random access,
and
| > has very heavy I/O traffic, some EXCP counts go to 1,000K.
| > The Wall Clock Time is 1.5-2 hours, the CPU utilization is low compared
to
| > the wall.
| > This system is mission critical ,and the JOB completion time is
monitored
| by
| > upper management.
| >
| > I'm looking at replacing some of the random access processes with
| sequential
| > processing, and using internal tables.
| > My GUESS is that I can get the system to run in well under 30 minutes.
| >
| > The amount of spaces you can use in a single application program is
| limited
| > by the compiler. But, you can write a modular system.
| >
| > The main advantage to such a large address space is storing data in main
| > storage, rather than on DASD.
| > If you can store all operational data in main storage, you eliminate I/O
| > waits.
| >
| You also deny resources to other applications that may be competing for
| it...

That's true, but the run time is about a minute, and the Operating System
grabs control so I don't hog the machine.
At the time of day that this would run in production, the CPU is at about
60-80% utilization.

|
| > Think about how much faster your applications would be if you eliminate
| I/O
| > waits.
| >
| Think about how much data you will lose if the system crashes or your job
| abends <G>...

It runs in under a minute<G>.
Just re-run.
With the current update archecture, they have to restore the VSAM files, and
then rerun.
They also have to back-up the files in case the next update prograsm fails.
|
| If this is a purely Batch process and you are experiencing I/O problems,
| wouldn't the first line of attack be to try and improve the VSAM
| performance? Talk to your DBAs and Systems people before you re-design
this
| application.

The system was looked at by our "in-house VSAM experts", they found nothing.

What I found out is this:
1. The VSAM files are ALL defined with one or more Alternate Indexes.
2. The Alternate Indexes are all defined as NONUNIQUEKEY, (duplicate
Alternate Key values are acceptable) even though there is no valid reason
for allowing Duplicates.
3. The Update process is totally random.

4. The I/O counts are reduced significantly by:
a. Defining the Alternate Indexes with UNIQUEKEY (no duplicate alternate
Index Keys)
b. Updating in Primary Key, Secondary Key sequence in Sequential, rather
than Random mode.

There are other alternatives that I'm aware of, but that involves getting
support from the Systems area. It would only take me about 6 months to fill
out the necessary paperwork <G>.
.
|
| I don't know whether your system is supporting on-line access concurrently
| with the batch (most mainframes are) but it is worth considering the
| fundamentals:
|
| 1. On-line performance depends on 3 factors: load time, queue wait time,
and
| capture time. Without a great discourse on this, what it means is that
| "SMALL is beautiful" and memory MUST be available for on-line modules to
be
| loaded. (The more memory you have, the less they need to be swapped.)
|
| 2. Batch processes should utilise as much memory as possible (which is
| exactly what you are proposing) PROVIDED they do not do so at the expense
of
| the on-line system. So, if this is an overnight Batch run, no problem.
("BIG
| is beautiful", for Batch)
|
| But hang on a minute... Even if your process is overnight Batch and there
is
| no danger of it degrading other key processes, won't it require
significant
| changes to convert what are currently I/O areas into (VERY LARGE) tables?

EVERYTHING is tables.
Sequential files are tables, you just get one element at a time.

The Operating System will probably save these batch files in memory until
they aren't needed in a subsequent step. The programmer won't even be aware
of it.

| Then you must provide a mechanism for loading these tables (and storing
| them), change all existing access to be subscripted/indexed, and
regression
| test.

Not a big problem.
I have application programs that tell me what has changed, and I have
regression testing already set-up.

|
| All of this can be done of course, but what I am saying is don't undertake
| to do it overnight...<G>
|
| If the problem is really that the VSAM random access is slowing everything
| down, you should certainly discuss it with your DBA (he/she might be glad
of
| something interesting to do for a change...<G>). If this is the case, and
| everyone agrees that the I/O has been optimised as far as it can, then you
| could try progressively replacing the most frequent random accesses. (In
| other words, I would approach this on a phased basis.)
|
| Before you decide to go and use the entire address space, you MUST discuss
| it with your system people. There are likely to be repercussions for JCL
| (Job Classes), system paging, load balancing, scheduling, etc. etc...
|
| Yes, your job will be allocated a Virtual space as big as you desire, but
| that Virtual space still has to be mapped to real memory. It would be a
pity
| if you simply exchange an I/O problem for a thrashing one...
|
| Talk to DBA, talk to systems, and Good Luck!
|

I may need the luck!