Re: speed up calculation suggestions

You are an ACTUARY.
I assume your purpose is to create actuarial tables from a statistical
analysis of data supplied from your sources in text form.

This is very much the same kind of situation that survey companies
have, except these have a smaller sample and ask more questions, but
the dta sizes are comparable.

As noted by others, I believe the speed of your computation is mainly
a function of the disk I/O access to your data, both as sheer time to
get a record in, plus the time the computer takes to transform the
data to a binary-computable format internally.

A question arises: do you ADD data to the SAME data files you
processed last time?

In other words are you always in a state of reprocessing the same
older data with a lot of newer data added to it? If so, one approach
is to convert your input data to a common binary format, using only
the demographic and other data measurements you need from the data and
keep the new file. When further data arrives, only this is converted,
than appended to the earlier file.
The computation is then performed on the binary file. Only the new
data is accessed twice.

From a mathematical standpoint a mathematician would simply generate
tables from an existing set of data and then add new information to
these tables by adding data from the new data text files before saving
the new tables and then extracting new statistics from the new table.

Both approaches save conversion and I/O time.




.