Speed versus memory

I am going through and updating a computation fluid dynamics solver
from F77-style programming to "modern" Fortran (F95 not 2003 since it
doesn't seem to be generally available). The code will be used to run
very large jobs (upwards of 100 million grid points and many thousands
of time steps) on massively parallel architectures.

My initial impulse was to use vector processes everywhere. To some
extent this was driven by books such as "Numerical Recipes in F90" and
"Fortran Explained 90/95" which take the stance that do-loops
introduce an artificial order to the processes (do i=1, then i=2, then
i=3...) which limit a compiler's ability to generate fast executables.
Algorithms written using vector operators are also generally less
verbose, which I find makes them easier to read and check for bugs.

However, I've noticed that writing complex algorithms (such as an
upwind quadratic interpolation routine) using vector notation requires
an absurd amount of memory. Such a routine only has two input arrays
and one output array but goes through ten intermediate arrays all of
which must have space allocated for them when using vector processes.
Even if the compiler splits the thread among a large number of
processors I have found that the routine will use thousands of times
more memory than the same routine written with do-loops (with ten
continually redefined scalars). I can't imagine this is
computationally fast and it's limiting the size of jobs I can run.

I have hacked together some methods using pointers and elemental
routines that seem like they might be ideal. As I understand it
pointers don't really use up memory (just a few bites per array) and
it seems that intermediate variables within an elemental are allocated
efficiently (they aren't blown up to the full array size unless the
computer can operate simultaneously on all those components) but I
have no idea how compiler and platform dependent that will be.

I'm wondering if other people have any hard-and-fast rules or guiding
principles in regards to the problem of speed versus memory
allocation. Thanks.

.

Relevant Pages

  • Memory-management when calling Compaq Array Visualizer library from Delphi
    ... I'm trying to use the old Compaq Array Visualizer v1.6 ActiveX control ... get a strip of memory through the aglMalloccall. ... aglReshaperoutine, which expects as one of its parameters an array ... Am I right in assuming that calling aglMallocfrom the Delphi .exe ...
    (comp.lang.pascal.delphi.misc)
  • Re: Speed versus memory
    ... I assume from context that by "vector processes" you mean what are more ... Optimization of whole array expressions is a much newer field. ... And that allocation/deallocation takes time independent of memory use ... also known as a bug... ...
    (comp.lang.fortran)
  • Re: The problems arrayed before me...
    ... The problem is that I suspect the amount of memory used will exceed the ... the arrays are dumped to disk and screen. ... I am going to look at overhauling the entire routine and have it store ... Eg set initial array size to 100MB ...
    (alt.comp.lang.borland-delphi)
  • Re: passing large arrays to functions
    ... functionally identical subroutine will not. ... typically create new memory, they just pass the address and work on ... of an array is passed, ... In some cases the called routine works ...
    (comp.lang.fortran)
  • Re: Fast string operations
    ... Looping: I thought looping over arrays in managed code was "slow" ... array handling and such. ... The problem with TrimHelper is that it always returns a new string instance. ... The customer perceives this as a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)
Loading