memory allocation overhead

From: Mike M?ller (mmueller_at_dgfz.de)
Date: 04/28/04 

Date: 28 Apr 2004 03:38:28 -0700

I am porting a f77 application to f90. One of the most outstanding
features of the application is its memory management. Since the f77
standard allows only static memory allocation and vendor specific
dynamic memory allocation at this time (20 years ago) were slow (inner
loop, where allocations maybe called million of times) it has its own
scheme.

The application allocates a large junk of memory at start up. This
memory can be accessed by index. Therefore, the memory can be used for
dynamic size arrays at run time.

There are two parts to the memory. One for arrays that stay around
over the lifetime of the application and the other for scratch arrays
that can be used in a subroutine and then "deallocated".

The first type of memory (i.e. global arrays that are needed for most
part of program run) can be allocated once in a module and than made
available through use association.

The scratch arrays can be easily implemented as automatic arrays. But,
since those automatic arrays are allocated and deallocated in each
subroutine call the overhead for this might be considerable for very
frequent calls.

It would be interesting to know how big the overhead for automatic
arrays is. How does it depend on compiler, OS, hardware? Are there
compiler options to reduced such kind of overhead? ...

Since scratch arrays are used again and again, allocating them once
and just reuse them might be a solution. But there are a lot of
scratch arrays. So total memory usage might be enormous. Lots of
rarely used arrays would stick around but are of no use anymore.

Another way I though of, would be to allocate a bunch of arrays for
scratch purposes when they are needed the first time and keep them in
a module for use by other subroutines. That means, those subroutines
use the same arrays, or parts thereof, and just overwrite their
content, maybe setting a default first (zero or so). This would also
be some kind of own memory management but achieved with f90 means
(modules, dynamic allocation).

I would be very interested in the experience of others with similar
problems.

Thanks.

Mike

Relevant Pages

  • dgemm subroutine in BLAS - I think Ive cracked the difference, please confirm
    ... see what I'm talking about), so in the current state of memory allocation, ... but I have a problem when it comes to applying the dimensions. ... > In the Fortran 66 days it was because dynamic allocation didn't exist, ... > you allocate arrays big enough for the largest problem you might run into. ...
    (comp.lang.fortran)
  • memory leak in (?)... (redux)
    ... the mem leaks in our long-running tcl daemons. ... when actively exercised the app grows continuously. ... the mystery is that while our audit code does show us a couple of arrays ... and memory are checked and meanwhile externally, ...
    (comp.lang.tcl)
  • Re: Memory Space
    ... more memory to store data arrays. ... HD and 4G of RAM and am only using several hundered meg of RAM in an array. ... a pared-down piece of code that shows the allocation ...
    (microsoft.public.vb.general.discussion)
  • Re: StackOverflowException.
    ... The Status object monitors the ... so I'm really at a total loss as to where I'm leaking stack memory. ... private static extern bool EscapeCommFunction(IntPtr hFile, ... Good to know about arrays. ...
    (microsoft.public.dotnet.framework)
  • Re: "new byte[132]" alignment on 16 bytes
    ... What exactly do you mean with "C# to always align a....". ... Byte arrays are allocated on the managed heap and this allocation ... the starting pointer MUST be starting at a 16 byte memory ...
    (microsoft.public.dotnet.languages.csharp)