Re: "Insufficient virtual memory" error message in Linux

Lorents wrote:
I am running a fortran program (it's a quantum chemistry program, a modification of "Columbus") on a machine with a Pentium D (that is, a dual core pentium 4), 4GB of ram and linux redhat. The program was compiled with Intel Fortran compiler 9.1

My problem is that if I tell the program to use more than 2 GB of ram the program eventually crashes with a message "insufficient virtual memory". I am still trying to understand if this issue arises because of an intrinsic limit of the CPU, of linux or of the compiler (or combinations of all three).
I'm not an expert and I've heard a lot of contrasting opinions about this. This is what I've gathered so far, and I'd very much appreciate if anybody more skilled than me could tell me

1) CPU limitations
Intel's IA32 architecture traditionally uses (from the 386DX on) 32-bits to address the memory, hence it can handle at most 4GB of ram (physical+virtual).

1a) Intel introduced from the Pentium Pro onwards a wider 36-bit address bus ("Physical Address Extension" , PAE ), thus raising the amount of addressable memory to a good 64GB.
Anyway, a single application still uses 32 bit addresses and so can see at most 4GB. The advantage of PAE is that, provided the OS supports it, several applications can be run simultaneously, each one taking at most 4GB.
However, there seems to be some "tricks" that allow a single application to access more than 4GB. In windows, this should be called "Address Windowing Extensions" (AWE). I don't know if something of this kind exists for linux.

1b) From the latest versions of the Pentium 4 (from the second half of 2004) Intel introduced its "Extended Memory 64-bit Technology" EM64T, an implementation of AMD's AMD64 extention to the IA32 architecture. This theorically makes available 2^64 bytes of memory, even if present implementations support "just" 2^40 bytes = 1TB.
To take advantage of this both the OS and the application at hand should be compiled so as to take advantage of the 64 bit extention.

So, from all this, I understand that **potentially** my CPU is capable to handle more than 4GB of ram, either via EM64T or perhaps via PAE.

2) Linux limitations
Here the situations get a bit more fuzzy (to me).

2a) The "standard" IA32-compiled version of linux's kernel supports only up to 4GB or memory (physical+virtual). Moreover, the actual amount of memory available for a single application is always less than that; from what I understood, in the usual configuration this amount is of 2GB (and this would explain why my application crashes at this point) but it can be increased to 3GB. The remaining 1GB is reserved to the kernel (that much?!)
Can anybody confirm this important point? If this is the case it'd be good news for me -- at least I could give my application one extra GB of memory.

2b) It is possible to compile linux with the PAE, but amont of memory available to a single applications should still top 3GB: see http://www.spack.org/wiki/LinuxRamLimits


2c) It should be possible with my CPU to install a 64 bit version of linux, which should remove all these memory limitations.
Would the application require recompiling as well? After all I just want to make available to it 4GB, no more than that.

I don't know of any adequate support for PAE in Fortran compilers. 32-bit linux is quite limited in usable address space available to a Fortran program. Normally, there is little point in installing more than 2GB RAM for 32-bit operation, unless MPI or OpenMP is in use.
Running a 32-bit application under the 64-bit linux should give some expansion in address space available to a program, possibly making use of up to 4GB RAM. Re-compiling with a 64-bit compiler will make available 2GB space for each static data object (array or common block), and relatively unlimited space for allocatable data.
.

Relevant Pages