Re: Heaps and Foreigners

From: Duane Rettig (duane_at_franz.com)
Date: 05/04/04 

Date: 04 May 2004 13:00:24 -0700

tfb+google@tfeb.org (Tim Bradshaw) writes:

> David Steuber <david@david-steuber.com> wrote in message news:<87u0yx86yr.fsf@david-steuber.com>...
>
> > I mean in the sense of reserving all the available address space for
> > heap. From the way I gather that malloc works on some systems, you
> > can ask for the memory and get a pointer to it. So far, so good.
> > Once you start to use that memory, the OS then tries its darndest to
> > give it to you. Bad things happen if there is not enough physical
> > memory to back the virtual/logical memory. (I hate it when terms are
> > used imprecisely. I no longer know what is being talked about.).
>
> I think this is right.

No, no, no! This is correct in a descriptive sense, but this line
of pursuit is absolutely wrong for most applications that do
any significant allocation (i.e. any app that grows), and will likely
cause unfathomable deaths when swap space gets low or is exhausted.
And, of course, Lisp will be blamed.

> You get all the address space, but it will not
> be mapped to anything until you actually ask for it.

But the asking for it is critical. You already have asked for it, and
have been promised it, but then when you go to get it, the operating system
renegs, and you're left with a cryptic error message that is completely
undebuggable. See below.

  This is pretty
> much always the right thing to do, I think as it allows tricks like
> mapping enormous spaces which you expect to use only sparsely (for
> instance, the Solaris filesystem snapshot system memory maps a file
> the size of the filesystem it is snapshotting, and then fills it
> sparsely). It would be good to have an `and if I actually try and
> touch all this memory now, what will happen?' call.

The way to get that to occur is to insist that all memory allocated be
backed by swap (not necessarily swap disk space - memory can also act as
swap backing). See below.

> On Symbolics
> machines, you could run out of space, get into a break loop, then add
> swap space and continue, and there's no real reason why systems can't
> do that now, I think.

It's probably because there is some reserve memory to print out errors
and potentially to recover. But many systems don't reserve such space;
it depends on the operating system.

Let me first define terms that I will use, and then I'll use them in
explanation below of our experiences with this issue:

 - Physical memory: memory that has physical addresses; e.g. address zero
has all address lines at binary zero, and resides at precisely one location
in Random Access Memory.

 - Virtual memory: memory that is not physical, but which instead resides
somewhere on storage not necessarily at the physical location. A process
might define address 0 for itself, which (depending on whether it is
shared or not) may not be the same physical address as another process's
address 0. If the storage representing the address is disk space, then the
virtual memory is said to be "swapped out".

 - Virtual swap (an SGI term): memory that is not backed up by actual
physical storage. A system can have a mixture of physical swap (consisting
of RAM and disk space) and virtual swap (which is just a number managed
by the operating system).

Examples of virtual swap:

 1. SGI: "man swapctl" will describe virtual swap and how it can be
controlled. Irix did away with the distinction between fork() and vfork()
and instead allowed all forks to operate, even if there was not enough
swap space to perform a copy of the entire address space, as long as the
virtual swap requirements were met. This was done so that a huge
program (say, a program which consumed more than half the swap space,
such as a lisp :-) could fork itself for the purpose of exec-ing a much
smaller program, such as pwd.

 2. Solaris: mmap has the MAP_NORESERVE attribute, which provides for a
mapping which does not grab actual swap space until it was actually
accessed.

 3. AIX: AIX also allows virtual paging space, but also provides a SIGDANGER
signal, which says that the paging space is getting low. The default action
of SIGDANGER is to terminate the application.

We first encountered this problem of virtual swap on NeXT boxes, which
would _always_ allocate virtually, but when there was no more paging space,
the system would hang until the space became available, If most programs
were not going to terminate, and some programs needed yet more swap,
then this would cause deadlock situations. I don't remember what we did
to work around this problem; I didn't do the NeXT port of Allegro CL,
but I do remember it being a problem.

The next time we saw this problem was on AIX. Interestingly, AIX had
accounted for this problem, and had provided SIGDANGER, which we
call signal() on to catch (so our lisp doesn't go away). However,
ironically, the X11 that was provided with AIX was not compiled with
any calls to handle SIGDANGER, so although our lisp did not terminate
when swap space got low, X itself would terminate. The percentage of
memory that triggers a SIGDANGER is configurable (and the amount of
physical blocks is very easily configured, as well) but it is hard
to know how much memory one needs to recover from an ou-of-swap situation.

Finally, we have had customers with huge applications on both SGI and
on Solaris (which eventually pressed us to do 64-bit versions of
Allegro CL on many platforms) who also noted that when their physical
swap space was recduced and virtual swap was increased (due to them
needing the extra disk space for file storage purposes, or wanting to do
vforks, etc) the number of incidents rose in which unexplainable deaths
of the lisp _and_ the operating systems would occur. These deaths
could _never_ be reproduced when the virtual swap was zero (i.e. all
virtual memory was backed up by physical swap), but when we dug in to
debug these situations, we found that inevitably what had happened was
that the program snarfing up more physical pages (by touching them) was
then grabbing the last physical page, and then when it couldn't get any
more (and, meanwhile, other system daemons would try to allocate memory
and fail, thus terminating citical operating system programs) the first
touch of virtual space which failed would cause SIGBUS or SIGSEGV, but
an error message couldn't always be printed out (because such a call to
printf required a buffer for the first time, and memory for such a buffer
wasn't available!)

In short, don't go the route of allocating virtual swap; it will
inevitably lead to disaster. The rub is, that disaster will likely
happen on a lisp, because it will be one of the few systems that can
take an application to such large memory requirements, and so thus Lisp
will be accused for the failure.

I think that what David is / should be really looking for is
virtual _address_ allocation, rather than virtual swap (or overcommit,
or whatever else you want to call it). It is ok to reserve address
ranges, and to adjust its permissions to No Access so that stray pointers
don't accidentally commit pages that weren't asked for by the heap
manager. Such stray accesses will cause Segvs, as is appropriate, and
when the heap allocator wants to turn portions of the reserved address
ranges into usable heap space, it can remap that range of memory (but
not virtually) and adjust the permissions again to cover exactly that
new memory that is desired. This also guarantees that any allocation
errors that will occur will occur at heap-management time, and not in
some random function that happens to cons (with no indication why it
failed). 

-- 
Duane Rettig    duane@franz.com    Franz Inc.  http://www.franz.com/
555 12th St., Suite 1450               http://www.555citycenter.com/
Oakland, Ca. 94607        Phone: (510) 452-2000; Fax: (510) 452-0182

Relevant Pages

  • Re: Is shared memory pinned or swappable on HP-UX B.11.23 U ia64?
    ... TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME ... That's allowed to be reserved as if it were disk swap to allow ... large memory systems where the workloads fit in core to not bother ... That's approximately the maximum size of a new segment -- ...
    (comp.sys.hp.hpux)
  • SUMMARY : RE: RAM usage
    ... RMCmem package which quickly identifed the amount of reserved swap vs used ... lot of money is they report just about all memory always used. ... This memory is in RAM ... Please reserve it for me". ...
    (SunManagers)
  • Re: Mem requirement for additional JREs
    ... pessimistic and assume that all the memory will actually be written & ... reserve enough swap for that, but since most of it actually is *not* ... There is a story here about overcommit: I think there is a paper ...
    (comp.unix.solaris)
  • [HPADM] RE: Clarification on swapinfo/glance output
    ... Seems to me you need to increase your swap by adding another device swap, ... seems that your Oracle instance is needing to reserve more swap area. ... memory line of swapinfo isn't actually physical memory it is pseudo swap. ...
    (HP-UX-Admin)
  • Re: Is Greenspun enough?
    ... Most OSes memory map executables directly from the file system so code doesn't pollute the file cache or swap space. ...
    (comp.lang.lisp)