Re: calloc/free: a preplexing observation

From: Lawrence Kirby (lknews_at_netactive.co.uk)
Date: 02/03/05 

Date: Thu, 03 Feb 2005 13:11:24 +0000

On Thu, 03 Feb 2005 00:38:23 +0000, Keith Thompson wrote:

...

> malloc() is a C library function, part of the C implementation, not
> (necessarily) part of the operating system.

Ultimately everything that affects behaviour, including the operating
system, has to be considered part of the implementation.

> Whatever requirements the
> C standard imposes on malloc(), it's up to the implementer to do
> whatever is necessary to make sure that it meets those requirements.

Absolutely. I say though, as I explained in another article, that the
standard does not prohibit memory overcommitment.

> If the OS provides, say, an ALLOCATE() function that allocates space
> that isn't necessarily properly aligned, malloc() has to arrange to
> return a pointer to properly aligned memory.

Agreed.

> If ALLOCATE(0) aborts the
> calling program, malloc(0) can't invoke ALLOCATE(0). And so on.

As long as the abort process dosn't generate anything that is considered
to be program output or a normal termination condition, it could. Of
course that may be unacceptable on QOI grounds, it depends on the
situation where the abort can happen. The standard has to be very loose in
this area to allow conforming implementations to exist at all.

> If lazy malloc() is non-conforming, an implementation of malloc() can
> attempt to access the allocated memory before returning the pointer;
> since the code that implements malloc() can be a system-specific as it
> needs to be, presumably it can handle any traps, deallocate the memory,
> and return a null pointer. I'd much rather have malloc() do that than
> expect the caller to take care of it.

Even if you do that the OS can decide subsequently that it dosn't have
enough memory to go around and yours is the program it is going to kill to
recover some. The kill action might be initiated by your program accessing
part of a malloc'd array that had been paged out. The point is that it
isn't always possible for a C compiler/library to work around what the OS
does.

Consider this another way. It has come to light that many OSs in common
use can overcommit, and there are good reasons (as considered by some
people) for them to do so. This is an OS level issue. It is NOT the
responsibility of a language level standard such as C's to legislate on
this sort of thing. The purpose of a standard like C's is to provide a
language definition that is useful and portable across as many platforms
as possible. If it specified something that made it unimplementable on a
significant class of platforms and the "usefulness" isn't terminally
impacted (overcommitment is presumably deemed acceptable on the systems
that do it), that would have to be considered a MAJOR defect in the
standard. It isn't a matter of liking or hating a feature like
overcommitment it is a matter of recognising that something exists as a
common feature of real systems and it is somebody else's problem (i.e. not
the C standard committee's) to decide whether it is a good thing or not.
If I use an overcommitting system I don't want C programs subverting that
to the detriment of non-C programs. If I don't like overcommitment I
configure the system I have appropriately or use a different one. The
appropriate solution to this is not at the language level, unless you want
to add features for a program to request that its memory be reserved fully
for it.

> The standard says:
>
> The malloc function allocates space for an object whose size is
> specified by *size* and whose value is indeterminate.
>
> I argue that if I can't access the space, it wasn't really allocated.

In the abstract machine, yes, which is what the majority of the
standard including this describes (see 5.1.2.3p1). OTOH show me the part
of the standard that says an actual implementation can't terminate the
execution of a strictly conforming program at any point it pleases for any
reason, IOW that the program will successfully execute to completion.
There is a requirement for (at least) one program as specified by 5.2.4.1.
But the "at least one program" wouldn't make sense there if it was already
the case for all strictly conforming programs.

> Having said that, I have to admit that my argument against lazy
> allocation is largely based on the fact that I don't like the idea, at
> least for an explicit allocation request.

What you are guaranteed is that while the execution of the program
continues the object created by malloc() will behave correctly as an
object. It just doesn't guarantee continued execution of the
program subsequently.

Lawrence

Relevant Pages

  • Re: close fd while select/poll/epoll
    ... out of memory other than malloc returning NULL. ... POSIX *cannot* make this illegal. ... my proposal becomes part of the standard. ...
    (comp.os.linux.development.system)
  • Re: malloc under linux
    ... It means what it says, i.e., even if malloc returns a non-null ... memory might not actually be allocated. ... Standard even when some other process is killed and sufficient memory is ... I can't see how the Standard does not permit lazy allocation, ...
    (comp.lang.c)
  • Re: Help with Enter and Leave Instructions
    ... >>> for Memory Accesses, ... > The only standard way to do it is via malloc. ... Uh, SBRK/BRK is a standard, documented system call. ... > I really don't understand the objection to using C libraries. ...
    (alt.lang.asm)
  • Re: Whats the difference between the heap and the freestore?
    ... your assumption that new and malloc share something ... > What I though was that both new and malloc allocated memory in a common ... still different types and if a compiler chose to make longs 64 bits wide, ... The standard allows for more important differences. ...
    (comp.lang.cpp)
  • Re: Memory Overcommit issue under c++
    ... jon wayne wrote: ... I hope you realize that you may be asking for a lot more memory here ... even if you have overcommitment switched on - ... malloc() isn't meant to throw an exceptions - it's your responsibility ...
    (comp.os.linux.development.system)