Re: When to check the return value of malloc

On Sun, 27 Jan 2008 21:18:15 +0000, Malcolm McLean wrote:

"Kelsey Bjarnason" <kbjarnason@xxxxxxxxx> wrote in message
No, I don't. I want a response back from my allocation function that
tells me it couldn't honour the allocation, that I can do any of a
million possible things with it.

In one application, for example, it will continue to work, simply a
little less efficiently, with a smaller buffer.

What you are describing is a classic case of over-engineering.

No, it's a case of engineering the application to do what it *should* do,
and engineering it *well* to do that job.

Overengineering is what you get when the designs are significantly beyond
the requirements, like writing a calculator app that handles everything
from bignums to integration, when what was asked for was a simple 4-
function calculator for kitchen-scope math.

Yes, your
program will be better, but you have to write lines and lines of custom
code to handle the failure of every single allocation.

No, I don't. I have to write one relatively simple loop which does the
job for me, consisting of a grand total of about ten lines of code.


Remember that
means cleaning up, unwinding the program state, destroying every object,
until you come to a point where you can recover.

No, it doesn't. I'm coming to the conclusion you have no idea how to
engineer an application, as what you're saying is complete and utter
bollocks.

If the memory request
was trivial, most of the time the program will fail to get memory again
on recovery, and will eventually close down or be closed down anyway.

What "most of the time" are you talking about here? In an MP system such
as Windows or Linux, an allocation that fails now may work ten seconds
from now, meanwhile the app may be able to do other useful things.

In other cases - such as the one I mentioned - an allocation failure is
not only not a fatal condition, it is an expected and trivial condition;
the app asks for as much memory as it can use for maximal efficiency, but
this has no bearing on how much memory it requires _to work_.

As a simple example, let's ponder a function which parses data from a
file, where the expectation is that the typical file so parsed is
relatively large - say several gigabytes.

Option 1: run through it, character by character, using fgetc() or some
equivalent. This was actually tested, and performance sucked.

Option 2: run through it line by line, using fgets or some equivalent.
This was also tested, works better, but not as well as it should.

Option 3: allocate a buffer of size N, read N bytes' worth of data,
process in memory. This, based on testing, is the most efficient, with
efficiency increasing, on average, with increasing values of N, to a
point, say around 64MB, where the benefits of more memory are
insufficient to justify increased usage.

If I'm building such an app (I did), I'm going to want to allocate 64MB.

However...

The app will continue to work, just less efficiently, with considerably
less than 64MB allocated. It *needs* a minimum of 1-2MB, which means
that 2MB would be the "fail point", not 64MB.

Any allocation system which allows me to ask for the full 64MB *but
doesn't allow me to detect the failure and adjust the requested size* is
completely useless to me for such a case.

And that's exactly what xmalloc does: it allows me to request the 64MB
I'd like, but prevents me from detecting the failure and adjusting the
request.

You complain that malloc causes problems. Even assuming you had a point,
your solution is worse than the problem. At least with malloc, *I* can
adjust to conditions. *I* can choose to use a smaller buffer. *I* can
detect the error and decide that perhaps I should try another task, then
check back later to see if memory is available. With your code, no such
options are available.

Granted, with your "user function" or whatever you call it, I can, in
principle, say "try again, with a smaller amount of memory", but that
doesn't help, as I cannot tell the calling code that the change has
occurred; it asked for 64MB, what was *actually* allocated was 16MB, how
the hell does it know that? It doesn't - there's no way to respond back
that the request was not honoured.

But what if the memory requests are so small that in fact the chance of
the computer breaking is greater than the chance of malloc() failure?

And what if the sun was covered in bright green puppy dogs? Such
questions make no sense.

Here's some reality for you: multitasking systems, particularly multi-
user systems, do not generally have fixed memory consumption. My word
processor requires 2MB now, but 20MB five minutes from now, as I open a
larger document. In the background, logrotate fires, calling a
compression program to archive the old logs. A scheduled app - leafnode,
say - fires, sucking up memory to get its work done.

It is hardly impossible to exhaust available memory, even in a
comparatively lightly-loaded system - and my boxes, as a rule, are
anything *but* lightly-loaded.

The notion an allocation failure is so unlikely it won't happen before
the hardware fails is simply so blindingly naive I can't believe you
actually think that makes a sensible argument - which in turn supports my
earlier contention, that you have no idea how to engineer an application.

If
you've a backup computer, that runs a mirror of the program, sure, the
application must never terminate. It's running a nuclear power station
or something. But if you don't provide such a hardware mirror, can you
really say "the application must never terminate?"

Who said "the application must never terminate"? You did. I said that
unmanaged, ungraceful termination is a bad idea. An even worse idea is
to build code in such a manner as to *force* such failures. A truly,
stupendously idiotic idea is to build code such that your only available
option is to terminate ungracefully, rather than do so gracefully, or
choose to perform another task, or whatever else the developer sees as a
reasonable course of action.

By your thinking, we could drastically simplify the entire C standard
library.

Take, oh, fopen. Use your design: you either get back a properly-opened
file, or the app dies. No option whatsoever to respond back to the
calling code, perhaps to log that the file didn't exist, or couldn't be
opened, so carry on with the next one.

Or how about, oh, fwrite. Can't write the data? No problem, after all,
it's not like the data matters or anything, so we'll just try the write
and if it works, we'll say so and if it fails, we'll abort.

How about fread? You wanted 512 bytes, but there were only 300 in the
file? That could be considered an error, so let's not return an error
result and handle it, let's pop up a window saying "Please add more data
to the file" and wait for the user to do so, or abort the application.

Error returns, count returns, these things exist for a reason, and the
reason is *not* so we can simply disregard them, it is so that we can
*detect* them and *deal with* them.

Your entire approach is so fundamentally flawed from a software design
perspective, never mind a coding perspective, that it simply is not of
any use at all, not in a toy program, not in a real program. Maybe it
keeps you happy at night, I don't know, I don't care; it - the entire
methodology, never mind the specific code - will never make it into any
application I have the fortune to be working on, and if I ever do
encounter it, it will be summarily removed, with extreme prejudice.

Frankly, if I saw that sort of thing in anything claiming to be real
code, I'd take steps to have the programmer fired; he's simply not
competent to be working on anything more complex than "Hello, world".

I'm not trying to insult you, Malcolm, I'm trying to point out something
very basic here. Your design is to, in essence, pretend that such
failures don't exist, when the simple reality is that they do. Hell, the
code for xmalloc _itself_ admits they do, else it wouldn't need to
validate the call to malloc. In fact, the very *existence* of xmalloc
says such failures occur; if they didn't, you wouldn't want xmalloc,
you'd just use malloc and not bother to check for NULLs.

Software designed with a rose-coloured view may make life easier for the
developer initially, as he doesn't have to write the error handling code,
but when it's faced with a big bad world where such things happen, not
just rarely but regularly, the code written in such a blinkered manner
suddenly needs a major overhaul, because error handling needs to be
hacked in after the fact, instead of designed in up front.

Using functions such as xmalloc, the way it works, isn't just dangerous
to our apps - it can't be, as there's simply no way most of us would ever
let such blinkered notions into our code in the first place - it is
dangerous to *your* apps as well, as it means you're not designing proper
error handling in, and when you finally discover you need it, you haven't
got the mechanisms in place to deal with it.

If you actually have something to replace malloc which *actually improves
the resultant application* then by all means, bring it on. xmalloc isn't
it. It's designed from an inappropriate view of the world, it lies to
the user and it doesn't even meet the lies it tells, so it's not a viable
solution.


.

Relevant Pages

  • Re: xmalloc string functions
    ... than a NULL return from malloc(). ... Memory is quite a different kind of resource. ... I try to write my code so any allocation failure is handled ... its not likely that one will be more susceptible to failure than the other. ...
    (comp.lang.c)
  • Re: xmalloc string functions
    ... There is nothing to recover because failure of fopen ... then you can safely abortwhen fopenfails. ... doesn't have memory to draw stuff it draws. ... If a fairly large allocation fails, ...
    (comp.lang.c)
  • Re: A solution for the allocation failures problem
    ... reduce the effort involved in managing memory. ... Releasing such buffer may not effect the ability of malloc() to return ... says it releases the memory for subsequent allocation, ... Dealing with the failure "at point of failure" isn't inherently evil. ...
    (comp.lang.c)
  • PROBLEM: unchecked returns from kmalloc() in linux-2.6.10-rc2
    ... Checking whether memory allocation succeeded is important, ... Failure might cause a kernel crash. ... the return from kmalloc() is validated before ... Here, memory is allocated for pcpu_size on line 335, and then ...
    (Linux-Kernel)
  • Re: xmalloc string functions
    ... than a NULL return from malloc(). ... Or I can use the debugger to put a breakpoint in and change the pointer value to null at the point I want to trigger the failure. ... Memory is quite a different kind of resource. ... I try to write my code so any allocation failure is handled ...
    (comp.lang.c)