Re: xmalloc string functions

On Sun, 03 Feb 2008 12:08:02 +0000, Flash Gordon wrote:
Jeffrey Stedfast wrote, On 03/02/08 06:57:
On Thu, 31 Jan 2008 11:44:57 +0000, Richard Bos wrote:

ymuntyan@xxxxxxxxx wrote:

But it's not true, there are ifs and buts. Using glib, you *can* do
stuff when malloc() fails. Yes, using glib all you sensibly can do on
malloc() failure is some sort of emergency work and quit. If your
application requires more, then you don't want to use glib, that's
it. But I claim that for 'regular' desktop applications that is quite
enough.
If your regular desktop application is a clock, perhaps. If it holds
any user input, and for this the application can be as simple as a
calculator, it is not nearly enough. Consider how irritating it would
be to type half a dozen numbers, of twenty digits each, and a handful
of operations, only to have your application crash because it could
not get the memory for that last comlpex calculation. At the very
least, your app should put up a message box saying "Out of memory -
could not compute"

what if the system doesn't even have enough memory to pop up such a
dialog? Likely it'd take far more resources to display said dialog than
it would take to make the calculation ;-)

Have it pre-created together with all the resources it will require,
then it doesn't need any more resources.

if a dialog takes more resources than you're ever likely to need in a
calculator app to do a calculation, doesn't this feel wasteful for the
99.9999999% cases?

Let's also not forget that the act of /showing/ the dialog may, in fact,
require memory allocations depending on the way the system works.

For example, requesting that a dialog be shown may not actually show the
dialog immediately... it might only queue the operation for the next
rendering pass.

Said rendering pass may require more allocations, but at this point it's
too late to simply unwind the stack to the point where you requested the
show(). Since no widget toolkit I know of has a way of notifying the
application of said error, what is it to do?


For Gtk+, you actually do have an option... GLib uses a vtable for malloc/
realloc/calloc/free that you can initialize with your own routines at
init.

You could potentially do your own NULL-check there so that you can be pre-
warned about memory allocation errors coming up, but it'll lack context
(who tried to allocate this memory? for what purpose?), but I suppose if
you had everything pre-allocated, ready to go - you could call some
global prepare_for_abort() function that could perhaps iterate thru all
of your unsaved files and save them quick before the abort() call in the
g_malloc() wrapper. This wouldn't allow you to pop up any dialogs,
however, because at this point its too late.


, and then let you copy the previous twenty-digit result into another
document for safe keeping.

ah, but that also requires a clipboard memory buffer be allocated...
but you have no memory left ;-)

Have it pre-created, then if you need a larger buffer for the next step
and you can't enlarge it you only loose that last step.

this one indeed is likely an easier and more reliable method for this
particular instance, but not all desktop applications can go around using
this type of approach.

For example, would it be a good idea for an email application to set
aside this clipboard buffer? :)

I think we'd both agree the answer is no.


Sorry to be a smart-ass here, but you clearly have not thought about
this problem in the context of the average desktop application.

Some of us have and have already suggested having and using a buffer
that is large enough for the emergency action.

the problem with this approach (and it's not a terrible one), is that it
means you have to be diligent about making sure it's a big enough buffer
to handle all your possible failure cases gracefully. In an application
that is 2 million lines of code, this is not trivial to accomplish.

Oh, and it's only 2 million instead of 2.5-3 million because it uses
g_malloc() :)


In a small daemon, if you get a malloc() failure, you have a lot more
options open to you than you do in a graphical desktop application
because you can do a lot more with very small amounts of memory (or
none at all).

A few examples:

- start dropping idle client connections (which would likely not
require any /new/ allocations) until you have enough memory to do the
critical operation you need to do

Drop that background print-job or spell check that is consuming memory
for your desktop app.

That means you'd have to have that print job context or spell-check
context global somewhere, or have some way of getting it from a lot of
different locations... GUI apps can't always pass errors up to main() (or
where ever your main event loop lives) quite so easily as the average
daemon can.


- print an out-of-memory or "sorry, can't do that right now" error to
the client socket (or terminal) which would likely not require any new
memory allocations (error strings could be static)

Pre-create the out-of-memory dialogue and any resources it requires.

doable if you don't want to give any specifics... daemons are often not
very user-friendly in their error reporting... depending on the daemon,
it might be as simple as an integer error code or as forthcoming as a
string from strerror(), but rarely do they report something that the user
is able to understand. Sure, "out of memory, cannot perform that
operation" may work for simple applications where only 1 thing at a time
is ever going on, but if the application happens to be doing many things
at once the user will want to know /which/ operation could not be
completed because memory was unavailable?

Trust me, this is the case... applications I've worked on have actually
had these sorts of complaints filed against them. It's funny, because all
the user testing I've seen indicates that users never read the dialogs
anyway ;-)


- wait until memory becomes available

With an appropriate pre-created dialogue you can do that on a GUI
application as well.

see above.


On the other hand, say, a word processor application, if the user
requests some sort of action and a malloc() fails for 12 bytes, what is
it supposed to do?

Any of the above.

Easier said than done, I'm afraid...


If the documents the user has open have already-opened file
descriptors, the app might be able to save them before going down -
but:

That is easy to arrange.

1. it certainly doesn't have the option of displaying an error dialog.

Yes it does if it pre-creates it during application start-up.

see above, although I suppose if you really wanted to, you could make an
exception for the "out of memory" dialog case as opposed to other error
dialogs your application might use.

First... I wonder if there are any widget toolkits that don't already
abort() (or similar) when they run out of memory or in any other
conditions without giving my calling code a chance to handle it?

As someone else mentioned, X already has this limitation... so right
there, that means there's no Unix toolkits that you can use. Guess we'll
all just have to write applications for ... does Windows or MacOSX handle
this? I somehow doubt it.


2. if any of the files are unnamed or otherwise would require any of:

Don't allow them to be unnamed. You can create a name at the same time
as creating the otherwise unnamed document.

What if the act of creating a name is what finds the out-of-memory
condition?


a) filename generation (would require string building)

Which can be done using a pre-allocated emergency buffer

what about memory that some lower-level stuff might require that you
can't control? Perhaps you use a library for writing said file out to
disk once you have the filename... even if only libc, you still need
enough memory for an fopen() to succeed - you can't make /it/ grab from
your emergency buffer.

Well, actually I suppose you could always replace the malloc symbol, but
that still leaves you the problem of not necessarily knowing the memory
requirements of lower-level library functions which means you can't
accurately calculate how much emergency buffer you need.


b) file descriptor opening (which takes memory)

Which could have been opened when the document was created

this isn't doable if it's possible to have a large number of things
opened. You may quickly run out of file descriptors.


c) user-interaction (this one is right out)

Which can be avoided or use pre-created dialogue boxes that won't need
any more memory than they already have.

see above


the application would certainly not be able to save the documents at
that point...

If it was planned for it could. I've come up with ways of dealing with
all of the problems.

not very good ways.

I maintain that a better way is simply to auto-save user's work... far
far simpler to implement and far more reliable a solution unless you are
able to provide 100% test coverage for your application.

Remember, we are talking about free software desktop applications written
by volunteers, here, not programmers paid 150k/yr to write branches for
each memory allocation they ever make in their application or library.

Especially provided that these programmers can at best assume that the
authors of the libraries they are building on top of have also done their
work of error checking every malloc and properly handling it and/or
chaining it up to the caller.

This is, in fact, where your whole argument falls apart. As a developer
of GNOME desktop applications (hell, scratch that - of X11 based
applications), you already KNOW that I cannot rely on glib or gtk+ (or
Xlib) to gracefully handle all memory allocation errors... so I have no
choice but to resort to my auto-save approach.

It's too easy to forget that large applications are generally built on
top of code that other people wrote that you may or may not be able to
read the source code for to verify that they properly handle all errors.
You could even make this argument for daemon authors - how many of you
have actually read through all of the source code for the libc you build
your applications on top of? None? Remember that and you might want to
rethink not using the auto-save approach (in addition to error checking).


Oh wait, I forgot that this whole thread is actually a pissing contest
more than anything else, so that people who don't actually write
desktop applications can feel superior to those who do.

I think it is more annoyance at application we might otherwise consider
using that would just throw away our hard work in situations some of us
do hit.

I've hit it as well, and yes, I'm also annoyed when it happens - but it
is a problem that cannot be easily remedied by the application developers
if the software stack somewhere underneath the code they've written is
faulty (and I have personally run into Linux kernel and GNU libc bugs
that have cause problems in applications I've written).

Since you cannot rely on error checking to catch all errors, it's best to
have a fallback plan - which is auto-save. It's a lot less likely to fail
and a lot more tolerant of buggy code (either in your application or
below your stack).

In the real world, developers do not have the luxury (or desire, most of
the time) to write applications from the ground up. They build on top of
software that already exists.

Jeff
.

Relevant Pages

  • Re: xmalloc string functions
    ... require memory allocations depending on the way the system works. ... If the toolkit being used is not one of those, then it is irrelevant that some provide a means to do so, particularly if the "some" are not available for the platform being targeted. ... Not enough context for most real-world applications to recover at this point. ... At this point g_malloccalling abortbecomes a moot point, particularly if your auto-save code is robust against memory allocation errors. ...
    (comp.lang.c)
  • Re: xmalloc string functions
    ... malloc() failure is some sort of emergency work and quit. ... not get the memory for that last comlpex calculation. ... you can reuse the solution so even if it takes a week amortise thatover all of the applications you will write! ... The open source community wants to be taken as a serious alternative to closed source, so it should take the same effort to produce robust applications and libraries. ...
    (comp.lang.c)
  • Re: ProDOS Plus
    ... operating system was not considered worth the problems when it was just as easy to make the applications support 128k or more ram. ... your suggested P-code system could do something similar by running in the aux 64k memory range $D000...FFFF or perhaps the aux ram used by the P8 /ram driver code space. ... the OS could fit in *only* the space that ProDOS now occupies. ... if practicality were a concern we probably wouldn't be using old hardware. ...
    (comp.sys.apple2)
  • Re: xmalloc string functions
    ... If these were the only choices (crashing applications or a frozen ... trying to malloc some rediculously large amount of memory - e.g. in the ... because their malloc wrapper decided to exit. ... Exiting on malloc failure makes sense for a utility like sort. ...
    (comp.lang.c)
  • Re: xmalloc string functions
    ... require memory allocations depending on the way the system works. ... Not enough context for most real-world applications to ... It is /more/ reliable to routinely auto-save the user's work (as you ... particularly if your auto-save code is robust against memory allocation ...
    (comp.lang.c)