Re: xmalloc string functions

Jeffrey Stedfast wrote, On 03/02/08 14:51:
On Sun, 03 Feb 2008 12:08:02 +0000, Flash Gordon wrote:
Jeffrey Stedfast wrote, On 03/02/08 06:57:
On Thu, 31 Jan 2008 11:44:57 +0000, Richard Bos wrote:

ymuntyan@xxxxxxxxx wrote:

But it's not true, there are ifs and buts. Using glib, you *can* do
stuff when malloc() fails. Yes, using glib all you sensibly can do on
malloc() failure is some sort of emergency work and quit. If your
application requires more, then you don't want to use glib, that's
it. But I claim that for 'regular' desktop applications that is quite
enough.
If your regular desktop application is a clock, perhaps. If it holds
any user input, and for this the application can be as simple as a
calculator, it is not nearly enough. Consider how irritating it would
be to type half a dozen numbers, of twenty digits each, and a handful
of operations, only to have your application crash because it could
not get the memory for that last comlpex calculation. At the very
least, your app should put up a message box saying "Out of memory -
could not compute"
what if the system doesn't even have enough memory to pop up such a
dialog? Likely it'd take far more resources to display said dialog than
it would take to make the calculation ;-)
Have it pre-created together with all the resources it will require,
then it doesn't need any more resources.

if a dialog takes more resources than you're ever likely to need in a calculator app to do a calculation, doesn't this feel wasteful for the 99.9999999% cases?

Last time I checked (which was a long time ago I know) a single dialogue box did not require that much especially as we are talking about a calculator which is already a GUI one.

Let's also not forget that the act of /showing/ the dialog may, in fact, require memory allocations depending on the way the system works.

So you have to look to see what workarounds there are for this. Once you've done that one, you can reuse the solution so even if it takes a week amortise thatover all of the applications you will write!

For example, requesting that a dialog be shown may not actually show the dialog immediately... it might only queue the operation for the next rendering pass.

On at least some there are ways to get it rendered immediately. I know I've done that in the past. Once you have solved it for one application reuse the solution for others.

Said rendering pass may require more allocations, but at this point it's too late to simply unwind the stack to the point where you requested the show(). Since no widget toolkit I know of has a way of notifying the application of said error, what is it to do?

So you are saying all widget toolkits are badly designed. This is possible. However, it has enough memory to do it. If you try and fail you are no worse off, if you try and succeed you are better off.

For Gtk+, you actually do have an option... GLib uses a vtable for malloc/
realloc/calloc/free that you can initialize with your own routines at init.

See, there *are* ways to deal with it!

You could potentially do your own NULL-check there so that you can be pre-
warned about memory allocation errors coming up, but it'll lack context (who tried to allocate this memory? for what purpose?), but I suppose if you had everything pre-allocated, ready to go - you could call some global prepare_for_abort() function that could perhaps iterate thru all of your unsaved files and save them quick before the abort() call in the g_malloc() wrapper. This wouldn't allow you to pop up any dialogs, however, because at this point its too late.

No, it's not too late, since as you save documents you can free up the memory they used giving you the memory to pop up dialogues! Or, as previously mentioned, have stuff pre-allocated.

, and then let you copy the previous twenty-digit result into another
document for safe keeping.
ah, but that also requires a clipboard memory buffer be allocated...
but you have no memory left ;-)
Have it pre-created, then if you need a larger buffer for the next step
and you can't enlarge it you only loose that last step.

this one indeed is likely an easier and more reliable method for this particular instance, but not all desktop applications can go around using this type of approach.

For example, would it be a good idea for an email application to set aside this clipboard buffer? :)

I think we'd both agree the answer is no.

In that instance you would use a different solution, probably saving the email in a draft folder or something like that. People have already said there is not a one-size-fits-all solution!

Sorry to be a smart-ass here, but you clearly have not thought about
this problem in the context of the average desktop application.
Some of us have and have already suggested having and using a buffer
that is large enough for the emergency action.

the problem with this approach (and it's not a terrible one), is that it means you have to be diligent about making sure it's a big enough buffer to handle all your possible failure cases gracefully. In an application that is 2 million lines of code, this is not trivial to accomplish.

How easy it is depends on a lot more that the line count. It also depends on the design.

Oh, and it's only 2 million instead of 2.5-3 million because it uses g_malloc() :)

If 25-33% of the code is handling out-of-resource conditions then you probably have a very badly designed application. Even Malcolm does not expect it to be that high for the application as a whole!

In a small daemon, if you get a malloc() failure, you have a lot more
options open to you than you do in a graphical desktop application
because you can do a lot more with very small amounts of memory (or
none at all).

A few examples:

- start dropping idle client connections (which would likely not
require any /new/ allocations) until you have enough memory to do the
critical operation you need to do
Drop that background print-job or spell check that is consuming memory
for your desktop app.

That means you'd have to have that print job context or spell-check context global somewhere, or have some way of getting it from a lot of different locations...

You probably need mechanisms to signal the spell checker and print process anyway to cope with the user choosing to abort them.

GUI apps can't always pass errors up to main() (or where ever your main event loop lives) quite so easily as the average daemon can.

Maybe not, but it still has mechanisms for the different parts of the application to communicate.

- print an out-of-memory or "sorry, can't do that right now" error to
the client socket (or terminal) which would likely not require any new
memory allocations (error strings could be static)
Pre-create the out-of-memory dialogue and any resources it requires.

doable if you don't want to give any specifics... daemons are often not very user-friendly in their error reporting... depending on the daemon,

That it depends on the daemon shows that is is possible, or it would be a simple case that none do.

it might be as simple as an integer error code or as forthcoming as a string from strerror(), but rarely do they report something that the user is able to understand.

Normally they report something the system administrator is able to understand. At least, most that I use do.

Sure, "out of memory, cannot perform that operation" may work for simple applications where only 1 thing at a time is ever going on, but if the application happens to be doing many things at once the user will want to know /which/ operation could not be completed because memory was unavailable?

Yes, which is why things like xmalloc are a problem, because they do not have that context.

Trust me, this is the case... applications I've worked on have actually had these sorts of complaints filed against them. It's funny, because all the user testing I've seen indicates that users never read the dialogs anyway ;-)

Depends. I've had a report come back to me (via at least a couple of layers of intermediaries) that had exactly the information that the "dialogue" provided (it was not a GUI application).

- wait until memory becomes available
With an appropriate pre-created dialogue you can do that on a GUI
application as well.

see above.

Again, see above ;-)

On the other hand, say, a word processor application, if the user
requests some sort of action and a malloc() fails for 12 bytes, what is
it supposed to do?
Any of the above.

Easier said than done, I'm afraid...

Worth the effort though.

If the documents the user has open have already-opened file
descriptors, the app might be able to save them before going down -
but:
That is easy to arrange.

1. it certainly doesn't have the option of displaying an error dialog.
Yes it does if it pre-creates it during application start-up.

see above, although I suppose if you really wanted to, you could make an exception for the "out of memory" dialog case as opposed to other error dialogs your application might use.

You have to take extra care with any out-of-resource error to ensure you can report it without the resource in question.

First... I wonder if there are any widget toolkits that don't already abort() (or similar) when they run out of memory or in any other conditions without giving my calling code a chance to handle it?

Lotus Notes has given me an out-of-memory dialogue. I'll leave you to draw your own conclusions from this.

As someone else mentioned, X already has this limitation... so right there, that means there's no Unix toolkits that you can use.

Maybe you cannot trap and deal with all of them if the underlying system does not let you, but that does not mean you should ignore those you can deal with!

Guess we'll all just have to write applications for ... does Windows or MacOSX handle this? I somehow doubt it.

It may depend on exactly where you hit it. Of course, any time when your application or library calls malloc it has the opportunity to do it!

2. if any of the files are unnamed or otherwise would require any of:
Don't allow them to be unnamed. You can create a name at the same time
as creating the otherwise unnamed document.

What if the act of creating a name is what finds the out-of-memory condition?

Then that is before the user has had a chance to enter any data in the unnamed document, so they won't be as upset of it pops up a dialogue saying "Out of memory, cannot create new document".

a) filename generation (would require string building)
Which can be done using a pre-allocated emergency buffer

what about memory that some lower-level stuff might require that you can't control? Perhaps you use a library for writing said file out to disk once you have the filename... even if only libc, you still need enough memory for an fopen() to succeed - you can't make /it/ grab from your emergency buffer.

I suggested other alternatives for if this is not possible.

Well, actually I suppose you could always replace the malloc symbol, but that still leaves you the problem of not necessarily knowing the memory requirements of lower-level library functions which means you can't accurately calculate how much emergency buffer you need.

b) file descriptor opening (which takes memory)
Which could have been opened when the document was created

this isn't doable if it's possible to have a large number of things opened. You may quickly run out of file descriptors.

I suggested other alternatives as well.

c) user-interaction (this one is right out)
Which can be avoided or use pre-created dialogue boxes that won't need
any more memory than they already have.

see above

Again, see above :-)

the application would certainly not be able to save the documents at
that point...
If it was planned for it could. I've come up with ways of dealing with
all of the problems.

not very good ways.

A single logging file to allow recovery on application restart is possible. It requires some work on synchronisation, but if designed in from the start is possible.

I maintain that a better way is simply to auto-save user's work... far far simpler to implement and far more reliable a solution unless you are able to provide 100% test coverage for your application.

Yes, regular auto-save is another way to protect users data, as long as you are not saving over the original.

Remember, we are talking about free software desktop applications written by volunteers, here, not programmers paid 150k/yr to write branches for each memory allocation they ever make in their application or library.

You may only be talking about free software written by volunteers, I am talking about all software whether written by volunteers or not. The open source community (some of it at least) wants to be taken as a serious alternative to closed source, so it should take the same effort to produce robust applications and libraries.

Especially provided that these programmers can at best assume that the authors of the libraries they are building on top of have also done their work of error checking every malloc and properly handling it and/or chaining it up to the caller.

Yes, you do need the libraries you are building on to pass up the errors, hence the comments about glib.

This is, in fact, where your whole argument falls apart. As a developer of GNOME desktop applications (hell, scratch that - of X11 based applications), you already KNOW that I cannot rely on glib or gtk+ (or Xlib) to gracefully handle all memory allocation errors... so I have no choice but to resort to my auto-save approach.

Or a form of logging as the user goes that you can use to recover (I've used non-gui applications that do this). Ov course, you have to make sure your auto-save and/or logging handle resources very carefully so that they do not loose the last good state if they run out of memory.

It's too easy to forget that large applications are generally built on top of code that other people wrote that you may or may not be able to read the source code for to verify that they properly handle all errors.

It's very easy to remember I find since I *am* building my SW on top of 3rd party libraries.

You could even make this argument for daemon authors - how many of you have actually read through all of the source code for the libc you build your applications on top of? None? Remember that and you might want to rethink not using the auto-save approach (in addition to error checking).

These days no one has time to check all the code they rely on (and often the source code is not available for everything). So yes, you rely to a degree on others doing the job right. As part of that you point out when it is done wrong!

Oh wait, I forgot that this whole thread is actually a pissing contest
more than anything else, so that people who don't actually write
desktop applications can feel superior to those who do.
I think it is more annoyance at application we might otherwise consider
using that would just throw away our hard work in situations some of us
do hit.

I've hit it as well, and yes, I'm also annoyed when it happens - but it is a problem that cannot be easily remedied by the application developers if the software stack somewhere underneath the code they've written is faulty (and I have personally run into Linux kernel and GNU libc bugs that have cause problems in applications I've written).

You can't deal with *everything* but we were talking about dealing with something where the libc *does* report a failure.

Since you cannot rely on error checking to catch all errors, it's best to have a fallback plan - which is auto-save. It's a lot less likely to fail and a lot more tolerant of buggy code (either in your application or below your stack).

I've no problem with autosave being part of the recovery strategy. As you say, it can help when there is nothing that can be done because the kernel has crashed.

In the real world, developers do not have the luxury (or desire, most of the time) to write applications from the ground up. They build on top of software that already exists.

The do have the luxury of choosing which libraries to build on and of reporting things which are a problem. You also have the luxury of not using malloc wrappers that don't allow you to do suitable recovery.
--
Flash Gordon
.

Relevant Pages

  • Re: xmalloc string functions
    ... require memory allocations depending on the way the system works. ... If the toolkit being used is not one of those, then it is irrelevant that some provide a means to do so, particularly if the "some" are not available for the platform being targeted. ... Not enough context for most real-world applications to recover at this point. ... At this point g_malloccalling abortbecomes a moot point, particularly if your auto-save code is robust against memory allocation errors. ...
    (comp.lang.c)
  • Re: Help with Enter and Leave Instructions
    ... >>> for Memory Accesses, ... > The only standard way to do it is via malloc. ... Uh, SBRK/BRK is a standard, documented system call. ... > I really don't understand the objection to using C libraries. ...
    (alt.lang.asm)
  • Re: [9fans] quantity vs. quality
    ... >>> Another example is using emalloc in libraries. ... > Well I wonder what people typically do when they can't malloc anymore ... > memory but need more... ...
    (comp.os.plan9)
  • Re: xmalloc string functions
    ... malloc() failure is some sort of emergency work and quit. ... But I claim that for 'regular' desktop applications that is quite ... not get the memory for that last comlpex calculation. ...
    (comp.lang.c)
  • Re: ProDOS Plus
    ... operating system was not considered worth the problems when it was just as easy to make the applications support 128k or more ram. ... your suggested P-code system could do something similar by running in the aux 64k memory range $D000...FFFF or perhaps the aux ram used by the P8 /ram driver code space. ... the OS could fit in *only* the space that ProDOS now occupies. ... if practicality were a concern we probably wouldn't be using old hardware. ...
    (comp.sys.apple2)