Re: xmalloc string functions

Jeffrey Stedfast wrote, On 03/02/08 19:35:
On Sun, 03 Feb 2008 15:54:25 +0000, Flash Gordon wrote:
Let's also not forget that the act of /showing/ the dialog may, in
fact, require memory allocations depending on the way the system works.
So you have to look to see what workarounds there are for this. Once
you've done that one, you can reuse the solution so even if it takes a
week amortise thatover all of the applications you will write!

but later you say there are no one-size-fits-all solutions? :)

That particular part of it is a common element that can be reused where appropriate :-)

For example, requesting that a dialog be shown may not actually show
the dialog immediately... it might only queue the operation for the
next rendering pass.
On at least some there are ways to get it rendered immediately. I know
I've done that in the past. Once you have solved it for one application
reuse the solution for others.

If the toolkit being used is not one of those, then it is irrelevant that some provide a means to do so, particularly if the "some" are not available for the platform being targeted.

You can always go straight to the X API or the Windows API or whatever for the emergency code.

Said rendering pass may require more allocations, but at this point
it's too late to simply unwind the stack to the point where you
requested the show(). Since no widget toolkit I know of has a way of
notifying the application of said error, what is it to do?
So you are saying all widget toolkits are badly designed. This is
possible.

I never said "badly designed", though I would agree "sub optimal in an ideal world". There's a difference (to me, at least).

There is a whole range of design quality, and it is not even a line more of a space with several dimensions. Perhaps I should have said "badly designed in this respect" since in other respects they might approach perfection.

However, it has enough memory to do it.

How can you assert this?

There was meant to be an if in there. As in, "However, it it has enough memory to do it.

If you try and fail
you are no worse off, if you try and succeed you are better off.

I'll agree with that, and wherever I use malloc() directly (or g_try_malloc(), I do write error handling - which may or may not include attempting popping up an error dialog depending on the situation).

Well, that is good :-)

For Gtk+, you actually do have an option... GLib uses a vtable for
malloc/ realloc/calloc/free that you can initialize with your own
routines at init.
See, there *are* ways to deal with it!

Right, but as you mentioned was a problem for xmalloc(), we have the same problem here. Not enough context for most real-world applications to recover at this point.

OK, yes, but if you override the malloc/calloc/free when running your emergency recovery code you can use your pre-allocated block for the allocations that Gtk+ does so avoiding further out-of-memory problems :-)

You could potentially do your own NULL-check there so that you can be
pre- warned about memory allocation errors coming up, but it'll lack
context (who tried to allocate this memory? for what purpose?), but I
suppose if you had everything pre-allocated, ready to go - you could
call some global prepare_for_abort() function that could perhaps
iterate thru all of your unsaved files and save them quick before the
abort() call in the g_malloc() wrapper. This wouldn't allow you to pop
up any dialogs, however, because at this point its too late.
No, it's not too late, since as you save documents you can free up the
memory they used giving you the memory to pop up dialogues! Or, as
previously mentioned, have stuff pre-allocated.

Easier said than done, not that it /can't/ be done - but one could easily argue that this is more effort than it is worth, and unless you are able to test your failure cases thoroughly, not even reliable.

As with all of life there are tradeoffs to be had.

It is /more/ reliable to routinely auto-save the user's work (as you mentioned elsewhere, to a file other than the original) because it is much easier to warn users about problems (potential or no) and certainly easier to implement recovery should the application crash due to uncontrollable (kernel crash, power outage, etc) error conditions on the next application startup.

I agreed that this can be part of your recovery strategy.

Depending on the document, one could write the application such that any button click (or whatever) would cause an auto-save in addition to some timeout, thus reducing the likelihood of there being any unsaved changes at any given point in time.

Or do what some editors I used to use did and literally save all changes as the user went along. This was saving in to a recovery file not over the original, and one recovery file would cover all the work done to all files in that session. The best was the one where you could literally sit watching it retype everything...

Since you obviously need this auto-save functionality in place if you are serious about protecting the user's data at all costs anyway, then it becomes no longer necessary to chain malloc() failures up your call stack in order to use static emergency buffers.

At this point g_malloc() calling abort() becomes a moot point, particularly if your auto-save code is robust against memory allocation errors (keeping a small subsection of code bug free and robust against all possible error conditions is a lot easier and less costly in developer time than it is to do that for an application several million lines of code long).

You should *still* do your damnedest to pop up a dialogue box so the user knows the crash is due to out-of-memory! Also you want the program to exit using exit not abort otherwise files might not be flushed before being closed. Especially important of you use the method I just suggested of logging everything as you go along rather than an autosave.

, and then let you copy the previous twenty-digit result into
another document for safe keeping.
ah, but that also requires a clipboard memory buffer be allocated...
but you have no memory left ;-)
Have it pre-created, then if you need a larger buffer for the next
step and you can't enlarge it you only loose that last step.
this one indeed is likely an easier and more reliable method for this
particular instance, but not all desktop applications can go around
using this type of approach.

For example, would it be a good idea for an email application to set
aside this clipboard buffer? :)

I think we'd both agree the answer is no.
In that instance you would use a different solution, probably saving the
email in a draft folder or something like that. People have already said
there is not a one-size-fits-all solution!

Hey, guess what? Evolution did this using an auto-save approach and it used g_malloc() in much of the application code.

Well, I didn't like evolution anyway, I found it was hogging too many resources on this machine for not enough benefit ;-)

Different approaches, same end result. Oh, sure, maybe in your ideal case, the application exits from main() with a 'return 0;' as opposed to an exit() call (or abort()), but that is irrelevant.

[snip]
Drop that background print-job or spell check that is consuming memory
for your desktop app.
That means you'd have to have that print job context or spell-check
context global somewhere, or have some way of getting it from a lot of
different locations...
You probably need mechanisms to signal the spell checker and print
process anyway to cope with the user choosing to abort them.

This is true, however you still need context information in order to do so. I never said that the application wouldn't have the ability to cancel the spell checker or printing, but in order to do so you need context. If you are in a function being called asyncronously from somewhere which might not even be your code which may not pass up your particular error condition, then you are pretty much screwed unless your contexts are all globally accessible.

Such things are likely to be separate threads, so you just send them an appropriate signal (not necessarily in the C sense of the word) and give them a chance to terminate. Then you only need to know how to signal them which is information you could make globally available (maybe by exposing a "kill print job" API that keeps static state).

While this may suggest the application (or the libs it depends on) is poorly designed (or at least not suitably designed), the argument does little to solve the actual problem at hand.

There are ways.

In the real world of end-user software development (e.g. not software written for space ships or other areas where human lives are on the line)

Um, those *are* real world situations! ;-)

where the application's design is based on incomplete specifications (as in they tend to change mid-development)

That applies to SW written for space ships and SW where lives are at stake as well. Although such things do tend to be better documented and change controlled.

in combination with insufficient allotted time,

That certainly applies in the defence industry where I spent 15 years :-/

designing the perfect solution is downright impossible,

Well, nothing is ever perfect!

and so it is, unfortunately, not all too uncommon for the application's design to be insufficient for every possible error condition.

If this is new to you, then you've never written real-world software and I would appreciate having your pitty... because I, too, would love to live in Ideal World where I have sufficient time and specifications to use in order to come up with a proper design before I'm forced to begin implementation :)

In the real world when asked to reduce an estimate I've been known to sit and think for a few minutes and then say...

"Well, in my opinion you don't really need these pieces of functionality since the things they are there for can be said to be covered by these other things, and so by removing these requirements you remove this amount of work."

I didn't get any complaints about my response either. I was the expert and they had to accept my word on it.

[snip]
doable if you don't want to give any specifics... daemons are often not
very user-friendly in their error reporting... depending on the daemon,
That it depends on the daemon shows that is is possible, or it would be
a simple case that none do.

it might be as simple as an integer error code or as forthcoming as a
string from strerror(), but rarely do they report something that the
user is able to understand.
Normally they report something the system administrator is able to
understand. At least, most that I use do.

Key word: most :)

The others *could* and not doing so is another aspect of poor quality. Also a reason I would consider switching to an alternative.

Sure, "out of memory, cannot perform that operation" may work for
simple applications where only 1 thing at a time is ever going on, but
if the application happens to be doing many things at once the user
will want to know /which/ operation could not be completed because
memory was unavailable?
Yes, which is why things like xmalloc are a problem, because they do not
have that context.

Agreed in so much as they are not an ideal solution to the failing malloc problem :)

Well, that is a start :-)

They are, however, /a/ solution to the problem and might, in some situations, be more than ample.

Better than dereferencing a null pointer. An application specific alternative is better than a general purpose one as it can do application-specific clean-up that a generic wrapper can't.

Trust me, this is the case... applications I've worked on have actually
had these sorts of complaints filed against them. It's funny, because
all the user testing I've seen indicates that users never read the
dialogs anyway ;-)
Depends. I've had a report come back to me (via at least a couple of
layers of intermediaries) that had exactly the information that the
"dialogue" provided (it was not a GUI application).

As have I, in my gui applications even.

So yours aren't all bad ;-)

- wait until memory becomes available
With an appropriate pre-created dialogue you can do that on a GUI
application as well.
see above.
Again, see above ;-)

On the other hand, say, a word processor application, if the user
requests some sort of action and a malloc() fails for 12 bytes, what
is it supposed to do?
Any of the above.
Easier said than done, I'm afraid...
Worth the effort though.

In an ideal world, perhaps. If you've already got an auto-save feature then it is not necessarily worth the extra effort.

I would agree that it /is/ worth the effort in the case where the failing malloc() call is in the auto-save code, however :)

So you aren't as bad as some :-)

If the documents the user has open have already-opened file
descriptors, the app might be able to save them before going down -
but:
That is easy to arrange.

1. it certainly doesn't have the option of displaying an error
dialog.
Yes it does if it pre-creates it during application start-up.
see above, although I suppose if you really wanted to, you could make
an exception for the "out of memory" dialog case as opposed to other
error dialogs your application might use.
You have to take extra care with any out-of-resource error to ensure you
can report it without the resource in question.

First... I wonder if there are any widget toolkits that don't already
abort() (or similar) when they run out of memory or in any other
conditions without giving my calling code a chance to handle it?
Lotus Notes has given me an out-of-memory dialogue. I'll leave you to
draw your own conclusions from this.

I would conclude, that, like some parts of Evolution, if it is unable to allocate resources for some non-critical data structure(s), that it is able to report the "out of memory" issue to the user.

I seem to recall you claiming VMWare reported "out of memory" conditions to the user as well, but as Ben Pfaff noted, VMWare uses xmalloc-like wrappers as well.

It was some king of out-of-resource, and as it was less than a month ago I've not had it happen again to check exactly what the error message is ;-)

As someone else mentioned, X already has this limitation... so right
there, that means there's no Unix toolkits that you can use.
Maybe you cannot trap and deal with all of them if the underlying system
does not let you, but that does not mean you should ignore those you can
deal with!

Never said otherwise!

OK :-)

Guess we'll
all just have to write applications for ... does Windows or MacOSX
handle this? I somehow doubt it.
It may depend on exactly where you hit it. Of course, any time when your
application or library calls malloc it has the opportunity to do it!

Sure, the same goes for any application written on top of glib!

So don't write it on top of glib ;-)

(Not the case if you use g_malloc() of course, but you are hardly forced to use only g_malloc() just because you link with glib).

Only if none of the other bits you call use g_malloc.

2. if any of the files are unnamed or otherwise would require any of:
Don't allow them to be unnamed. You can create a name at the same time
as creating the otherwise unnamed document.
What if the act of creating a name is what finds the out-of-memory
condition?
Then that is before the user has had a chance to enter any data in the
unnamed document, so they won't be as upset of it pops up a dialogue
saying "Out of memory, cannot create new document".

Not necessarily, but I will agree that this is /likely/ the case.

If you attempt to allocate the space on creating the document it is *definitely* the case that the user will not have had a chance to do anything with it.

A single logging file to allow recovery on application restart is
possible. It requires some work on synchronisation, but if designed in
from the start is possible.

I've used this approach for some simpler applications.

Auto-save is actually not that much different to this.

The logging has the advantage of never having to open a new file to it after application start up, and generally all the resources needed to write to a file are allocated when you open it :-)

<snip>

I am
talking about all software whether written by volunteers or not. The
open source community (some of it at least) wants to be taken as a
serious alternative to closed source, so it should take the same effort
to produce robust applications and libraries.

Amusing to me is that none of these developers are writing GUI apps or libs afaict ;-)

Well, I do it occasionally.

It's not hard to find command-line programs and/or general purpose libs that /are/ robust, like Ben Pfaff's AVL tree library for example, but none of the ones I know of for writing GUI applications are of this quality.

If I wanted to write an application that would meet your ideal criteria, I'd have to write my application from the ground up, including the widget toolkit. This is not only impractical from the development standpoint, but also from the user's perspective where the application does not look like any of his other applications. It would also not be able to share much with the other applications running on the user's desktop and so would use a lot more resources than a Good Enough solution.

Of course, you could work on improving the toolkits to deal with the problem ;-)

Equally, if I had the time I could.

Especially provided that these programmers can at best assume that the
authors of the libraries they are building on top of have also done
their work of error checking every malloc and properly handling it
and/or chaining it up to the caller.
Yes, you do need the libraries you are building on to pass up the
errors, hence the comments about glib.

Anyone using glib stand-alone should probably reconsider, especially if they are writing "mission critical" applications.

Most people, however, use glib via Gtk+ - and being that is the /only/ practical widget toolkit available to C software developers for Unix, you can't easily write off glib altogether.

There where widget sets around before the start of the GNOME project and they have been used successfully.

I honestly would not be surprised if the other major contender in the widget toolkit space (being Qt) had similar problems wrt memory allocation failure conditions, but even if it did, you wouldn't be able to write the application in C afaik (you'd have to switch to c++).

Or write the GUI front end in C++ and the rest in C. Or do the GUI front end in C# or Java or...

This is, in fact, where your whole argument falls apart. As a developer
of GNOME desktop applications (hell, scratch that - of X11 based
applications), you already KNOW that I cannot rely on glib or gtk+ (or
Xlib) to gracefully handle all memory allocation errors... so I have no
choice but to resort to my auto-save approach.
Or a form of logging as the user goes that you can use to recover (I've
used non-gui applications that do this). Ov course, you have to make
sure your auto-save and/or logging handle resources very carefully so
that they do not loose the last good state if they run out of memory.

Yes, this is what I've been saying.

So you don't call glib functions from within that code :-)

It's too easy to forget that large applications are generally built on
top of code that other people wrote that you may or may not be able to
read the source code for to verify that they properly handle all
errors.
It's very easy to remember I find since I *am* building my SW on top of
3rd party libraries.

You could even make this argument for daemon authors - how many of you
have actually read through all of the source code for the libc you
build your applications on top of? None? Remember that and you might
want to rethink not using the auto-save approach (in addition to error
checking).
These days no one has time to check all the code they rely on (and often
the source code is not available for everything). So yes, you rely to a
degree on others doing the job right.

Glad we agree so far.

:-)

As part of that you point out when
it is done wrong!

Well, discussing it here isn't going to get the problem solved. If you truly feel that strongly about it, then you should either fix the problem (free software, afterall) or at the very least submit a bug report! ;-)

Oh that I had the time to bug-hunt free SW. Unfortunately I only occasionally have time to bug-hunt the free libraries I use in the SW my company sells and definitely don't have the time for other free SW. When I do have the time I do submit bug reports and/or bug fixes.

Oh wait, I forgot that this whole thread is actually a pissing
contest more than anything else, so that people who don't actually
write desktop applications can feel superior to those who do.
I think it is more annoyance at application we might otherwise
consider using that would just throw away our hard work in situations
some of us do hit.
I've hit it as well, and yes, I'm also annoyed when it happens - but it
is a problem that cannot be easily remedied by the application
developers if the software stack somewhere underneath the code they've
written is faulty (and I have personally run into Linux kernel and GNU
libc bugs that have cause problems in applications I've written).
You can't deal with *everything* but we were talking about dealing with
something where the libc *does* report a failure.

You /assume/ that all code paths properly handle OOM conditions internally and propagate them back up the call stack. But libc is still only implemented by humans last I checked, so there is a possibility of bugs.

Yes, and therefore it is pretty much guaranteed that all libc implementations have bugs somewhere. I would, however, expect the memory allocation functions to be amongst the most hammered and hence least buggy parts of the library.

That's a pretty hefty assumption that you CANNOT rely on for mission critical user data (since that's what your whole argument revolves around in the g_malloc()-is-evil argument).

It is an evil that could have been avoided.

Because of this possibility, you MUST implement a safety net - aka auto-
save. Once you have auto-save in place and properly written to handle every conceivable error condition that /it/ may encounter (OOM being one), then the value gained by using malloc() over g_malloc() in the remaining areas of the code begins to rapidly lose their practical value (if the goal is simply to make sure the user's data is saved before exiting).

Wouldn't you agree?

Personally I would still be likely to switch to a (possibly commercial) application if I hit crashes on out-of-memory if they did not inform me they that they were shutting down due to out-of-memory. If they let me know I am a bit more tolerant as long as the recovery on restart gets back my work.

<snip>

In the real world, developers do not have the luxury (or desire, most
of the time) to write applications from the ground up. They build on
top of software that already exists.
The do have the luxury of choosing which libraries to build on and of
reporting things which are a problem. You also have the luxury of not
using malloc wrappers that don't allow you to do suitable recovery.

Not always.

OK, *I* always have the choice, even if sometimes it involves changing job. Not had to go that far yet though :-)

For bonus reading, you might check out Richard Gabriel's paper on Worse Is Better.

GLib's g_malloc() must be "good enough" because more and more Gtk+ applications keep popping up like wildfire just as C overtook LISP due to the Worse Is Better rule.

What becomes popular is not always determined by what is good. There are many examples of the worse solution winning out.

Personally I would seriously consider using a language that supports exception handling if error propagation was going to prove too hard to be "worth the effort". This would mean switching from C, but I consider a language to be merely a tool so switching is not a problem.

As to LISP, I never liked it, but that is not an argument for here.
--
Flash Gordon
.