Re: realloc but not copy

ivan.leben@xxxxxxxxx wrote:
Let's say I have a piece of allocated memory which I want to expand and
reuse if possible or allocate in a different part of RAM if resizing is
not possible, however, in the latter case I don't care about the old
data and I don't need to copy it to the new location. Is there a
standard function that could do that?

Short answer: no.

[...] As far as I understand the
description of the realloc() function, it _always_ copies the data to
the new location, even if that is not desired. Can I somehow just
_check_ if the expansion is possible and invoke a new malloc() myself
in case it is not? I'd like this operation to be as fast as possible
because the size of the memory I need is very large.

Well, if a realloc fails, then a malloc then copy, then free should
probably fail too (ignoring multitasking and weird heap
implementations.)

I had a similar issue in implementing the Better String Library. For
reasonably large allocations, the performance is usually going to be
limited by the copy. Now, if your data is anything like Bstrlib's the
problem is that the allocated buffer is usually larger than the amount
of valid data that its holding. But realloc doesn't know this, and so
will copy the entire buffer on realloc if a data move is required.

There is a possible approach here that will work on some heap designs.
You can first reallocate the allocation to the *smallest* possible
containing buffer (removing any unnecessary tail data) *then*
reallocate it to your intended final buffer size. As you can see, this
should avoid the unnecessarily large copies. One problem is that it
calls realloc() twice, and realloc is not guaranteed to keep the buffer
in the same spot even if you are making the size smaller. Another
problem is that realloc may not actually shrink the buffer when you
reallocate it with the smaller size. But if, instead, the heap behaves
in the ideal way that we all hope, then this will perform the
equivalent of a minimal copy on realloc so long as your valid data is
packed toward the beginning of your allocation.

In the Better String Library, I decided that I could not rely on good
heap design, and instead went with a probabilistic model. I basically
guessed that realloc()'s cause a shift 7 out of 8 times (which I
estimated by doing a random malloc/free/realloc test on a couple of
heap implementations) and so I just do the calculation of what the
expected cost of a straight realloc() versus a malloc() shortcopy and
free(), and I choose whichever strategy is like to be faster.

--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/

.

Relevant Pages

  • Re: Proposed addition of malloc_size_np()
    ... for future realloc(), etc. ... are other uses such as for having a debug malloc wrap the real one, ... We can store that information, ... getting the size of each allocation, I don't know how much to subtract ...
    (freebsd-arch)
  • Re: callocs, call by ref and function returning in C
    ... > memory allocation and freeing). ... What in case realloc returns NULL? ... clobbering buffer, so you can never free that memory again. ...
    (comp.programming)
  • Re: resizing an array, is there a better way?
    ... >> a large virtual memory space such that they can be trivially extended. ... > when realloc() is called with a size of 13; ... > allocation is done just after the existing block, ...
    (comp.lang.c)
  • Re: Proposed addition of malloc_size_np()
    ... Since malloc_sizeis defined on at least one platform to return the requested size, maybe a name like malloc_allocated_sizewould help avoid that confusion, and make it clear that the consumer is getting back a commitment and not a hint for future realloc(), etc. ... This is what I want malloc_usable_sizefor: garbage collection. ... By using dlsym, I can intercept all malloc/calloc/realloc/free calls and track current memory usage. ... However, unless there is a way of getting the size of each allocation, I don't know how much to subtract from the count for realloc/free calls. ...
    (freebsd-arch)
  • Re: Proposed addition of malloc_size_np()
    ... Since malloc_sizeis defined on at least one platform to return the requested size, maybe a name like malloc_allocated_sizewould help avoid that confusion, and make it clear that the consumer is getting back a commitment and not a hint for future realloc(), etc. ... Suppose that we are about to write to a string that we know is malloced, and we want to be sure that we aren't overflowing the allocated buffer. ... We can store that information, or we could just query the size. ... However, unless there is a way of getting the size of each allocation, I don't know how much to subtract from the count for realloc/free calls. ...
    (freebsd-arch)