Re: realloc, copy and VM




In article <BEE253C2.744F%jean-claude.arbaut@xxxxxxxxxxx>, Jean-Claude Arbaut <jean-claude.arbaut@xxxxxxxxxxx> writes:
>
> When you realloc for more size, it may be necessary to allocate a new block,
> copy memory and deallocate the old, for example if there is not enough free
> space after the original block (maybe the only example ?).

There are other possible cases. Not all allocators simply grab the
next available suitable free chunk. Some group objects into size
classes, for example, and your realloc may change which class the
object fits in.

The allocator scheme is not specified by the standard. It can be
*anything*, as long as the required semantics are preserved.

> But, is it really
> necessary to _copy_ bytes ? When in a virtual memory environment (off-topic
> here, be I still think the question is worth), you don't access directly
> physical memory, but allocated pages, so would it be possible on some
> implementations, to just change page allocation ?

That *is* extending the original block.

Take the typical case: the implementation uses a linear virtual
address space for all allocated objects, and the contents of C object
pointers are addresses as used by the virtual memory manager. If an
object occupies the page at address n*pagesize, and you attempt to
extend it with realloc, then:

- There may be no page mapped at address (n+1)*pagesize, and the
implementation could request that the VMM map a new page at that
address. Aside from performing its internal housekeeping, the
realloc is done; no copying needs to take place.

- There may be a page mapped at address (n+1)*pagesize. If it's part
of some other object, then obviously the implementation cannot steal
that address for the object being resized, because pointers into the
other object will refer to that portion of the virtual address space.
The realloc'd object will have to be moved.

- Some other case may apply (eg the implementation is unable to
secure additional memory).

> Some kind of immobile
> trip. It could permit optimizations even when not knowing beeforehand the
> exact amount of memory needed.

The malloc implementations I'm familiar with are already capable of
doing this. (Actually, for many of them it happens automatically
in many cases, via lazy allocation and copy-on-write.)

In short: yes, it's a good idea, for implementations where it's
appropriate; but it's one they typically already use.

On another off-topic note: in a large (eg 64-bit) virtual address
space, it's usually possible to space objects sparsely, so that they
nearly always can be extended this way. That's one of the advantages
of a large address space.

--
Michael Wojcik michael.wojcik@xxxxxxxxxxxxxx

Sure we're tossing out fluff, but tell me, where does anyone deal in words
with substance? -- Haruki Murakami (trans Alfred Birnbaum)
.



Relevant Pages

  • Re: How to release heap memory that is marked as free
    ... As I said, fragmentation is a very serious problem, and one of the most serious problems ... my allocator was accused of using massive amounts of memory. ... I'm going to have to re-think the memory allocation that I'm ... process's 'working set'. ...
    (microsoft.public.vc.mfc)
  • Re: [PATCH 00/28] Swap over NFS -v16
    ... memory they can consume. ... So we need the extra (skb) ... included in the reserve? ... if the allocation had to dip into emergency reserves, ...
    (Linux-Kernel)
  • Re: Memory leak with CAsyncSocket::Create
    ... read my essay on how storage allocators work. ... Create method is consuming system memory that is not released back to ... The memory consumption is either shown as "Mem Usage" on the Task ... many levels of allocation going ...
    (microsoft.public.vc.mfc)
  • Re: malloc
    ... If I use both the calls ie. malloc and realloc together, how the allocated memory will be ... Again, an old pointer value, valid for the first allocation of p, will ...
    (comp.lang.c)
  • Re: OT: C++ overloading operators
    ... dynamic allocation, no matter how many "clever tricks" are used... ... though there's enough memory in the system, ... all these "flexible data types" map into CPU command ... The computing environment is completely ...
    (comp.dsp)