Re: "".join(string_generator()) fails to be magic

On Thu, 11 Oct 2007 07:02:10 +0000, thebjorn wrote:

On Oct 11, 8:53 am, Marc 'BlackJack' Rintsch <bj_...@xxxxxxx> wrote:
Even if `str.join()` would not convert the generator into a list first,
you would have overallocation. You don't know the final string size
beforehand so intermediate strings must get moved around in memory while
concatenating. Worst case: all but the last string are already
concatenated and the last one does not fit into the allocated memory
anymore, so there is new memory allocates that can hold both strings ->
double amount of memory needed.

Perhaps realloc() could be used to avoid this? I'm guessing that's
what cStringIO does, although I'm too lazy to check (I don't have
source on this box). Perhaps a cStringIO.getvalue() implementation
that doesn't copy memory would solve the problem?

How could `realloc()` solve that problem? Doesn't `realloc()` copy the
memory too if the current memory block can't hold the new size!?

And `StringIO` has the very same problem, if the `getvalue()`
method doesn't copy you have to make copies while writing to the `StringIO`
object and the buffer is not large enough.

Ciao,
Marc 'BlackJack' Rintsch
.

Relevant Pages

  • Re: How to take in a string of any size?
    ... >the contents into the allocated memory. ... nobody said the string started at the beginning of the file. ... >number of calls to realloc() isn't going to be excessive). ...
    (comp.lang.c)
  • Re: A better BinaryConverter function in C++
    ... string ConvertToBinary(string Msg) ... You seem to believe these two lines copy the string into a newly ... allocate a block of memory and store its address in pCMsg pointer. ... and the allocated memory is leaked. ...
    (microsoft.public.vc.language)
  • Re: Fast string operations
    ... Looping: I thought looping over arrays in managed code was "slow" ... array handling and such. ... The problem with TrimHelper is that it always returns a new string instance. ... The customer perceives this as a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Discovering variable types...
    ... >- but I suppose MS expect us to use wrappers ... memory allocations for your variables from disk as well. ... >They most certainly are of fixed size, changing the size of a String ... >>me to keep buffer size and current postion right in the memory block. ...
    (comp.lang.pascal.delphi.misc)
  • Re: Optimize
    ... if you use it like me to get 'the next string ptr' in addition. ... | |mov ebx eax;dw aligned strings are faster ... | Would it be enough to touch each 16th bytes in the 64K memory area ... Cache-line size is fixed, my AMD got 64 bytes per line. ...
    (alt.lang.asm)