Re: Problem with program (Pirko's RPSORT)

=Robert===================================
... must be able to eliminate dulicate records,
...
==========================================

This implies that a record ID has to be kept in memory somewhere to check
for duplicates, this ID could be a variable length string perhaps. With
400_000 records, you'll probably run out of memory pretty fast.

This could also be the source of the speed problem. As records grow, the
look-up hash table (if that is used) would grow as well (probably
resulting in large chunks of mem->mem copies), plus there are concerns of
swapdisk memory usage.

-kain


.

Relevant Pages

  • Re: Fast string operations
    ... Looping: I thought looping over arrays in managed code was "slow" ... array handling and such. ... The problem with TrimHelper is that it always returns a new string instance. ... The customer perceives this as a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Discovering variable types...
    ... >- but I suppose MS expect us to use wrappers ... memory allocations for your variables from disk as well. ... >They most certainly are of fixed size, changing the size of a String ... >>me to keep buffer size and current postion right in the memory block. ...
    (comp.lang.pascal.delphi.misc)
  • Re: Optimize
    ... if you use it like me to get 'the next string ptr' in addition. ... | |mov ebx eax;dw aligned strings are faster ... | Would it be enough to touch each 16th bytes in the 64K memory area ... Cache-line size is fixed, my AMD got 64 bytes per line. ...
    (alt.lang.asm)
  • Memory Question
    ... I have wrote a helper to my program which monitors object count, memory utilization and a threshold of 10% to determine how many objects are sticking around and how many are being garbage collected. ... occured right before the original String count). ... def print_threshold_breakers hsh1, hsh2, threshold ... putsf 'Building mem usage', mem_usage ...
    (comp.lang.ruby)
  • Re: Theodore Adorno, a prophet of data systems design
    ... > memory leaks, overruns etc., and I can even prove this without knowing ... possible to design a language in which buffer overruns are impossible. ... C makes the allocation of fixed-size arrays possible because there are some ... design and implement your own string model. ...
    (comp.programming)