Re: Efficient scather-gather-copy

Hynek Schlawack <spamtrap@xxxxxxxxxx> wrote:

It's UNIX and yet filtering is simply too slow. I'd rather use memcpy()
than external programs.

I'd just like to emphasize the "rather" part, because this is likely to be
a personal preference issue, rather than a proven performance problem.
Remember that Unix filters in a pipeline do not pass through the file
system.

Well, I'm getting it as a buffer and anything that remotely could
involve access to file systems is out of question.

Imagine dozens of mails per _second_ that want to be
processed. Probably, memcpy() might be fast enough on a decent machine,
however it would definitely be a bottleneck.

Have you done the math? The typical e-mail is about 20k bytes long. Dozens
of mails per second would make it hundreds of kilobytes per second, perhaps
up to a megabyte per second.

That's trivial for a file system, and nothing at all for memcpy.

However, the message cannot possibly increase by more than 2x. So, as a
worst case:

char * WindowizeMessage( char * InputBuffer, unsigned long Length )
{
char * newbfr = malloc( Length * 2 );
char * bfr = newbfr;
while( *InputBuffer )
{
if( *InputBuffer == '\n' )
*bfr++ = '\r';
*bfr++ = *InputBuffer++;
}
*bfr = 0;
return realloc( newbfr, bfr-newbfr );
}

I doubt you can do significantly better than that.

Only, that "pretty fast" ain't "fast enough" in this case. ;)

Can you prove that? As I said, have you actually DONE THE MATH? An
enormous amount of effort is wasted on unnecessary optimization.

And may I ask why you need to do this? Every major mail server on the
planet handles either \n or \r\n.
--
- Tim Roberts, timr@xxxxxxxxx
Providenza & Boekelheide, Inc.

.

Relevant Pages

  • Re: Sizeof query
    ... object can be treated as an array of char, at least not in so many ... � � Except for bit-fields, ... Note that it talks about *copying* the value into an array of unsigned ... The key point in putting together the inference is the memcpy() ...
    (comp.lang.c)
  • Re: Sizeof query
    ... object can be treated as an array of char, at least not in so many ... Note that it talks about *copying* the value into an array of unsigned ... The key point in putting together the inference is the memcpy() ... explicitly stated as being the only way of doing it, implies that ...
    (comp.lang.c)
  • Re: Alignment problem (maybe)
    ... so memcpy should work correctly... ... dword ... to the data structure was pointing within a memory buffer, ... So, when I was writing the values char by char, there were no problems ...
    (comp.os.linux.development.apps)
  • Re: help understanding character arrays
    ... the standard functions like 'memcpy' and 'memmove' live. ... what you get when the object 'mychar' "decays" to a pointer to its ... first element: a pointer to char. ... The address of the array is the same as the address of its first ...
    (comp.lang.c)
  • Re: Alignment problem (maybe)
    ... so memcpy should work correctly... ... dword ... to the data structure was pointing within a memory buffer, ... So, when I was writing the values char by char, there were no problems ...
    (comp.os.linux.development.apps)