Re: Efficient scather-gather-copy

Jerry Coffin <spamtrap@xxxxxxxxxx> writes:

I'm in the unlucky situation to convert mails from the "\n"-lineendings
to "\r\n"-ones.
I'm doing the rest of the program in C, but I hope to be able to use
some special CPU-capatibilities via inline-assembly to make this
_really_ fast.
If I'm not mistaken, I have to count the lines, alloc
sizeof(mail)+num_of_lines(mail), copy each line seperately with a space
between them to the buffer and replace the "\n" through "\r\n".
Unless the buffer is really huge, I'd cheat: instead of counting the
lines, just allocate the second buffer twice the size of the first
(big enough for the largest possible output size) and then do a
byte-wise copy, adding \r where appropriate.

This would be an viable idea. We're limiting the size of mails to be
scanned to 100k or 64k anyway. If my "copy-plan"-idea can't be backed up
by some CPU feature, it'd be the fastest approach I guess.

To gain much, you need to improve bandwidth to main memory. Under the
right circumstances, movntq can help in that regard -- but at least to
me, these don't look like the right circumstances. movntq is useful
for writing 64 bits of data to a 64-bit aligned memory location. Since
here you're expanding the output data arbitrarily, maintaining 64-bit
aligned writes is problematic.

I guess so too. :/

That leaves very little opportunity for hand-written assembly to gain
you much. I am a little puzzled at how you'd receive enough email fast
enough for this to matter though. Email typically implies networks,
and networks fast enough to stress the CPU are rare -- most of us
don't have our very own OC-192 lines (and my code/machine would
theoretically about keep up with that, except that its dual GbE
connections are nowhere close to fast enough).

It's rather a problem of parallely (does this word exist? ;)) incoming
mails. We're talking about a medium-sized ISP here, so the mails come
maybe slowly in, but many of them at once.

Thanks for the ideas!
-hs

.