Re: Appending files


"vertuas" <vertuas@xxxxxxxxxxxxxxxxx> wrote in message
news:r2cJi.32648$ka7.29851@xxxxxxxxxxxxxxxxxxxxxxx
Hi all,

I need my application to append some rather large files, (almost 2 gig
each).

I am currently opening each file into tfilestream objects, on then using
copyfrom function to append one stream to the end if another.

As i am appending all of the file, if there a faster was to do the append.
Get windows to chaneg to filesystem pointer, for example, to make 3 files
appear as one?

Not easily, files aren't contiguous. You could write a file system system
handler,
which would be a pain.
If managed in you own application a wrapper to manage filemapping without
copy is
simple,
some housekeeping attributes at the end of file ,
or a seperate housekeeping index file
or file metadata or associated streams using ntfs.
If you are using the files as arrays the overhead is 900% on a memory
access
but 100% better than windows virtual memory , with attendant thrashing,
over a few hundred meg and infinitely better for multi gig arrays.
The actual file lookup overhead is trivial.

If there a faster way to copy the stream than tfilestream.copyfrom? This
takes ages to copy all the data over.

Copyfrom is fairly efficient, the 61k buffer is a good size , fiddling with
it doesn't
give much improvement over other methods, any tweaks will be system
specific.

On the same drive on ata66 I get 30 meg/second to read then write with
copyfrom,
not bad for multiple seeks

With source and dest on seperate drives on the same cable it gives 47
meg/second, which is 2/3 of
66.6 MB/sec data rate for the transfer, the drives can in fact write to
files at full rate .

ie 23 meg read then written
2 gig append is 4 gig transferred, it would take around 87 seconds across
drives, 140 on same drive ( give or take a chunk)
scale appropriately for ata33, pata/133 or sata/150 meg per second
drive rates

if you aren't seeing that sort of speed and speedup across drives something
is wrong,
if you are then there isn't much leeway for speedup, the transfer can't go
very much faster
and copyfrom is doing well


.

Relevant Pages

  • Re: Appending files
    ... In not really seeing any difference between drives ... Would it help if i changed the windows VM setting from "programs" to "system ... I need my application to append some rather large files, ... Copyfrom is fairly efficient, the 61k buffer is a good size, fiddling ...
    (comp.lang.pascal.delphi.misc)
  • Re: [OT] organizing your scripts, with plenty of re-use
    ... a file in a package to somewhere else, and you don't tell the package ... some earlier incarnations too) WOULD search all active drives if no ... there was the APPEND command. ... c:\foo> edit readme.txt ...
    (comp.lang.python)
  • Re: hda becomes hdc. Can I still boot?
    ... But when I boot off the old ... than either that append line should be gone or changed. ... a CD-RW was the secondary master. ... AM>>use e2label to label the drives and make sure /etc/fstab uses the ...
    (comp.os.linux.misc)