Re: Appending files

In not really seeing any difference between drives (both SATA150)

Would it help if i changed the windows VM setting from "programs" to "system
cache". of even disable the page page?





"Terry Russell" <trochilus@xxxxxxxxxxxxxxxxxxxxx> wrote in message
news:46f5d898$0$6277$afc38c87@xxxxxxxxxxxxxxxxxxxxxxx

"vertuas" <vertuas@xxxxxxxxxxxxxxxxx> wrote in message
news:r2cJi.32648$ka7.29851@xxxxxxxxxxxxxxxxxxxxxxx
Hi all,

I need my application to append some rather large files, (almost 2 gig
each).

I am currently opening each file into tfilestream objects, on then using
copyfrom function to append one stream to the end if another.

As i am appending all of the file, if there a faster was to do the
append. Get windows to chaneg to filesystem pointer, for example, to make
3 files appear as one?

Not easily, files aren't contiguous. You could write a file system system
handler,
which would be a pain.
If managed in you own application a wrapper to manage filemapping without
copy is
simple,
some housekeeping attributes at the end of file ,
or a seperate housekeeping index file
or file metadata or associated streams using ntfs.
If you are using the files as arrays the overhead is 900% on a memory
access
but 100% better than windows virtual memory , with attendant thrashing,
over a few hundred meg and infinitely better for multi gig arrays.
The actual file lookup overhead is trivial.

If there a faster way to copy the stream than tfilestream.copyfrom? This
takes ages to copy all the data over.

Copyfrom is fairly efficient, the 61k buffer is a good size , fiddling
with it doesn't
give much improvement over other methods, any tweaks will be system
specific.

On the same drive on ata66 I get 30 meg/second to read then write with
copyfrom,
not bad for multiple seeks

With source and dest on seperate drives on the same cable it gives 47
meg/second, which is 2/3 of
66.6 MB/sec data rate for the transfer, the drives can in fact write to
files at full rate .

ie 23 meg read then written
2 gig append is 4 gig transferred, it would take around 87 seconds across
drives, 140 on same drive ( give or take a chunk)
scale appropriately for ata33, pata/133 or sata/150 meg per second
drive rates

if you aren't seeing that sort of speed and speedup across drives
something is wrong,
if you are then there isn't much leeway for speedup, the transfer can't go
very much faster
and copyfrom is doing well




.

Relevant Pages

  • Re: Appending files
    ... I need my application to append some rather large files, (almost 2 gig ... copyfrom function to append one stream to the end if another. ... MB/sec data rate for the transfer, the drives can in fact write to ...
    (comp.lang.pascal.delphi.misc)
  • Make the Most of Your New PC
    ... mint Windows Vista computer this holiday ... the Workgroup name is exactly the same for all the computers in the ... that doesn't support USB 2.0) and low-capacity portable hard drives. ...
    (microsoft.public.windows.vista.general)
  • Re: Problem with XP scheduler?
    ... While it is possible to write a BIOS vector, ... With SATA drives, ... The reason for a lack of malware on Unix, Linux, MAC, etc. is that those ... With 90% of computers running Windows ...
    (microsoft.public.win32.programmer.kernel)
  • Re: Possible Sleeping Virus
    ... Remember that System Restore only tracks SYSTEM files - not yours. ... You should periodically defragment your hard drives as well as check them ... using Windows XP "prettifications". ... You should at least turn on the built in firewall. ...
    (microsoft.public.windowsxp.newusers)
  • Re: Can Only Boot with Boot CD in Drive
    ... install are pointing to the wrong drive, because when you installed the 2nd Windows, THAT became the C: ... When I insert the Recovery disk to re-install windows, it always refers to my drives backwards ... ... Place the drive on which you want to install Windows at the Master position on the Primary IDE controller. ... When you say "Place the drive on which you want to install Windows at the Master position on the Primary IDE controller", I'm not really sure I know how to do that. ...
    (microsoft.public.windowsxp.general)