Re: disk file reads slow down for file sizes greater than 2 GB

From: Roy Lewallen (w7el_at_eznec.com)
Date: 03/26/04 

Date: Thu, 25 Mar 2004 23:21:10 -0800

I might be able to offer some insight into your problem -- I think it's
the same one I've been chasing for several days now on a Windows XP
system. I'm also trying to write a fairly large file, several hundred
MB. I'm doing it in a program that's already using up a good fraction of
the machine's (1 GB) RAM, but I think the end result is the same as
you're seeing. Here's what happens --

When The WRITE statement occurs, the available RAM is reduced by the
amount of data written. Why Windows does this is a mystery to me, except
it's apparently buffering the written data and not returning the memory
after the write is complete. (Yes, I've tried FLUSH.) The program I
started with was writing it in chunks of about 32 kB. Every time another
chunk was written, the available RAM decreased by 32 dB, and it wasn't
given back. I'm monitoring the RAM use by calling Windows API function
GlobalMemoryStatus at various points and writing the result to a text file.

After all the available RAM is consumed by the write operations, memory
required by the program is furnished by Windows by means of paging it
from the virtual memory on the disk. This is a terribly slow and
inefficient process when a large amount of memory is being actively
used, and it's very likely the cause of your slowdown. There are
undoubtedly a number of ways to detect that this is happening, but I've
found a utility called FileMon to be handy -- it spots the writes to the
page file as they happen.

I'd really, REALLY like to figure out how to keep Windows from sucking
up the RAM every time I write a big file, or at least a way to get it
back. As it is, my program is hopelessly crippled after the first big
write operation, since from then on it's stuck with Windows virtual RAM
paging operation. Ironically, the file writing is part of a virtual
memory management scheme built into the program to avoid Windows getting
so low on RAM it goes into paging. But the writing itself is gobbling
the RAM. I've tried calling FLUSH after each write operation, different
sizes of chunks to write, and so forth, but no joy. I'm using CVF v.
6.6, and the file type is unformatted direct. I'd work out a way to deal
with binary or some other file type if I thought it would solve the
problem. Does anyone have any idea how to get around this? A different
compiler? Any ideas?

Thanks,

Roy Lewallen

Relevant Pages

  • Re: Windows uses 4 GB? Really?
    ... I bought 4 GB of RAM because I thought it would speed up my system a bit. ... Some info that is posted here as reference seems to suggest that it is a Windows issue, but others seem to suggest that it is also a hardware issue. ... "I have install total 4GB memory on my motherboard. ... On a SLI system, since PCI-Ex graphic cards will occupy around 256MB, another 256MB will be occupied after you install a 2nd PCI-Ex graphic card. ...
    (microsoft.public.windowsxp.general)
  • Re: Problem with M2N32 and 4x1GB RAM modules
    ... I put 2x1GB corsair DDR2 800 modules into it. ... The IDENTICAL RAM modules that I bought when i built it ... Every time i sped through the windows boot process and got to the ... Since the same 2 GB memory was really ...
    (alt.comp.periphs.mainboard.asus)
  • Re: if i have many windows or programs running, I cannot open anym
    ... Ken Blake - Microsoft MVP Windows: ... much memory. ... If you don't have enough RAM, ...
    (microsoft.public.windowsxp.general)
  • Re: running Linux with no swap space (but lots of RAM)
    ... Some amount of caching is necessary to achieve such I/O scheduling. ... The amount of memory allocated to ... Someone who thinks writing to /dev/null would result in lots of data being ... TB of RAM and fits in an ATX case using no more than 550 watts of power. ...
    (comp.os.linux.development.system)
  • Re: Kernel_data_inpage-error
    ... It is a Windows system file. ... sticks of Ram and not tested the ram yet. ... You can check RAM memory. ... the System and Application logs in Event Viewer for the last boot. ...
    (microsoft.public.windowsxp.general)