disk file reads slow down for file sizes greater than 2 GB

From: John S (shaeffer_at_bigzoo.net)
Date: 03/15/04

  • Next message: glen herrmannsfeldt: "Re: first time using direct access files: recl ??" 
    Date: 15 Mar 2004 11:40:17 -0800

    Have been developing an EM integral equation solver using sparse
    methods and a special block LU matrix factorization for really large
    method of moment problems. Have done problems with 100,000 unknowns
    in 8 hours.

    The platform is a 3 GHz PC, 2 GB memory, Win XP pro, CVF 6.6b, and a
    pair of 10,000 rpm disks set up in raid 0 (disk striping) for the
    scratch file.

    The scratch disk file is opened as direct access, RECL = 4096, and
    form = 'BINARY' (non standard).

    When the file is less than 2 GB, the (wall / cpu) time ratio for disk
    reads is unity (and fast). But when the file gets larger than 2 GB,
    the (wall/cpu) time ratio approaches 30 and the MBits/sec read rate
    drops significantly.

    The complex sub matrices are read as:

    read(iU, rec = iRecordStart, ioStat=iErr ) zArr

    where zArr is known to the subroutine as zArr(nRows,nCols) and
    iRecordStart is integer(8) and is the record number of the
    corresponding write for this zArr.

    The goal is to do problems which will have file sizes in excess of 20
    GB.

    Why should the disk IO slow when file sizes exceed 2 GB?

    John Shaffer

  • Next message: glen herrmannsfeldt: "Re: first time using direct access files: recl ??"

    Relevant Pages

    • Re: How to tell if CF card is faulty?
      ... disk sizes in si units instead of traditional computing units. ... but dd might not be counting partial records after an ... why am I getting different file sizes? ...
      (comp.os.linux.misc)
    • Re: NEW RESULTS ...
      ... > notifications so that it can keep disk, directory and file sizes ... So any file creation or deletion events may cause ... My test is running completely from memory, there is zero disk access. ...
      (borland.public.delphi.thirdpartytools.general)
    • Re: How to detect file size and file size on disk...
      ... for the size of all files, namely, file size, and file size on disk. ... I've got a script that will recurse through folders ... and files, and totals up the file sizes, but I can't figure out how to get ... using WSH/VBScript? ...
      (microsoft.public.scripting.wsh)
    • Re: Disk drive "Dissapears"
      ... >> error occuring when trying to write to a disk when there's still data ... >> caching becasue of the performance hit). ... >what file sizes? ...
      (microsoft.public.win2000.general)