Re: Why is the memory doubled when writing data to files?

robert.corbett_at_sun.com
Date: 01/28/05

  • Next message: Robert Oeffner: "linker mistakes entry point in dll" 
    Date: 27 Jan 2005 23:26:10 -0800

    Shi Jin wrote:
    > Hi there,
    >
    > I observed something that I cannot explain.
    >
    > Is there any explanation why the memory would get doubled when
    writing data
    > to files?

    Yes. In the worst case, the size of a sequential unoformatted record
    cannot be determined until the entire output list has been evaluated.
    Most modern implementations of sequential unformatted I/O encode
    records using a header that indicates the size of the record, followed
    by the data, followed by a trailer that indicates the size of the
    record.
    When writing to a file that does not allow seeks, the header must be
    written before the data is written. Therefore, in the worst case, all
    of the data must be buffered before the record can be written.

    Example:

    Consider the WRITE statement

    WRITE (10) (F(I), I = 1, N), (G(I), I= 1, M)

    where the external unit 10 is connected to a file that does not
    allow seeks, and F is a function that modifies M. The processor
    must evaluate the N calls of the function F before it knows the
    size of the output record. The values returned by the calls
    must be saved so that they can be written after the header.

    The worst case almost never occurs in practice, but a complete
    implementation must allow for the possibility. Many Fortran processors
    will special case the cases where the size of the output record can be
    determined in advance. Others special case output to files that allow
    seeks. Either optimization eliminates the need to copy the data.

    The simple fact is that a modern Fortran implementation is big. An
    unbounded number of optimizations are possible. Vendors choose the
    optimizations they implement based on their best guesses of users'
    wishes. You should let the vendor know you would like his
    implementation
    to use less space for sequential unformatted output.

  • Next message: Robert Oeffner: "linker mistakes entry point in dll"

    Relevant Pages

    • Re: Why is the memory doubled when writing data to files?
      ... > records using a header that indicates the size of the record, ... a reference to Fis allowed to bugger up a subsequent ... > The simple fact is that a modern Fortran implementation is big. ... > unbounded number of optimizations are possible. ...
      (comp.lang.fortran)
    • Re: LCC Optimizations: Where is the IR to optimize?
      ... their book "Computer Architecture: A Quantitative Approach". ... I'll be writing a compiler for a 32-bit machine with dual-core ... I think peephole optimizations would really help. ... address IR to DLX machine code should be fairly trivial. ...
      (comp.compilers.lcc)
    • Re: Steve Pederson is a horses ass.
      ... street and beaten by an angry mob for writing.. ... :>I fixed your subject header for you. ... :That would be incredibly insightful if you had a clue about what you're ...
      (rec.sport.football.college)
    • Re: Writing File Header
      ... And also i need to know whether the header ... after i finish writing my actual data to a file. ... // Alessandro Angeli ... // a dot angeli at psynet dot net ...
      (microsoft.public.win32.programmer.directx.video)
    • Re: Synchronising read and write ops
      ... You won't usually get any help on file system filters in this newsgroup. ... I am writing a very basic on the fly encryption file system filter driver ... Reading and writing the header as part of non-cached and paging ... I was planning to use a driver managed queue of Irps so that I could ...
      (microsoft.public.development.device.drivers)