Re: Is Greenspun enough?

Ulrich Hobelmann <u.hobelmann@xxxxxx> writes:

> George Neuner wrote:
>>>>> mmap()ed files are also cached, no?
>>>> No. Mapped files are handled by the virtual memory system and all
>>>> modern systems DMA pages directly to/from disk with no buffering.
>>> Seriously? I thought every common OS would buffer/cache most pages
>>> in those files, so it doesn't need to access disk every time
>>> something is read or written (just with normal file reads/writes).
>> Seriously, you need to read up on MMUs, virtual memory and demand
>> paging systems.
>
> Been there, done that. What I (think I) know about modern OSes is
> that they cache everything from disk, I suppose by associating file
> blocks with in-memory-blocks. Surely I could be wrong, but I'd really
> like to know where you got your information that mmap()ed data isn't
> cached at all, because I don't believe it.

The caching sometimes happens at first access, and not at mmap time. It
depends on whether there is backing for the mmap, or if it is
"MAP_NORESERVE", which means that it is using "virtual swap" (which is
different than virtual memory). There is also copy-on-write options
which sometimes create new pages but which don't write them back out - this
allows many processes to map the same file without disturbing the original,
and yet without incurring the extra virtual page cost of a private mapping;
any pages which have not been written to can use the same actual page
across process boundaries.

>
> [... how mmap works; no problems here ...]
>> [IMO, the control offered by mmap() et al. is too coarse grained.
>> Windows file mapping API is easier to use and more flexible.
>> Application level control of the process's virtual memory is one of
>> the few areas where Windows really outshines Unix/Linux.]
>
> Hm, haven't ever looked at the Windows API. I suppose it's both more
> modern in some ways, and more bloated in other ways, than Unix.

I'll speak directly to George about this one; I do have a beef with
one of MS's arbitrary restrictions....

>>> I wager most libraries and maybe even executables on Unix systems
>>> are mmap()ed, so I really can't believe those files aren't ever
>>> cached, but reread from disk on every access...
>> See above.
>
> Which didn't in any way differ from my opinion or understanding of
> mmap(). AFAIK mmapped stuff is written to disk when you modify it,
> but until then it is cached in memory, until the VM system decides to
> drop the pages.

This is also different on different operating systems. We implemented
a memory-mapped file device in our simple-streams implementation which
works for all architectures; you just specify :mapped t as a keyword
pair to cl:open. It keeps mapped data in memory, but the decision as to
when to flush it is made by the mapping strategy; some systems flush
aggressively, and others require a flushing operation (on unix its an
msync() call, and on windows it's a FlushViewOfFile() call). You can see
the difference in behavior by looking at the file in a shell while
having it open in the lisp - if the file contents change when writing is
done, then the flushing strategy is aggressive.


--
Duane Rettig duane@xxxxxxxxx Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
.

Relevant Pages

  • Re: file access via read/write vs mmap() interface
    ... Memory] can't support any file access with mmap() whose size is more ... using any virtual memory and you are not using any RAM, ... pool is allocated and the disk page loaded into the RAM. ...
    (comp.unix.programmer)
  • Re: Code density and performance?
    ... programs for better virtual memory characteristics. ... 370 was initially announced with only real storage. ... packing in small real storage configurations. ... size was more than offset by the overhead of the smaller disk transfer ...
    (comp.arch)
  • Re: Virtual memory
    ... A common cause of low virtual memory message is that you have insufficient ... How large is your hard disk and how much free ... The Windows XP files alone can take around 3 gb. ...
    (microsoft.public.windowsxp.perform_maintain)
  • Re: Virtual memory leading to complete collapse of all programs
    ... Some programmes do not release memory ... reliance on virtual memory. ... You have insufficient free disk space on your hard disk. ... Charles Charles wrote: ...
    (microsoft.public.windowsxp.help_and_support)
  • Re: how much swap size did you take?
    ... virtual memory is used to contain executing processes. ... processes than can fit into real storage. ... to allocate a corresponding page on disk. ... This "duplicate" allocation scenario requires that amount ...
    (alt.os.linux.redhat)