Re: Ranting about JVM's default memory limits...

On Sun, 03 Aug 2008 10:19:43 -0700, Mark Space <markspace@xxxxxxxxxxxxxx> wrote:

Daniele Futtorovic wrote:

Come to think about this issue, I'm not quite sure I see how this is
that much of a burden... Of course, I understand it's more convenient
not to have to bother about Xmx. But then again convenience isn't always
the most important thing, and wouldn't having to bother about Xmx


So I need a technical question answered here. How does demand-paged virtual memory work?

No answer that would fit in a newsgroup-sized message could suitably address that question. But the short answer is: the OS reserves a range in the virtual address space for the _process_, and when the process tries to access a memory location within that range and it's not already mapped to a physical memory location, the hardware detects that, allowing the OS to respond and map that memory location to some physical memory location.

There are a lot of complexities about _how_ that actually happens, including things like "if all the physical memory has already been mapped, how does the OS decide what current mapping is removed to make room for the new one?", "what's the difference between a page that hasn't been touched by the process yet and one that has?", "when a current mapping is removed, where does the data go?", and of course "just how does the hardware support all of this in the first place?" But the basic idea is that virtual address pages aren't actually mapped to physical memory locations until a process "demands" them.

Oddly enough, I'm not sure any of the above actually addresses the question you _really_ seem to be asking, at least based on what appears to be an elaboration of your question later in your post:

When my Java app asks for 1.6 G of virtual memory on a Windows system, does it actually hog all the virtual memory available for all applications?

That depends on whether the "virtual memory available for all applications" was actually as little as 1.6GB (or so).

But (very important!): don't confuse the virtual _address_ space allowed for a single process with the virtual _memory_ available for the entire computer. The former is limited (in the default case) to 2GB for each process. The latter is limited mostly only by your available disk space. With inexpensive hard drives in the TB range now, you could theoretically have hundreds, if not thousands, of gigabytes of virtual memory.

(Caveat: I haven't been keeping up with the exact OS limitations versus disk space. In truth, there must be some limit to how much virtual memory Windows can manage, and I suppose it's possible that with TB drives, we've finally reached that limit. The important thing is that whatever the limit is, assuming it exists at all, it has _nothing_ to do with the 2GB-per-process limit being discussed in this thread).

If your disk space was indeed constrained, limiting the total virtual memory size to 1.6GB, then yes, a process allocating 1.6GB could at that point exhaust all of the virtual memory available across all applications. Fortunately, that's not usually the case. Typically, if you're running an application that needs that much memory, you've got a lot more free space on your hard drive than that.

Of course, there are other issues. In particular, 1.6GB is still a lot of memory, and if your computer has a lot of other applications running, you'll start to run into problems with each process needing access to their own allocated memory. The OS will alternately load and unload data used by a given process as it needs it, and then as another process needs its own. When this happens, most of the time is spent moving data back and forth between physical memory and the disk, at which point performance slows to a crawl.

But, assuming you have enough physical memory and few enough other applications to avoid that, a 1.6GB allocation in one process has basically no effect on any other process.

Or does each application get it's own 1.6 G byte virtual memory space?

Each process gets its own 2GB virtual _address_ space.

Pete
.

Relevant Pages

  • Re: [Lit.] Buffer overruns
    ... > floating point support or a memory expansion option. ... had virtual memory support grafted on. ... > where the modified instruction was fetched from. ... vis-a-vis the official coporate strategic operating system TSS/360. ...
    (sci.crypt)
  • Re: virtual memory
    ... you can have virtual memory ... that software transparent swapping requires virtual memory. ... (since swapping it into physical memory is all or nothing). ...
    (comp.arch.embedded)
  • Re: Why is VS 2005 so slow?
    ... You are confusing physical memory with virtual memory. ... Note also that the number of apps you can run at the same time doesn't change. ...
    (microsoft.public.vc.mfc)
  • Re: scf /scannow & INFO in Event Viewer
    ... I know all about restore to ... pagefile to a low end of 2, ... The memory used by the user's ... Virtual Memory: Total paging file size for all drives is 2046MB ...
    (microsoft.public.windowsxp.general)
  • Re: Relocating application architecture and compiler support
    ... > then that is just virtual memory support. ... it wasn't virtual memory in the conventional sense. ... memory you could have addresses larger than physical memory that were mapped ... the base register was invisible to the program. ...
    (comp.arch.embedded)