many files and NIO

From: Ethan Metsger (metsger_at_cse.ohio-state.edu)
Date: 01/27/05


Date: Thu, 27 Jan 2005 03:15:35 GMT

Hello, all. I'm new to the group, but I've been doing Java programming for a
couple years fairly consistently.

First, the specs: I'm running a Fedora Core 3 Linux system with JRE 1.4.2
(Athlon XP 1800+/512MB RAM).

I am working on a research project for my master's thesis which requires that
I load several seconds of video into memory for processing. (We're attempting
to track human torsos.) Each frame is stored on disk as a PPM (for ease of
use and portability purposes). I am using binary PPMs, so there is some data
compression, but nothing terribly significant, so they consume around 250kB
apiece. My test data set is 110 frames (about 25MB). This is a small data
set, representing around 3.6s of video. I would not expect loading 110 PPM
objects to stress the JVM or the system.

However, I get several java.lang.OutOfMemory errors when attempting to load
them all together. At present, I am trying to optimize the choice of the
torso segment, so it is possible for me to look at each frame individually and
see if I have acquired a better choice this time around. I have threaded this
process using a thread pool (five threads, 200ms poll). Each thread also
spawns another Process (which blocks the current thread execution).

Unfortunately, the time will come when I will need to have the pixmap data
handy. I could reread it, but this would incur a performance hit that I'm not
really willing to take unless I absolutely have to. I have done some reading
on NIO, particularly using MappedByteBuffers, which proponents say can
increase performance of reads and writes.

But I am particularly interested in their properties of memory management.
Since a MappedByteBuffer allocates memory on the native heap instead of the
JVM, will it clear up this problem? (Obviously, others may arise.) Can I use
a MappedByteBuffer to improve not only R/W performance, but also memory issues
related to reading in data?

If you could, please cc: me along with any replies to the newsgroup.

Sincerely,

Ethan Metsger



Relevant Pages

  • Re: How does this make you feel?
    ... > handle ccNUMA memory placement. ... would not likely produce significant frame coalescence. ... the OS has to split up the swap block. ... free block and not necessary be contiguously available for outswap ...
    (comp.arch)
  • Re: Direct Copying To Share Memory In NDIS ProtocolReceive
    ... access the buffer. ... one driver using IRp that is pended, ... Another concern is locking memory. ... frame that is transmitted to the miniports via the protocol must ...
    (microsoft.public.development.device.drivers)
  • RFC/Doc/BUGs: CONFIG_PACKET_MMAP
    ... configurable circular buffer mapped in user space. ... circular buffer of unswapable memory mapped in the capture process. ... the name indicates, this function allocates pages of memory, it allocates a power ... There is another vector of pointers, wich hold references to each frame ...
    (Linux-Kernel)
  • Re: Class Constants - pros and cons
    ... Your solution is procure memory. ... it seems to me that `MappedByteBuffer` and a bunch of little ... public void write(ByteBuffer bytes, int index, T item); ... Each time you read a `Star` in `ElementIO` you do mint a new `Star` so that is like Flyweight in some way, but seems like a little `Bridge` or `Adaptor`. ...
    (comp.lang.java.programmer)
  • Re: Website Upload- Easy Reading Promotion of Numberless Cryptography - Adacrypt
    ... He may regulate asleep monopolys, do you frame them? ... ruling drafts to large Abdul, ... Try not to shoot a memory! ...
    (sci.crypt)