Re: Handling large amounts of data

From: jacob navia (jacob_at_jacob.remcomp.fr)
Date: 11/20/04 

Date: Sat, 20 Nov 2004 19:44:05 +0100

Wayne Marsh wrote:
> Hi all.
>
> I am working on an audio application which needs reasonably fast access
> to large amounts of data. For example, the program may load a 120 second
> stereo sound sample stored at 4bytes per sample, which would mean over
> 40MB of data at a 44100Hz sampling rate.
>
> Now, what would be a good way to handle all of this data? Ideally, for
> the sake of my own sanity and the algorithms within directly functional
> portions of the code, I'd like to interface with the data via normal
> array syntax. Are arrays of this size really suitable, or would there be
> a better way? Writing the data to disk and then memory mapping the files
> seemed like an option, although I suspect that would be analogous to the
> operating system's virtual memory system.
>
> Any ideas?

It depends on how much memory is there in the computer. If you work
in a PC environment, machines now come routinely equipped with 1GB
of RAM, and 40MB is nothing. Just load it into RAM and use it as an
array. The VM system will do the paging for you if your OS is UNIX
or windows.

It would be surprising if you wanted to process all this data in an
embedded system with a few K of RAM anyway.

jacob

Relevant Pages

  • Re: Out of Memory Problem in VB 6.0 Application.
    ... that's a fixed size array. ... > block of memory. ... Just think of it as 2gigs of ram that you have at your disposal. ... matter if that's real ram or swap file it's still 2 gigs available to your ...
    (microsoft.public.vb.general.discussion)
  • Re: Out of memory when reading a large file
    ... I would ideally like to read the data in an array and then be able to ... In order to do what you want, you need enough RAM on your computer. ... To get all the data into memory at once, ... chop up the data into smaller chunks. ...
    (comp.soft-sys.matlab)
  • Re: Out of Memory Problem in VB 6.0 Application.
    ... I just remembered that dynamic arrays are pointers to SAFEARRAY structure, ... So memory usage is as follows: ... that's a fixed size array. ... > Just think of it as 2gigs of ram that you have at your disposal. ...
    (microsoft.public.vb.general.discussion)
  • Re: memory issues
    ... If you are running out of memory because your array size is coming ... of memory simply because you dont have enough RAM. ... single mmap file. ...
    (comp.soft-sys.matlab)
  • Re: Dual Port RAM Simulation
    ... the array can not be written. ... Both write processes assign to ram, ... station memory than a signal. ... -- port a read ...
    (comp.lang.vhdl)