Re: Memory Useage

mhamel_98@xxxxxxxxx wrote:
Hello c.l.a. Another question, I have a program that stores data on
the disk using sequential_io. When I later read that data into an
array, the memory growth after ingesting a file is much much larger
than the disk footprint. A file that takes 26.8MB on disk (over 134k
records) causes the program to swell by over 600MB! Holy bloatware.
A short overview of what I'm trying to do - each sequential_io data
file has an associated header file with stuff like number of records,
etc. The header is read, and an array is then created based on how
many records are said to be in the data file. The data file is then
read, sticking a node into the array. Some abbreviated code below,
the spec:

generic
type Node_Type is private;
package Node_Manager is

package Seq is new Sequential_Io (Node_Type);

type Node_Array is array (positive range <>) of Node_Type;
type Node_Ptr is access Node_Array;

Is the actual type for Node_Type a record type with variants? If so, is the size of the largest variant much larger than the size of the most common variants? I don't know about ObjectAda, but in GNAT the Node_Array would have a size that lets you store the largest variant in every array element, while the Node_Type objects stored in the sequential_io file probably use only as much file-space as the actual variant of each object requires.

The solution in GNAT would be to allocate storage for each Node_Type object separately and have an array of accesses:

type Node_Ptr is access Node_Type;
type Node_Array is array (positive range <>) of Node_Ptr;
type Node_Array_Ptr is access Node_Array;

This increases memory overhead by allocating more blocks from the heap, but it may reduce the overall memory requirement if the largest variant of Node_Type is much larger than the average variant.

--
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
. @ .
.

Relevant Pages

  • Re: Size of arrary
    ... know how much memory my application really takes, ... > knowing how much memory each array takes? ... > For the string array, you'll need to loop through all the elements because ... > all the same, but in that case, why are you using a variant?). ...
    (microsoft.public.vb.controls)
  • Re: Where is this array stored in memory?
    ... how the array is declared and what statements do the allocation. ... The possible relevance is that sometimes dynamically allocating ... arrays can help by using only as much memory as the particular ... > wrote the array on disk after each iteration, ...
    (comp.lang.fortran)
  • Starting an Application
    ... I am storing application data as a byte array in my database. ... back to a file on disk to enable the parent application to run it or can I ... somehow avoid disk operation and activate the application form the file ... contents in memory in the form of a memorystream or some other in memory ...
    (microsoft.public.dotnet.framework.windowsforms)
  • Re: ForwardOnlyMemoryStream?
    ... array it's using for storage, so it would have to create a new array and copy the remaining data to it. ... This could cause even more memory problems than you have now... ... I would suggest that you make your own stream reader class that uses a large buffer, that way you reduce the frequency of the disk access while keeping the memory usage at a controllable level. ... that someone else had already done a smarter memory stream. ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Memory Useage
    ... the disk using sequential_io. ... and an array is then created based on how ... the size of the largest variant much larger than the size of the most ... This increases memory overhead by allocating more blocks from the heap, ...
    (comp.lang.ada)