Re: performance and memory usage.

arnold <arnold@xxxxxxxxxxx> wrote:
Thats actually quite terrible memory utilisation.

It's not all that bad; you're just storing a lot of data. In any case,
it's a realtively small constant multiple of what you'd get from any
other language.

In that case I have some questions:

1- Is there a more efficient way of storing this in memory, so you could
store up to, say, 100 million objects?

3- Are there any methods of making the data structures themselves more
memory efficient.

There are a few things you could do, if you're really desperate. For
example, you can save a lot of space by re-implementing HashMap and
ArrayList to use primitive int data instead of pointers to objects of
class Integer.

2- Is there a method of creating simple data placeholders which are not
objects, similar to structs in c/c++.

No.

In C I could write this so the program would not require more than
approx 48MB. (where approx 32MB is the real data and 16 MB is the arrays
with the pointers to the structures containing the data).

No, you couldn't. You're still ignoring the overhead for memory
allocation data structures. That overhead exists in C as well as in
Java.

The only difference is that in Java, objects carry an additional 8 bytes
or so of object header, and that the HashMap and ArrayList classes work
with Object and not int (solvable, as I mentioned, if you are willing to
re-implement those classes).

To me its seems that you should not really use java for in-memory
storage, except for smaller data sets.

You should, ideally, use a database for storing large data sets.
Ideally, you wouldn't implement your own database anyway, in any
language.

It's definitely possible to implement a database in Java if you so
desire, but you wouldn't store everything in object-oriented data
structures; instead, you'd use memory mapped files and java.nio.Buffer
to access the data. In-memory data structures are designed to be
convenient and capable for manipulating working data, not to be an
efficient means of data storage.

--
www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation
.

Relevant Pages

  • Re: DataSet is good for ASP.NET?
    ... Yes, the DataSet does take up space in memory, but there are several reasons ... the page...Still no trips to the database and the benefits of caching. ... Even if the initial page request must go to the database, storing the data ... > dataset object created in an aspx page and this page will be accessed by ...
    (microsoft.public.dotnet.framework.aspnet)
  • Re: Trouble getting file owner on large amount of files
    ... Try to store file names into database instead of storing in memory. ...
    (microsoft.public.vc.language)
  • Re: Ideas on storing and retriving files in a Windows Application
    ... files etc. 2) Because it is stored in a relational database, ... blobs takes some minor modifications other than just using SELECT blah FROM ... horror stories of people who started storing BLOB data in a db only to have ... > will open the appropiatte application to display the file and 2. ...
    (microsoft.public.dotnet.framework.adonet)
  • Re: Number + 1
    ... Using the same number in a variety of databases may be a reason for storing ... incremented portion and the extended portion. ... The value for dmrdate is assigned automatically as today's date and time, ... My experience not only with my database but with other customers & ...
    (microsoft.public.access.gettingstarted)
  • Re: Image ctl causes bloat when using text field to store path/fil
    ... Are you storing the image in the database? ... it in your database or embedding it in the report and/or form. ... got the same bloating problem in a report containing the Image control. ... Compact on Close will attempt to do so while the database is use... ...
    (microsoft.public.access.tablesdbdesign)
Loading