Re: performance and memory usage.
- From: arnold <arnold@xxxxxxxxxxx>
- Date: Sun, 12 Feb 2006 21:25:43 +0100
Chris Smith wrote:
> arnold <arnold@xxxxxxxxxxx> wrote:
> There are a few things you could do, if you're really desperate. For
> example, you can save a lot of space by re-implementing HashMap and
> ArrayList to use primitive int data instead of pointers to objects of
> class Integer.
I suppose there are some libraries which has alredy done these implementations.
>> Thats actually quite terrible memory utilisation.
>
>
> It's not all that bad; you're just storing a lot of data. In any case,
> it's a realtively small constant multiple of what you'd get from any other language.
>
>> In C I could write this so the program would not require more than approx 48MB. (where approx 32MB is the real data and 16 MB is the arrays with the pointers to the structures containing the data).
>
>
>
> No, you couldn't. You're still ignoring the overhead for memory allocation data structures. That overhead exists in C as well as in Java.
how so? Yes, its a lot of data, but the point is that the data is only about 32MB of pure data, the rest of the approx 150MB are internal data for the structures and objects etc (I tested the limits). That yields a utilisation percentage of 15%, which is quite terrible.
In C what you need is a hash array of 4 bytes per entry, and a second array for iteratiing the keys of 4 bytes per entry. then there is the hash data which is 8 bytes per entry. Thats 4 + 4 + 8 * 2million which is 32MB, and that includes all intrinsic data the strucures need. If I am missing something please let me know.
(the hash structure might need some more space (i dont know the actual implementation of the function), but its most likely not more than an additional 4-8 bytes per entry, and that would take us up to 48MB.
>> To me its seems that you should not really use java for in-memory storage, except for smaller data sets.
>
>
> You should, ideally, use a database for storing large data sets. Ideally, you wouldn't implement your own database anyway, in any language.
Databases and network operations are slow and costly (in money for customer solutions), plus you then have an additional 2-3 programming tiers, in addition the customer gets unnecessary system maintenance.
I am a bit sick of the notion in todays programming world that everything can only be solved by another heavyweight system which is excruciately complex to use and costs an arm and a leg in money and maintenance. Thats just a waste of money and time. Most problems can be solved by much simpler solutions (which are just as adaptable, scalable and possibly contains inherent maintenance), than the tomcat/struts/IBM/BEA/ORACLE/SAP type solutions. Considered BerkleyDB? JavaSpaces? Hashmap with ReiserFS? The problem is that most people dont realise this, so there are'nt many good or well known solutions for such systems.
arnie
.
- References:
- performance and memory usage.
- From: arnold
- Re: performance and memory usage.
- From: Chris Smith
- Re: performance and memory usage.
- From: arnold
- Re: performance and memory usage.
- From: Chris Smith
- performance and memory usage.
- Prev by Date: Re: performance and memory usage.
- Next by Date: Re: performance and memory usage.
- Previous by thread: Re: performance and memory usage.
- Next by thread: Re: performance and memory usage.
- Index(es):
Relevant Pages
|
