Re: please confirm something (jdbc)

Lew wrote on 29.11.2007 16:01:
Thomas Kellerer wrote:
Note that using ByteArrayOutputStream.toByteArray() will create a copy of its internal buffer so that will double the amount of memory that you need. That's the reason I wrote my own dynamic byte array (where I can access the byte array without copying it around)

Using ByteArrayOutputStream.toByteArray() will also guarantee that "the valid contents of the buffer have been copied into [the resultant array]", and that the resultant array is the right size. I find these guarantees to be worth the copy overhead for reasonable-sized objects (under a couple MB, as is typical for, say, graphic images, one use case for BLOBs).

Given all the memory re-allocation that goes on inside the ByteArrayOutputStream anyway, I'm not sure that one more allocation is all that much extra overhead. If you do it right, the internal array of the ByteArrayOutputStream will immediately become eligible for GC right away anyway, so the "efficiency" you gain from avoiding that one copy might not be worth the extra risk and effort of maintaining a custom class.

It depends on how L your BLOB is, naturally. I can see how a large enough object would require striped processing and other tricks, but like most optimization this strikes me as one of those "Don't do it 'til you have to, and don't think you have to until you've measured" scenarios.

Valid points, indeed, especially the one about the array beeing GC'ed pretty soon. But still, it will make a difference if I hold 20 or 40MB in memory.

Well I am processing BLOB where I don't know the size (and the number) in advance, and they could potentially be quite big. My own ByteBuffer does basically the same as ByteArrayOutputStream with the exception that I expose access to the array (which is also guaranteed to have the right size).

I only partially agree with your optimization strategy though. If I have two ways of coding something, and I know that one will definitely use less memory (or be quicker for whatever reason) then I will choose the one "better" one. I think one reason for slow software is the attitude "I'll fix the performance issue later", because "later" you'll have so many places to fix that it will probably be hard to nail down the real cause.

Cheers
Thomas

.

Relevant Pages

  • Re: Yet another question about array indexing
    ... I found several posts about array indexing but I am still confused on ... The reason why I ... entity memory is ... to try to implement this in a real RAM, ...
    (comp.lang.vhdl)
  • Re: please confirm something (jdbc)
    ... of its internal buffer so that will double the amount of memory that you need. ... That's the reason I wrote my own dynamic byte array ... Given all the memory re-allocation that goes on inside the ByteArrayOutputStream anyway, I'm not sure that one more allocation is all that much extra overhead. ...
    (comp.lang.java.programmer)
  • Re: Fast string operations
    ... Looping: I thought looping over arrays in managed code was "slow" ... array handling and such. ... The problem with TrimHelper is that it always returns a new string instance. ... The customer perceives this as a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: High Memory Consumption of Classes and Arrays
    ... Only the array itself has overhead. ... memory as a reference type. ... > least consume 40 bytes of memory. ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Fast linked list
    ... > amounts of memory rather than huge chunks of it. ... random insertions into a vector/dynamic array are not as slow ... to cause a cache miss. ... some hard numbers on speed differences between lists and arrays. ...
    (microsoft.public.vc.mfc)