Re: memory management

On 31 Okt., 11:32, Thomas Kellerer <YQDHXVLMU...@xxxxxxxxxxxxx> wrote:
josh, 31.10.2007 11:14:





On 31 Ott, 10:56, Thomas Kellerer <YQDHXVLMU...@xxxxxxxxxxxxx> wrote:
josh, 31.10.2007 10:22:

Hi, I have the following problem:
in a while loop I get from a db some records and in the same time
I put them with an insert preparedStatement in
a destination db but the jave process increase the memory until it
goes in out of memory.
How can I optimize that? also If I increase Tomcat java memory when I
process more than 500 records
tha java memory is already at 700mb of usage!
My guess is that you are buffering the retrieved rows somehow (or
keeping references to them). You will need to show us your code.

Thomas

ok I post the code:

st_from = conn.createStatement();

rs_from = st_from.executeQuery(query_from);

ps_into = conn_t.prepareStatement(insert_into);

int i_out_record = 0;
int i_in_record = 0;

while(rs_from.next())
{

i_out_record++;

out_record += "SELECT su record (" + i_out_record + ")";
in_record += "INSERT su record (";

row_error = rs_from.getString(campi_from[0]); // se errore su quale
id

for(int i=0; i < campi_from.length; i++)
{
out_record += campi_from[i] + "=" +
rs_from.getString(campi_from[i]) + "::" ;

ps_into.setString(i+1, rs_from.getString(campi_from[i])); //
prepara l'insert

in_values += campi_to[i] + "=" + rs_from.getString(campi_from[i])
+ "::" ;
}
out_record = out_record.substring(0, out_record.length()-2); //
tolgo il cancelletto finale

int res = ps_into.executeUpdate();

if(res != 0)
{
i_in_record++;
in_record += i_in_record + ")" + in_values;

// chiudi la statement
//ps_into.close();
}

out_record += "\n";
in_record += "\n";

/*
if(i_out_record > 10)
break;
*/
}

String log = "Per la tabella origine " + tabellaFrom + " processati
in uscita record: " + i_out_record + "\n";
log += "Per la tabella destinazione " + tabellaTo + " processati in
entrata record: " + i_in_record + "\n";
log += "-----------------------------------------------------------
\n";
log += out_record + in_record;

//out.print(st);
out.print(log);

After a quick look, I'd say the following lines are your problem:

>out_record += "SELECT su record (" + i_out_record + ")";
>in_record += "INSERT su record (";
>in_record += i_in_record + ")" + in_values;

As far as I can tell, this will create huge strings with the complete
result set, as they are declared outside the loop (actually twice the
result set because one for the out_Record and one for the in_record).

Actually 4-fold, since at the end he does:

log += out_record + in_record;
//out.print(st);
out.print(log);

Just changing this will save 50% memory:

out.print(out_record);
out.print(in_record);

But, of course, the out_record and in_record logging is totally bad.
First, avoid + on strings. Use a StringBuilder instead.
If this doesn't solve the problem, write the out_record and in_record
directly to a file. For example, through a PrintWriter.


.

Relevant Pages

  • Re: real issue of Eric Granger myth post.
    ... Rudy V also pointed out that adding strings in the loop can cause a disaster ... memory were being introduced. ...
    (borland.public.delphi.non-technical)
  • Re: null terminated strings
    ... It only saves memory if strings can be longer than 255 characters. ... Well, you also save the code that handles the count, since the termination condition is simply a condition-code check for null in the move loop. ... If you use counted strings is normally trivial to calculate the number of characters to process before executing the loop. ...
    (comp.os.vms)
  • Re: StringBuilder vs. String performance
    ... The predominant problems that occur with strings are centered ... memory is more likely to cause real problems. ... shouldn't actually bring the app to a crashing halt. ... "append in a loop" which is what StringBuilder is usually the cure for, ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: real issue of Eric Granger myth post.
    ... Mentioning loop at start of my post was my mistake. ... So "adding strings in the loop" is not the point. ... > Has this been a SHOW STOPPER for Delphi? ... If you prealocated memory with SetLength, it is fast, ...
    (borland.public.delphi.non-technical)
  • Re: Fast string operations
    ... > is why people use unsafe code now and then to use pointer arithmetic to ... > loop over arrays without all the unnecessary bounds checking. ... don't return the trimmed string". ... The customer perceives this as a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)