Re: memory management

josh wrote:
On 31 Ott, 11:32, Thomas Kellerer <YQDHXVLMU...@xxxxxxxxxxxxx> wrote:
josh, 31.10.2007 11:14:



On 31 Ott, 10:56, Thomas Kellerer <YQDHXVLMU...@xxxxxxxxxxxxx> wrote:
josh, 31.10.2007 10:22:
Hi, I have the following problem:
in a while loop I get from a db some records and in the same time
I put them with an insert preparedStatement in
a destination db but the jave process increase the memory until it
goes in out of memory.
How can I optimize that? also If I increase Tomcat java memory when I
process more than 500 records
tha java memory is already at 700mb of usage!
My guess is that you are buffering the retrieved rows somehow (or
keeping references to them). You will need to show us your code.
Thomas
ok I post the code:
st_from = conn.createStatement();
rs_from = st_from.executeQuery(query_from);
ps_into = conn_t.prepareStatement(insert_into);
int i_out_record = 0;
int i_in_record = 0;
while(rs_from.next())
{
i_out_record++;
out_record += "SELECT su record (" + i_out_record + ")";
in_record += "INSERT su record (";
row_error = rs_from.getString(campi_from[0]); // se errore su quale
id
for(int i=0; i < campi_from.length; i++)
{
out_record += campi_from[i] + "=" +
rs_from.getString(campi_from[i]) + "::" ;
ps_into.setString(i+1, rs_from.getString(campi_from[i])); //
prepara l'insert
in_values += campi_to[i] + "=" + rs_from.getString(campi_from[i])
+ "::" ;
}
out_record = out_record.substring(0, out_record.length()-2); //
tolgo il cancelletto finale
int res = ps_into.executeUpdate();
if(res != 0)
{
i_in_record++;
in_record += i_in_record + ")" + in_values;
// chiudi la statement
//ps_into.close();
}
out_record += "\n";
in_record += "\n";
/*
if(i_out_record > 10)
break;
*/
}
String log = "Per la tabella origine " + tabellaFrom + " processati
in uscita record: " + i_out_record + "\n";
log += "Per la tabella destinazione " + tabellaTo + " processati in
entrata record: " + i_in_record + "\n";
log += "-----------------------------------------------------------
\n";
log += out_record + in_record;
//out.print(st);
out.print(log);
After a quick look, I'd say the following lines are your problem:

>out_record += "SELECT su record (" + i_out_record + ")";
>in_record += "INSERT su record (";
>in_record += i_in_record + ")" + in_values;

As far as I can tell, this will create huge strings with the complete
result set, as they are declared outside the loop (actually twice the
result set because one for the out_Record and one for the in_record).

Thomas

Great, the problem was not in the resultset but in my logging string!
so I by-pass
that saving storing in the DB only errors cathed...
.... if I had to save, however, some logs may be I could use files ...
never string does it right?

Thanks for the help!




Specifically, look into a logging framework. log4j, commons-logging, or even the Java Logging API. They are designed for this type of logging. Log4j seems to be the most popular, and might as well be standard, commons-logging is a facade that lets you basically plug in whatever type of logging system you choose (such as log4j or Java Logging), and Java Logging comes standard.

--
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
.

Relevant Pages

  • Re: memory management
    ... I put them with an insert preparedStatement in ... also If I increase Tomcat java memory when I ... as they are declared outside the loop (actually twice the ... the problem was not in the resultset but in my logging string! ...
    (comp.lang.java.programmer)
  • Re: memory management
    ... I put them with an insert preparedStatement in ... a destination db but the jave process increase the memory until it ... also If I increase Tomcat java memory when I ... As far as I can tell, this will create huge strings with the complete result set, as they are declared outside the loop (actually twice the ...
    (comp.lang.java.programmer)
  • Re: memory management
    ... in a while loop I get from a db some records and in the same time ... I put them with an insert preparedStatement in ... a destination db but the jave process increase the memory until it ... also If I increase Tomcat java memory when I ...
    (comp.lang.java.programmer)