Re: concurrency with DBI questions

Andy schreef:

I'm trying to look into the best way of speeding up multiple insert or
update statements into a database table. The values to insert or
update are in a file which could contain upto 100,000 records.

You need to tell more. For example: Are indexes and constraints
involved?

You talk about inserts and updates on a single table. Isn't the data
normalized?

Often is is best to bulk upload the new data in a single non-normalized
temporary table, and create separate queries for the updates and the
inserts. First make all foreign-key-related tables (if any) up-to-date
with the new data in the temporary table.


My plan was to have multiple children to do the database calls and
feed them via a process just reading the data from a file.

No need for that. Just bulk load the data into the temporary table, and
leave the rest to the DBMS.

You can use Perl to prepare the file for the import functionality of the
DBMS, and maybe to create a text-file with all the update and insert
queries.

--
Affijn, Ruud

"Gewoon is een tijger."

.