Re: Transforming ascii file (pseduo database) into proper database
- From: Bruno Desthuilliers <bdesth.quelquechose@xxxxxxxxxxxxxxxxxxx>
- Date: Mon, 21 Jan 2008 23:15:40 +0100
p. a écrit :
I need to take a series of ascii files and transform the data
contained therein so that it can be inserted into an existing
database. The ascii files are just a series of lines, each line
containing fields separated by '|' character. Relations amongst the
data in the various files are denoted through an integer identifier, a
pseudo key if you will. Unfortunately, the relations in the ascii file
do not match up with those in the database in which i need to insert
the data, i.e., I need to transform the data from the files before
inserting into the database. Now, this would all be relatively simple
if not for the following fact: The ascii files are each around 800MB,
so pulling everything into memory and matching up the relations before
inserting the data into the database is impossible.
My questions are:
1. Has anyone done anything like this before,
More than once, yes.
and if so, do you have
any advice?
1/ use the csv module to parse your text files
2/ use a temporary database (which schema will mimic the one in the flat files), so you can work with the appropriate tools - ie: the RDBMS will take care of disk/memory management, and you'll have a specialized, hi-level language (namely, SQL) to reassemble your data the right way.
2. In the abstract, can anyone think of a way of amassing all the
related data for a specific identifier from all the individual files
without pulling all of the files into memory and without having to
repeatedly open, search, and close the files over and over again?
Answer above.
.
- Follow-Ups:
- References:
- Prev by Date: Re: Bug in __init__?
- Next by Date: Re: When is min(a, b) != min(b, a)?
- Previous by thread: Re: Transforming ascii file (pseduo database) into proper database
- Next by thread: Re: Transforming ascii file (pseduo database) into proper database
- Index(es):
Relevant Pages
|
