Re: Massive sort among dozens of files


"Steve O'Hara-Smith" <steveo@xxxxxxxxxx> skrev i en meddelelse
news:20070628142725.8ed95f08.steveo@xxxxxxxxxxxxx
On Thu, 28 Jun 2007 13:49:41 +0200
Pascal Bourguignon <pjb@xxxxxxxxxxxxxxxxx> wrote:


sort(1) should be able to process each of the input files
independently. Once you have 50 files sorted, you can easily write a
merge program (it could even be done in bash).

If you are working on a Delphi platform, I could suggest the following.
Sadly enough, I lost the source for it ...

- split your files in chunks of about 1000 records. if the sort key is not
the first part of the record, add a sortkey
- write one chunk at a time to a listbox with property sorted=true
- empty the listbox into a file
- next chunk etc.

- when all data has been sorted, take chunk 1 and 2, merge them, and write
them to chunk x + 1
where x equals the number of chunks written from list listboxes
- take 3 and 4, and write them to chunk x + 2
- etc
- at last, you will have no more input files

Nico


.