Re: seeking thru a file

Hi Mag,

Mag Gam wrote:
I have a compressed CSV gziped file. I was wondering if it is possible
to seek thru a file

For example:

I want to load the first 100 lines into an array. Process the data

Seek from 101 line to 200 lines. Process the data (remove lines 0 -
100) from memory

Seek 201 to 300 line. Process the data (remove 200-300)) from memory

etc..etc..


This would be very easy. Here is one way you could do it:
(I didn't test the code; I've just write it down, assuming you use
python 2.6, cause you didn't mentioned which version you are using...)

import gzip

step = 100
data_present = True
with gzip.open(foo.csv.gzip) as my_file:
counter = 0
my_buffer = []
while data_present:
while counter <= step:
line = my_file.readline()
if line:
my_buffer.append(my_file.readline())
counter += 1
else:
data_present = False
break
if len(my_buffer) > 0:
do_something(my_buffer)
counter = 0
my_buffer = []


Kind Regards, Nils
.

Relevant Pages

  • Re: fast stable sort
    ... For fastest load, at expense of slower sort, you ask the operating ... and you memory-map the file directly into that array. ... memory of your big array. ... possible to assure locality of reference and thus avoid thrashing ...
    (comp.programming)
  • Re: dealing with a time series piece wise (with gaps)
    ... You do not mention the size of the text file, if its small enough to fit in memory, using LOAD and then working off the data in memory might be faster than: ... However, since you will not know how much data to expect in a segment you will not be able to pre-allocate, this might slow your code. ... store this piece of triplet array (time, ...
    (comp.soft-sys.matlab)
  • Re: Save an array to memory for another program to read it
    ... I need the program to store into memory this array, end itself and when immediately runs again to load this array again from the memory. ...
    (comp.lang.fortran)
  • Re: remoting for file transfer
    ... going to have to load the contents of the file into a byte array in memory and then transmit it. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: cant read large files - help
    ... MULTIPLY YOUR MEMORY TEMPORARILY (10X LARGER SHOULD *DEFINITELY* BE ... think of to do), however, requirements stipulate that files that are ... an array by not using filebut by using my function above, bigfile, ... load the whole thing into memory first. ...
    (comp.lang.php)