Re: reading a loong table of numbers

On Oct 8, 12:49 pm, t...@xxxxxxxxxxxxx (Thomas A. Russ) wrote:
Mirko<mirko.vuko...@xxxxxxxxx> writes:
On Oct 8, 9:43  am, p...@xxxxxxxxxxxxxxxxx (Pascal J. Bourguignon)
wrote:
Mirko<mirko.vuko...@xxxxxxxxx> writes:
What I meant was based on the assumption that the reader reads the
next `item' (item's being delimited by blanks), and then tries to
figure out whether the item is a string or a number.
So my real question was whether I can circumvent this figuring out
part, and tell the reader to just interpret the characters as a number
(integer or float).

If your input contains only number, what difference would it make?

(read-from-string "123") and (parse-integer "123") will do the same
work.   And there's no parse-float, you could use a   parse-number
function existing in some library, I don't remember which.

I tried to profile your examples.  This is on sbcl:

CL-USER> (let ((times 5000000))
      (defun !read-from-string ()
        (dotimes (i times)
          (read-from-string "123")))
      (defun !parse-integer ()
        (dotimes (i times)
          (parse-integer "123"))))
!PARSE-INTEGER
CL-USER> (sb-profile:profile !read-from-string !parse-integer)
; No value

I would have made sure that !read-from-string and !parse-integer were
compiled first.



CL-USER> (progn
      (!read-from-string)
      (!parse-integer))

NIL
CL-USER> (sb-profile:report)
  seconds  |     consed    | calls |  sec/call  |  name
-----------------------------------------------------------
     7.715 | 3,519,909,712 |     1 |   7.715000 | !READ-FROM-STRING
     0.408 |             0 |     1 |   0.407999 | !PARSE-INTEGER
-----------------------------------------------------------
     8.123 | 3,519,909,712 |     2 |            | Total

estimated total profiling overhead: .000 seconds
overhead estimation parameters:
  8.000001e-9s/call, 1.542e-6s total profiling, 6.44e-7s internal
profiling
; No value
CL-USER>
So, a float point of parse integer might be beneficial.

I will first profile myreadroutines to try and eliminate
bottlenecks, and then maybe lookup that parse-float library that you
mentioned.

I'm sure that PARSE-INTEGER will be more efficient than reading from a
string, but you also need to figure in the effort to get the string to
pass to PARSE-INTEGER.

I would try some of the simpler suggestions such as just usingREADand
see if that turns out to be fast enough for your application.

How long does it take for you toreadthe 30000 line file?

--
Thomas A. Russ,  USC/Information Sciences Institute

Profiling results for reading three files (total 8GB) four times each:

seconds | consed | calls | sec/call | name
-----------------------------------------------------------
17.084 | 3,235,900,496 | 48 | 0.355916 | CL-TECPLOT::READ-
RECORD
-----------------------------------------------------------
17.084 | 3,235,900,496 | 48 | | Total

estimated total profiling overhead: .000 seconds
overhead estimation parameters:
8.000001e-9s/call, 1.542e-6s total profiling, 6.44e-7s internal
profiling


There are 48 calls because I am profiling a method `read-record'. This
method is also collecting other data of much smaller size (file
headers, variable descriptors, etc).

Over the weekend I was playing with read-sequence into a ring buffer
from which I would read-off successive numbers.

As noted before, I will first concentrate on reading into one large
array since I know its size - but this will have to wait for two
weeks.
.

Quantcast