Re: Challenge: reading ascii data

On Nov 15, 5:54 pm, "robin" <robi...@xxxxxxxxxxx> wrote:
"Eric I." <rubytrain...@xxxxxxxxx> wrote in message

news:1e833875-7435-43d5-9a8d-92c632fbdbdd@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx



On Nov 15, 9:16 am, "robin" <robi...@xxxxxxxxxxx> wrote:
"Eric I." <rubytrain...@xxxxxxxxx> wrote in message

news:1194913295.019301.94050@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

I see there was some earlier talk about whether the solutions needed
to store all the data before producing any output. Here is a Ruby
solution that does just that.

Because the data is incredibly sparse in these four dimensions, it
would be bad practice in terms of memory consumption to use a standard
array. So I use hashes to create a two-level "sparse array", where
the first level is indexed by the time and the second level is indexed
by a triplet of the three other values (i.e., a three-element array).

Seems overkill for what is basically a simple problem.

This runs pretty fast without any memory issues on the 13,381 line
input file. It takes less than 3.2 seconds to run on my old desktop
and 0.31 seconds on my one-year-old laptop.

Without using hashing, PL/I takes less than 2 seconds on my old desktop.

I'm not familiar with PL/I. Is your program storing all the data
ahead of time before displaying it? If so, in what type of data
structure is it being stored in?

If you were to use a 100 x 100 x 100 array for the data at each time
point, that's 1,000,000 elements. And since there are 400 time points
in the large data file, that would imply 400,000,000 elements. Am I
safe in assuming your PL/I code isn't using such a data structure?

For this type of task, where the number of time points is
open-ended, you wouldn't store all the data before outputting
results. An intimated in my post, at any one time, only the data
for one time-point is stored in my program.

Well the original problem definition was:

I have a Fortran solution to this problem and challenge the
AWK/RUBY/PLI/ETC advocates to post their solutions that can
process the U,V records below -> internal arrays and then
output UV product records in format requested by original
(homework?) assignment.

So the problem was only so "simple" if you redefine it to be a more
simple problem than originally stated.

And if you look back through the history of this thread, you'll find
that I submitted two solutions -- one which processed the time-points
incrementally and one which followed the original problem definition
as a demonstration that it could be done relatively easily in Ruby.
So it's unclear to me why you would view such a demonstration as
"overkill".

Eric
.

Relevant Pages

  • Re: Challenge: reading ascii data
    ... to store all the data before producing any output. ... would be bad practice in terms of memory consumption to use a standard ... So I use hashes to create a two-level "sparse array", ... Well the original problem definition was: ...
    (comp.lang.fortran)
  • Re: Shorter/better way to do this?
    ... need these constructs because Ruby has dynamics arrays, ... into the array size domain of the "sieve" array, ... understand what 'sieve' is supposed to do. ... So, when p8 < lndx is true, then all the other pi are < lndx too. ...
    (comp.lang.forth)
  • Re: Shorter/better way to do this?
    ... need these constructs because Ruby has dynamics arrays, ... into the array size domain of the "sieve" array, ... understand what 'sieve' is supposed to do. ... So, when p8 < lndx is true, then all the other pi are < lndx too. ...
    (comp.lang.forth)
  • Re: A couple of questions from a Ruby neophyte
    ... Before I start: I've done most of what you need to do successfully with Ruby before, for a little while now, so the news is good :-) ... I've handled MySQL and Access connectivity before, and they're both easy to get working. ... item to the array MyArray). ... Ruby's arrays aren't quite as feature-packed as PHP's arrays, which are more like mutant hashes with strange super-powers. ...
    (comp.lang.ruby)
  • Re: SWIG typemap for ta-lib
    ... const double inReal ... outRealarray i think. ... In Ruby I like to use all input parameters as function parameters. ... int iStartIx = *arg6; ...
    (comp.lang.ruby)