Re: Challenge: reading ascii data

"Eric I." <rubytraining@xxxxxxxxx> wrote in message
news:6a554d6c-75fa-4a32-b98c-62a617bd839c@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
On Nov 15, 5:54 pm, "robin" <robi...@xxxxxxxxxxx> wrote:
"Eric I." <rubytrain...@xxxxxxxxx> wrote in message

news:1e833875-7435-43d5-9a8d-92c632fbdbdd@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
On Nov 15, 9:16 am, "robin" <robi...@xxxxxxxxxxx> wrote:
"Eric I." <rubytrain...@xxxxxxxxx> wrote in message

news:1194913295.019301.94050@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

I see there was some earlier talk about whether the solutions needed
to store all the data before producing any output. Here is a Ruby
solution that does just that.

Because the data is incredibly sparse in these four dimensions, it
would be bad practice in terms of memory consumption to use a standard
array. So I use hashes to create a two-level "sparse array", where
the first level is indexed by the time and the second level is indexed
by a triplet of the three other values (i.e., a three-element array).

Seems overkill for what is basically a simple problem.

This runs pretty fast without any memory issues on the 13,381 line
input file. It takes less than 3.2 seconds to run on my old desktop
and 0.31 seconds on my one-year-old laptop.

Without using hashing, PL/I takes less than 2 seconds on my old desktop.

I'm not familiar with PL/I. Is your program storing all the data
ahead of time before displaying it? If so, in what type of data
structure is it being stored in?

If you were to use a 100 x 100 x 100 array for the data at each time
point, that's 1,000,000 elements. And since there are 400 time points
in the large data file, that would imply 400,000,000 elements. Am I
safe in assuming your PL/I code isn't using such a data structure?

For this type of task, where the number of time points is
open-ended, you wouldn't store all the data before outputting
results. An intimated in my post, at any one time, only the data
for one time-point is stored in my program.

Well the original problem definition was:

I have a Fortran solution to this problem and challenge the
AWK/RUBY/PLI/ETC advocates to post their solutions that can
process the U,V records below -> internal arrays and then
output UV product records in format requested by original
(homework?) assignment.

So the problem was only so "simple" if you redefine it to be a more
simple problem than originally stated.

As stated, the problem was only to store the U,V records.
There was no statement about storing all of them before
outputting any of the results.

But even if that were the case, the OP abandoned
that idea as impractical.

And if you look back through the history of this thread, you'll find
that I submitted two solutions -- one which processed the time-points
incrementally and one which followed the original problem definition
as a demonstration that it could be done relatively easily in Ruby.
So it's unclear to me why you would view such a demonstration as
"overkill".

Because it is. The OP abandoned that idea. As well,
Bill Kleb said "it's proven to be a bad idea".
Furthermore, it is impracticable because
there is no limit on the quantity of input data.

From: "Bil Kleb" <Bil.Kleb@xxxxxxxx>
Sent: Monday, November 12, 2007 10:54 PM

| David Frank wrote:

| > Why havent you shown us a sample of this "young programming language"s
| > declaration and use of 4d arrays?
|
| Because "as your later post reveals, one might exhaust
| available memory", i.e., it's proven to be a bad idea.



.

Relevant Pages

  • Re: Logical values
    ... to keep the memory consumption to minimum and thought that store this ... Then I discover that it appears that a logic array ... I would suggest that you store the data ... subsasgn operators to make it transparent that you are operating ...
    (comp.soft-sys.matlab)
  • Re: Finding the nearest match without reusing results
    ... comparing an array with a value generates an array of Trues and ... Any diff with wrong State or with used Store is implicitly zeroed out. ... Each must be larger than the absolute value of the maximum Sales ... go.....it just needs to return the closest match. ...
    (microsoft.public.excel.programming)
  • Re: read keyboard input and storing in an array?
    ... > I'm trying to store user input in an array, ... You have the beginnings of that logic already since your prompt tells the ... 201st value since the array only has room for 200 values. ... This will force you to store the int values as Integer ('Integer' ...
    (comp.lang.java.help)
  • Re: attempting an actual game...
    ... >>> and inflexible by the absurd decision to use a bit array for square ... as then one has 8 bits in which to store a color and a few flags ... Using a 2D int (or, ... > Change direction and you may eventually complete a game. ...
    (comp.games.development.programming.misc)
  • Re: Sparse arrays
    ... >access array element, set it, sit for a bit, and read it back ... ONLY operations are STORE and RETRIEVE with no structure to the ... If you can tolerate a bit of inefficiency ... They live in sorted lists. ...
    (comp.lang.fortran)