Re: Fun with Tasking

mhamel_98@xxxxxxxxx wrote:

Anyhoo Jeff, I'll try to explain in more detail what my program is
doing and hopefully you can see where I'm going wrong. The program
has a number of packages, each handling what I call a subsystem, to be
verbose: ephemeris, residual, vector, covariance and a few others
(this sort of equates to the "exec" procedure in my sample). Each of
these systems has a number of procedures of interest, like a
differencing of quantities, a statistical analysis, etc. Now each of
these procedures of each of these subsystems can have up to 60 (and
more as we go on) objects - each one I handle as it's own task. I've
found that throwing 60 threads at the OS (W2K in this case) is, while
better then serially and single-threaded on multi-cpu's, not terribly
efficient. For this application, I've found most procedures operate
best if N+1 threads are running, N being the processors available.
The output is all .csv files. For awhile, to show an ensemble of
data, I would do a lot of cut and pasting between .csv's, one day, I
thought, why not have the program do this? So I implemented these
"collating" tasks that all 60 or so threads would send some
information to for some sort of unified processing. Each procedure of
each subsystem/package is slightly different either in the data types
being handled or the manner in which the data is processed so a
"generic" collating task isn't really feasible.

That's what's happening, now how do I handle it? As the program
starts, I have each subsystem loads its procedures (via procedure
pointers) to a task manager. A command is recieved via CLI and the
task manager then inserts, let's say, 60 copies of a specified
procedure into a list which are doled out to worker threads so that
only N+1 are active at any time. The issue I'm having is, how do you
have a unique collating task created for those 60 processes to send
updates to? As it is now, I have all of these (nearly a dozen) unique
collating tasks defined in the task manager. I would rather have them
defined in the subsystem package to which they really belong. As the
subsystem packages "push" their procedures to the task manager, the
task manager has no visibility into the subsystem. My aim was to
define an area (a System.Address for example) in a control block
record that the task manager would have space for "a task". What this
task was, what it did, where it came from the task manager would not
know nor care, but it would be the owner, or master, in some sense so
the "preprocess" or initialization component could exit. I hope this
makes a little sense.

There's still a lot of implementation in here. Talking about procedure pointers and addresses is not only implementation, it's quite low level. One of Ada's great strengths is its ability to express, implement, and use high-level abstractions. The low-level stuff is still available, but generally best hidden well down in the implementation of such an abstraction.

What I get from your description is: You have classes of computations ("subsystems"); each class can have numerous computations carried on at the same time; each computation produces output; you want some of the output from all the computations of the same class to be subjected to collating.

Everything else seems like implementation detail that may be blocking your view of a solution.

Steve has given you some ideas about how to abstract the idea of computations and controlling their scheduling. He didn't have get into the resulting output or how to subject some of it, within a class of computations, to collating.

I would probably take a different approach than Steve, with different subsystems more independent, in order to ease the collating within the subsystems, but that may be more a matter of style than substance.

If you can produce high-level package specifications that only address the high-level concepts of your problem, without worrying about implementation (private parts, bodies), with a road map of how they work together, you'll probably find that you can then work out a way to implement them.

--
Jeff Carter
"Don't knock masturbation. It's sex with someone I love."
Annie Hall
45
.

Relevant Pages

  • Re: Fun with Tasking
    ... My approach was to start up this collating task and give it's ... package as the sample has it. ... has a number of packages, each handling what I call a subsystem, to be ... task manager then inserts, let's say, 60 copies of a specified ...
    (comp.lang.ada)
  • Re: unstable Win 16 subsystem
    ... check the processes under that task manager I would find wowexec was being ... I restored the system to the beginnging of the XP install and started to ... The 16 bit subsystem failed when the update called "KB943649" was installed. ... my Casio organiser data - I has always worked with XP before now. ...
    (microsoft.public.windowsxp.general)
  • Re: Fun with Tasking
    ... has a number of packages, each handling what I call a subsystem, to be ... differencing of quantities, a statistical analysis, etc. ...
    (comp.lang.ada)