Re: tcl and threads: installing and other info

susara schrieb:
Cameron Laird wrote:

Just for practice, then, say a few words about your
concurrency requirements. There's a fair probability that
we can suggest a Tcl-based approach that does everything you
need withOUT you having to mess with configuration options,
download new sources, or so on.

Firstly thanks VERY much for the doc pointers, I'll go read up. Then,
my reasons for going off on a thread...

I've got a process that connects to a postgres db on the one hand, and
on the other accepts socket connections from subscription processes. It
waits on database notification events. On notify it has to do some
intensive processing and forward the results to the subscribed
processes. The processing will be pretty time consuming, and I'm pretty
sure those db notifications are going to come in faster than what I'm
going to be able to process.

So I'd like to parallelise the notification processing. The master
process will listen for the db notifications; fire off slave threads to
handle them, get the results back from each thread as it completes
processing and send it to the subscribed processes.

Sounds like the correct architecture, a event based master process using
a recent Tcl postgreSQL interface and event based socket stuff to manage
the IO things.
But if your on linux you could use processes instead of threads for the
slaves, started with open | listening on a pipe, much easier and much
more stable in case one of the slaves crashes or does nasty things. If
in a thread it would kill you process, in another process you can just
close the pipe, kill the child and restart a new one.

As your using PostgreSQL you could even create much of the master
process as a stored procedure inside the database, called by triggers.

I'll have to protect some common data against concurrent access. I
could be lazy and use the database's locking mechanisms to coordinate
the slaves, but for performance I'd prefer to keep as much of this in
memory and somehow share it between them, using mutexes where
necessary.
If you don't have enough memory for the database to do its
locking/concurrency control in memory your into swapping anyway.

Tcls thread model is message passing based, so you don't have shared
memory and mutex stuff usually, unless you explicitly request something
like it, see the thread package documentation for details.

This in particular holds for the results that have to be
communicated back to the master.

If there is an elegant way of handling this without using threads I'd
be very happy. I really don't feel like messing around with installing
manually built libraries on a debian box, especially not a live system
that runs 24/7...

As i said above. Use a master process that spawns new slaves as
processes when needed. Works perfectly well. I had such a system for
managing a distributed web crawler, where each agent-controller process
managed from 0-50 Perl processes on pipes, doing job control, recovering
from crashes of the perl scripts etc.

Btw. Tclsh on debian is threaded by default, see the entry in the
tcl_platform array by doing:

tclsh
% parray tcl_platform

Michael


.

Relevant Pages

  • Re: Buffered IO and pipes
    ... > master with as little waiting around as possible, ... > slaves can get back to doing their computation. ... need to use setvbufto make a larger-than default buffer. ... > memory. ...
    (comp.unix.solaris)
  • Re: [RFC/PATCH 0/22] W1: sysfs, lifetime and other fixes
    ... defined in family driver. ... Bus master driver - is like NIC driver, it does not know about the rest ... It has it's logical slaves, it has it's own attributes and features, ... This is wrong - netlink notification is used and will be moved to ...
    (Linux-Kernel)
  • Re: The Grateful Slave
    ... I am told he has the power to control my life. ... My master must understand the world better than I, ... he says only he can protect my money. ... My master cares about other masters, who don't have good slaves. ...
    (rec.photo.digital)
  • Re: The Grateful Slave
    ... I am told he has the power to control my life. ... My master must understand the world better than I, ... he says only he can protect my money. ... My master cares about other masters, who don't have good slaves. ...
    (rec.photo.digital)
  • Re: Bind9 AXFR
    ... Can you elaborate (e.g. error messages in logs of both master and slaves)? ... logging there. ... When I attempt to do so bind9 quits. ...
    (comp.os.linux.setup)