Re: Parallel Common-Lisp with at least 64 processors?

usenet2.3.CalRobert@xxxxxxxxxxxxxxx (Robert Maas, http://tinyurl.com/uh3t) writes:

From: p...@xxxxxxxxxxxxxxxxx (Pascal J. Bourguignon)
On a linux system you can have a look at /proc/cpuinfo.

That does't help me on my shell account running FreeBSD Unix.

cat /proc/cpuinfo

% more /proc/cpuinfo
/proc/cpuinfo: No such file or directory

To know the load of a system, the right tool is obviously:
loadavg

% loadavg
loadavg: Command not found.

well, it may not exist (anymore) in which case, again obviously:
uptime

% uptime
5:01AM up 3 days, 25 mins, 9 users, load averages: 5.11, 6.07, 6.49
(Oh oh, there's that cliche about third time ... charm?)
Anyway, with load around 5 or 6 it looks too busy for me to run any
major CPU-speed benchmark.

Well, 5:00 AM, is often the time at which the last cron tasks of the
night are just started. Try it at another random time.


would give you the load average such as:
14:44:30 up 4:39, 2 users, load average: 0.05, 0.04, 0.13

I don't know what machine gave you *those* numbers, but that
machine is sorely underutilized?

Not all the time :-) Also it's a quad-core, so the average is about
divided by four, compared to a normal processor.



And if you want to be nice to the other users, then just that, be
nice, using the nice(1) command:
nice my-big-cpu-very-intensive-program

Since all my heavy computations are from a single CMUCL REP, I
guess I need to say "nice lisp" when starting up in the morning,
and leave it that way all day until I'm ready to (quit) and go to
bed?

Not necessarily. But you can only increase the nicety level of your processes.

$ clisp -x '(loop :for i :from 0 :do (* i i))' >/dev/null 2>&1 &
[2] 13589
$ renice +4 13589
13589: old priority 0, new priority 4
$ renice +6 13589
13589: old priority 4, new priority 6
$ renice +2 13589
renice: 13589: setpriority: Permission denied
$ renice +5 13589
renice: 13589: setpriority: Permission denied
$ renice +6 13589
13589: old priority 6, new priority 6
$ renice +7 13589
13589: old priority 6, new priority 7

Only root can reduce it.

So you could start with a normal nicety, do your loading and
compilation full speed, and once you launch your computation, you can
detach your process and renice it.


Another tool you may use is batch(1), which enqueues jobs, and
will start them only when the load average is low enough.

That's worthless for my purpose, unless you know a way to use it to
deal with interactive Lisp session.

Indeed, that would be useful only if you could cut your task in
smaller batches that could be run independently and without
supervision.

If you had a lot of little tasks to execute, you could use batch.

If I have a lot of little tasks to execute, I put them into a PROGN
or LIST, and if I'm going to do the same sequence of tasks many
times over different days then I might put them into a DEFUN.
For example, here is the LIST of tasks I do when I first start up CMUCL:
(list
(setq *gc-verbose* nil)
(unless (fboundp 'funct+shortdir+fn1-mayload)
(load "/home/users/rem/LispWork/2007-2-mayload.lisp"))
(funct+shortdir+fn1-mayload 'filenamebase-std-roll-before :LISP "2007-2-roll")
(funct+shortdir+fn1-mayload 'load-file-by-method :LISP "2005-8-readers")
(funct+shortdir+fn1-mayload 'dirspec+filnam+method+globsym-may-load :LISP "2008-4-MayLoad")
(funct+shortdir+fn1-mayload 'make-empty-heap :LISP "2001-B-heap")
(funct+shortdir+fn1-mayload 'string-read-words-batch-sort :LISP "2008-3-WordHist")
(funct+shortdir+fn1-mayload 'phvec-normalize :LISP "2008-3-ProxHash")
(funct+shortdir+fn1-mayload 'device-to-ar+mr+mc :LISP "2008-3-TextGraphics")
(funct+shortdir+fn1-mayload 'trans-skills-lines :LISP "2008-3-TopPH")
(funct+shortdir+fn1-mayload 'trans-skills-3d-012-links :LISP "2008-5-TopPH")
)
I don't see how batch could possibly help me with those tasks.

You could fork a lisp process for each of these tasks, suspend them by
sending them a STOP signal, and batch a kill -CONT $PID command to
resume them when the system load is low enough for the batch commands
to be run.

;; (pseudo-code, I don't have cmucl installed on this computer to check all of it).

(progn (setq *gc-verbose* nil)
(unless (fboundp 'funct+shortdir+fn1-mayload)
(load "/home/users/rem/LispWork/2007-2-mayload.lisp"))
(loop
:with pid = 0
:for task :in (list
(lambda () (funct+shortdir+fn1-mayload 'filenamebase-std-roll-before :LISP "2007-2-roll"))
(lambda () (funct+shortdir+fn1-mayload 'load-file-by-method :LISP "2005-8-readers"))
(lambda () (funct+shortdir+fn1-mayload 'dirspec+filnam+method+globsym-may-load :LISP "2008-4-MayLoad"))
(lambda () (funct+shortdir+fn1-mayload 'make-empty-heap :LISP "2001-B-heap"))
(lambda () (funct+shortdir+fn1-mayload 'string-read-words-batch-sort :LISP "2008-3-WordHist"))
(lambda () (funct+shortdir+fn1-mayload 'phvec-normalize :LISP "2008-3-ProxHash"))
(lambda () (funct+shortdir+fn1-mayload 'device-to-ar+mr+mc :LISP "2008-3-TextGraphics"))
(lambda () (funct+shortdir+fn1-mayload 'trans-skills-lines :LISP "2008-3-TopPH"))
(lambda () (funct+shortdir+fn1-mayload 'trans-skills-3d-012-links :LISP "2008-5-TopPH")))
:do (if (zerop (setf pid (unix:unix-fork)))
(progn
(unix:unix-kill 0 (unix:unix-signal-number :sigsstop))
(funcall task)
(unix:unix-exit 0))
(progn
(sleep 1)
(with-open-stream (cmd (ext:run-program "batch" :input :stream))
(format t "kill -CONT ~A~%" pid))))))


But you'll probably get better result with a nice process. It doesn't
run slower, but is only less prioritary that the other users'
processes.

In any case, for long running computing processes, you should probably
try to set up things to avoid a need for interactive use once the
computations are started. Save the results (and intermediary results)
to files, to be able to restart from new processes, etc. A good model
would be the SETI@home
Have a look at Boinc too: http://en.wikipedia.org/wiki/BOINCProject.

--
__Pascal Bourguignon__
.

Relevant Pages

  • Couple of (re)nice questions
    ... Is it possible for root to increse the range of nice/renice from -10 ... I notice that a normal user can lower the priority but not raise it, ... user@work:~> renice 19 12345 ... I remotely raised the nice level of a CPU intensive process to -5. ...
    (comp.os.linux.misc)
  • Nice Renice Verynice
    ... UNIX-like operating systems. ... Priority) I can do a "renice -15 -u username" which works fine for ... is started under that same username the nice level reverts to back to ... How do I permanemtly alter the priority for everything a ...
    (freebsd-questions)
  • Re: Gracefully logging off another user.
    ... If you want to go the renice way, only root can renice processes to give ... That will give the processes lowest priority. ... With the command ... With the suspend way it would be the command ...
    (Ubuntu)
  • Re: Is there a way to manage program priorities in Leopard?
    ... this would reduce by 1 the priority of process <pid> ... This command will increase the priority. ... According to the man page for renice, ...
    (comp.sys.mac.system)
  • Re: How-to reprio gcc (by default)?
    ... This may grab some interest from those running dog-slow servers ... setting the sendmail priority from a nice of 0 down to 7. ... renice every compile!) ... I've got another system-tuning question, ...
    (freebsd-questions)