Re: Lisp and very large arrays
- From: Ari Johnson <iamtheari@xxxxxxxxx>
- Date: Wed, 23 Apr 2008 17:49:52 -0500
wrf3@xxxxxxxxxxxxxxx (Bob Felts) writes:
I have a friend who took a Perl script and recoded it in C for a huge
gain in performance. I've been telling him how much I like Lisp and, in
an attempt to do a little bit of Lisp evangelism, I offered to recode
his C into Lisp.
And I've hit a brick wall. The program needs four large arrays: two
arrays of (* 128 1024 1024) elements of unsigned bytes, and two arrays
of (* 128 1024 1024) elements of type bit.
LispWorks Personal 5.0.2 on Mac OS X won't even create the arrays;
they're too large.
sbcl 1.0.15.13 on Mac OS X (10.5.2, Intel) gives inconsistent behavior.
If I do:
(defun main ( )
(let* ((size (* 128 1024 1024)))
(array1 (make-array size
:element-type 'unsigned-byte
:initial-element 0))
(array2 (make-array size
:element-type 'bit
:initial-element 0))
(array3 ...
(array4 ...))
...))
then the code essentially hangs (it ran for 40 minutes before I killed
it.)
Ok, so maybe there's a problem creating large arrays inside a function.
Experimentation from the sbcl prompt showed that:
(defparameter *array1* (make-array size ...))
worked. All four arrays could be made.
So I moved the arrays outside of main like this:
(defparameter *array1* nil)
...
(defun main ...
(setf *array1* (make-array size ...)))
[The arrays are really dynamically sized, based on a parameter to main.]
This runs, once.
If I invoke main again, it hangs. Ok, maybe it's collecting garbage.
Let's see if we can help it out:
(defun main ...
(setf *array1* nil) ; for all 4 arrays
(setf *array1* (make-array size ...))
...)
Still no luck. Inserting a call to (gc) doesn't help.
At this point, the only way I can develop this code is to incrementally
develop the next part, run it, quit Aquamacs/slime, possibly kill sbcl
from the terminal, restart Aquamacs/slime, lather, rinse, repeat.
This isn't helping me to sing the praises of Lisp. Anyone been through
this before and have any helpful suggestions?
Others have mentioned (UNSIGNED-BYTE 8) and array size limits. For
what it's worth, from SBCL genesis.lisp:
;;; a big vector, implemented as a vector of SMALLVECs
;;;
;;; KLUDGE: This implementation seems portable enough for our
;;; purposes, since realistically every modern implementation is
;;; likely to support vectors of at least 2^16 elements. But if you're
;;; masochistic enough to read this far into the contortions imposed
;;; on us by ANSI and the Lisp community, for daring to use the
;;; abstraction of a large linearly addressable memory space, which is
;;; after all only directly supported by the underlying hardware of at
;;; least 99% of the general-purpose computers in use today, then you
;;; may be titillated to hear that in fact this code isn't really
;;; portable, because as of sbcl-0.7.4 we need somewhat more than
;;; 16Mbytes to represent a core, and ANSI only guarantees that
;;; ARRAY-DIMENSION-LIMIT is not less than 1024. -- WHN 2002-06-13
In other words, to be portable at least in practice, SBCL's
genesis-core-building code doesn't trust the underlying Common Lisp
implementation to provide arrays large enough. Note that SMALLVEC is
just `(SIMPLE-ARRAY (UNSIGNED-BYTE 8) (,(expt 2 16))).
.
- References:
- Lisp and very large arrays
- From: Bob Felts
- Lisp and very large arrays
- Prev by Date: Re: tomcat without apache
- Next by Date: Re: Objects of type FUNCTION can't be dumped into fasl files.
- Previous by thread: Re: Lisp and very large arrays
- Next by thread: postmodern thread safety in prepared statements functionality
- Index(es):
Relevant Pages
|
Loading
