Re: Can I measure how much memory an object consumes?

On 2008-06-03, Mathias Dahl <mathias.dahl@xxxxxxxxx> wrote:
Now I am thinking of logging the latest X requests in a global
variable, to make debugging easy if error reports from users comes
in. I was thinking of maybe saving the last 100 or so, but due to the
limited memory I have, I would like to know if there is a way for me
to calculate or measure how much memory these 100 request objects will
consume, to see if this is feasible. Is there? Other than running
`top' and watching the memory consumption increase, that is.

You could use graph.lisp (linked below), which is an SBCL-specific
answer that I wrote some time ago. (It's a quick copy&paste hack based
on other code, so don't expect its source code to be particularly nice.)


Here is why I think graph.lisp is a helpful approach to this:

When someone asks the kind of question you posted, the usual
recommendation is to allocate a large number of objects instead and
observe the effects on memory allocation, e.g. using ROOM and TIME.

How does that approach differ from your original question about the
exact memory requirements of a particular number of objects?

One commonly cited difficulty is the definition of "object". For
example, there's a good chance that all your CLOS objects have constant
size: At least in SBCL, the actual data is stored in a separate vector,
which the "CLOS object" only points to.

But when you asked about the size of those objects, you probably wanted
to include the size of that vector.

Once you realize that the total size of the "object" includes the sizes
of other "objects" referenced from it, you are actually measuring the
size of a graph of objects, and the question becomes: "How deep into
which parts of the graph do you want to recurse?".

There is no answer to that question that will always be helpful -- it
depends on what you'd like to achieve (just as with the similar problems
of equality or copying of objects).

But there are some heuristics which yield good results. In your case:

- you'd like to follow the CLOS instance to its data vector

- you don't want to include the class object, which is referenced from
the CLOS instance (in SBCL's case, the LAYOUT)

- you probably don't want to include symbols, function, packages and
similar objects either

You'll also have to think about shared structure, so the size of the
graph starting at that list of 100 requests is possibly smaller than the
sum of the sizes of 100 one-element lists of one request each.


Based on all this, my approach is to draw a graph of the objects, and
let the user discover the difficult questions himself.

So for the objects you asked about, we would just draw the CLOS instance
and its data vector separately, exposing this implementation detail to
the user instead of covering it up.

We still need the heuristics mentioned above to keep the graph
reasonably small, since we could easily plot the entire lisp image if we
weren't careful here.

It's usually hopeless to try and plot even a single package with all its
symbols. But just as the traditional box&pointer diagrams for lists,
there pictures help explain the subject matter quite well, I think, and
once you've got an idea what the Lisp does under the hood, you'll also
feel more comfortable with the numbers ROOM and TIME give.

Here are some examples (object size in bytes):

* http://www.lichteblau.com/tmp/dot/pathname.png

A pathname, illustrating that we follow neither the layout to the
structure class nor the symbols.

* http://www.lichteblau.com/tmp/dot/defstruct-vs-defclass.png

An illustration of the difference between the implementation of
structure classes and standard classes.

* http://www.lichteblau.com/tmp/dot/package-empty.png

An empty package (as mentioned above, anything larger wouldn't be
very readable).

* http://www.lichteblau.com/tmp/dot/function-print.png

A larger example, showing the code component that #'print is a part
of.

The code is here:
http://www.lichteblau.com/tmp/dot/graph.lisp
http://www.lichteblau.com/tmp/dot/test.lisp

An installation of graphviz is required.

I am using the latest SBCL available in Ubuntu 8.04.

Don't know whether it still works with that version, but there's a good
chance that it does.


d.
.

Relevant Pages

  • Re: How do I get Graph to be "within" Word?
    ... Go to the Memory pane and raise Word's Preferred memory to ... Add a little bit more RAM to the preferred RAM setting for Word in ... >> I did exactly as you said and NOW I CAN EDIT GRAPHS! ... >> graph, place it, work on other graphs, but then when I go back to edit ...
    (microsoft.public.mac.office.word)
  • Tracking memory usage and object life time.
    ... I have programmed some python script that loads a graph (the ... memory usage of Python in top, I see it starts out with around 80 MB ...
    (comp.lang.python)
  • Re: Larry Wall, on Tcl
    ... If DAG means Directed Acyclic Graph, ... Yes, your are right the object graph isn't a DAG, ... "When the garbage collector starts running, ... thinking about memory allocation and cleanup. ...
    (comp.lang.tcl)
  • Re: OOP and memory management
    ... > reference counters such as those in boost.org work perfectly well. ... in which case your memory footprint can grow and grow... ... > When certain parts of the graph are unreachable, ... slower than for ordinary pointers), ...
    (comp.programming)