Help: JMX monitors lead to java.lang.OutOfMemoryError because of many threads

From: Andre Ranvik (google_at_ranvik.com)
Date: 12/30/04 

Date: 30 Dec 2004 01:24:03 -0800

I need to monitor approximately 100000 nodes in a system, where each
has around 50 fields that are required to be monitored (total of 50 *
100000 fields). The granularity period (time between checking each
field) needs to be no greater than 2 seconds.
JMX has been proposed to be used, but it seems like it is not capable
of monitoring as many fields.

What I have tried is to use both J2SE5 and MX4J. J2SE5 starts up a new
thread for each monitor, and runds quickly out of memory. I have
tried to set the -Xss startup option, but with only minor improvement.
MX4J uses a fixed number of threads, but it slows down to a crawl when
going beyond 10000 fields.

A solution seems to use a thread pool, where a fixed set of threads
would monitor X number of fields each, but this would require me to
write my own monitor MBeans, something that I want to avoid. The
monitor I am using is the GaugeMonitor for all fields.

Does anybody have a better approach for how to monitor these fields?

Thanks,
Andre Ranvik