Re: Fastcode MM B&V 0.41

Hello Pierre,

> As the results grid grows, the memory usage scores of the MMs
> drop... The first MM in a benchmark run has a slight advantage, because
> the
> B&V tool uses less memory at that stage.

We clearly don't want that, but are you sure this is significant?
If you run a benchmark batch with MMs in the order BucketMM first ...
FastMM3 last, and then another in the order FastMM3 first ... BucketMM last,
what's the order of magnitude of the change?

The easiest solution to this problem is to use a batch file in which all MMs
are run twice, the second time in inverse order. Like the following for 4
MMs:
MM1-MM2-MM3-MM4-MM4-MM3-MM2-MM1
In that way all MMs would be handled equally.

> I think we should revisit the way in which we calculate the memory usage
> score. I would propose some kind of sliding scale: A benchmark
> that uses a lot of memory should have a higher memory usage
> weight than one that doesn't.

Using maximum memory usage as a "weight" factor does not seem the right
thing to do. It's not because a benchmark uses more memory that it's more
significant... memory usage is very often just a matter of scaling.
IMO the only benchmarks for which we should so some adjustment is the replay
benchmark. There seems to be a constant overhead of around 28 MB, which we
could subtract from the final memory usage.

Kind regards,
Robert


.

Relevant Pages

  • Re: sparse polynomial arithmetic
    ... Fateman's benchmark, double-precision coefficients (16-bit ... memory usage, then probably the hash table gets resized but the memory ...
    (sci.math.symbolic)
  • Re: Fastcode MM B&V 0.41
    ... > role than the execution order. ... If you leave the system alone, the memory usage scores stays ... Opening up other programs during a benchmark is not ...
    (borland.public.delphi.language.basm)
  • Re: jruby vs mri memory management
    ... Linux 32bit when we benchmarked the Ruby Quiz #157 submissions, ... was not impressed (I didn't check the memory usage though): ... you just had to launch the benchmark to get it. ... JVM). ...
    (comp.lang.ruby)
  • Excessive Memory Usage with libxml
    ... I've got some basic functionality going, it's not finished, and when ... When I run the benchmark I get weird things happening. ... 10000 iterations were fine ... The memory usage though was stupid at 235Mb Real and 233Mb Virtual ...
    (comp.lang.ruby)
  • Re: MM B&V - ManyThreadsTest
    ... if this is something you feel some MMs would fail then I'd rather ... benchmark to run ones before we actully start counting..) ... > Hi Pierre, ... >> bytes and does this a million times in a loop (using threads again if you ...
    (borland.public.delphi.language.basm)