Re: Fastcode MM: category weights

After looking at first results of the benchmark I also made the assumption
that weights had a lot to do with the final result and I was wondering
whether it would be possible to split the results in two or three broad
categories (results for single treaded procedures, results for multithreaded
procedure and maybe a blended result for both).

Another suggestion would be for the inclusion of some qualitative statements
on different MM. For example, I noticed that BucketMM and FastMM3 seems to
perform best on different benchmarks. I don't know whether it would be
possible to find patterns in those benchmarks allowing someone to stress the
strength and weekness (Pros and Cons) of the major contenders. One could
say for example, that BucketMM is best for such kind of application, but
FASTMM3 should provide better performance for another kind of application or
memory operations. This would allow people to figure out which MM will
likely give better performance on the kind of application they have. I
personally did some tests on my P4 on a routine to create an inverted index
of words in documents, and then compare documents based on this index.
While both of the above MM were much better than the default one, FastMM was
at least 15% faster than BucketMM. I could hardly have guessed that from
the global result of such a benchmark.

Richard

"Pierre le Riche" <pleriche@xxxxxxxxxxx> wrote in message
news:427b3f8e@xxxxxxxxxxxxxxxxxxxxxxxxx
> Hi All,
>
> I think we should adjust the weights of the individual categories. Here is
> my suggestion (with reasons):
>
> {PLR 6/5/2005 changed weights: Multi-threaded weights were IMO
> disproportionately high vs. single threaded. Having multithreaded
> benchmarks
> carry more than 50% more weight than single-threaded benchmarks is not
> realistic?! Should it be exactly balanced?}
> WeightPerCategory: array[TBenchmarkCategory] of Double = (
> {If your application reallocs a lot in speed-critical code then it is
> badly
> written (hence the low weight)}
> 0.08, //bmSingleThreadRealloc
> 0.12, //bmMultiThreadRealloc
> {Alloc and free is the typical behaviour of most apps}
> 0.10, //bmSingleThreadAllocAndFree
> 0.15, //bmMultiThreadAllocAndFree
> {Replays count the most (this is the closest to real-world behaviour
> we've
> got at the moment, although memory is not actually "used")}
> 0.20, //bmSingleThreadReplay
> 0.30, //bmMultiThreadReplay
> {Some anomalies with some of the current speed tests precludes using a
> bigger weight.}
> 0.05 //bmMemoryAccessSpeed
> );
>
> Comments?
>
> Regards,
> Pierre
>


.

Relevant Pages

  • Re: Gforth tops the shootout
    ... >This is due to the shootout changing their default sort order to weight ... >lines-of-code and memory usage equally with execution speed, ... Looking at the benchmarks, I see that you have contributed many. ...
    (comp.lang.forth)
  • Fastcode MM: category weights
    ... disproportionately high vs. single threaded. ... carry more than 50% more weight than single-threaded benchmarks is not ... {Alloc and free is the typical behaviour of most apps} ...
    (borland.public.delphi.language.basm)
  • Re: Fastcode MM Rule
    ... I would like to suggest that we create more variations of the MemUsage ... benchmarks and perhaps increase their weight. ...
    (borland.public.delphi.language.basm)
  • Fastcode memory Manager B&V 1.1.3
    ... The only change is some clean up of the batch ... files, including adding back FastMM3. ... I've also posted a new copy of ProgressReport. ... one or more benchmarks, then that MM is marked ass Failed or Incomplete. ...
    (borland.public.delphi.language.basm)
  • Re: MM B&V - ManyThreadsTest
    ... First of all, for the benefit of others less familiar with BucketMM, let me ... Most Delphi developers tend to be rather careless about string ... benchmarks and starting a hopefully long series of Replay Benchmarks. ...
    (borland.public.delphi.language.basm)