Re: MM B&V Benchmark weights

From: Pierre le Riche (pleriche_at_hotmail.com)
Date: 02/24/05

  • Next message: Robert Houdart: "Re: MM B&V Benchmark weights" 
    Date: Thu, 24 Feb 2005 14:17:04 +0200

    Hi Robert,

    > For simplicity's sake, I suggest that we stick to the original formula.

    Then some benchmarks need to be fixed to bring out the intended scoring bias
    towards either speed or memory size. Many of the benchmarks are so short
    that the variance in the CPU time is too large for accurate measurements.

    For an application that never uses more than for example 64MB, memory usage
    should rarely be an issue provided that the usage does not grow unbounded.
    For such applications raw speed is by far the biggest consideration.

    Conversely, for applications that moves around large chunks of data the
    memory efficiency of the MM may be a bigger factor (avoiding disk swapping,
    etc.).

    Some MMs have an initial overhead of lookup tables, etc. I feel those MMs
    are unfairly penalised throughout all the benchmarks for this cost -
    especially in benchmarks that allocate very little memory. These lookup
    tables are a one-time cost and should not weigh as heavily in the scoring as
    they do. In situations where memory consumption is important (i.e. in
    benchmarks that allocate a lot of memory) this one-time cost is usually
    insignificant, yet these MMs are penalised in benchmarks where memory
    consumption is largely irrelevant.

    I had a look at your replay benchmarks. They allocate in the region of 50MB
    maximum. Surely you wouldn't mind if usage was 20% more if it got you an
    extra 10% in speed? If it allocated 1GB then it would have been an entirely
    different situation and you would probably want to sacrifice some speed for
    a significant reduction in memory usage. Unfortunately our scoring system
    does not accommodate this.

    So to summarise: Either we must drop the "relative performance" score, and
    work with two separate scores - the address space usage and the speed, or we
    must do a serious rethink of the way in which the relative performance is
    calculated.

    Having each benchmark calculate its own "relative performance" is still the
    best option IMO. We could have the benchmark pick a weight between say 30%
    and 70% for the memory usage and then the speed comprises the rest of the
    score.

    If there are no objections, I'll do this tonight. If we cannot agree on a
    weighting for a particular benchmark, then we leave it at 50%/50%.

    Regards,
    Pierre

  • Next message: Robert Houdart: "Re: MM B&V Benchmark weights"

    Relevant Pages

    • Re: Fastcode Memory Manager B&V 0.02
      ... > have a vested interest and I may be accused of doctoring the benchmarks to ... can be heavily influenced by the state of the memory, ... it's only decent when running the benchmark twice outside of the IDE and ... reduce the memory usage on common benchmarks ...
      (borland.public.delphi.language.basm)
    • Re: forth and virtual memory
      ... too, maybe even the same order, so ordering the blocks by allocation ... on systems with too little memory ... What Java is known for, and what it actually does, are two distinct ... My measurements indicate that some of the benchmarks (from SpecJVM98, ...
      (comp.lang.forth)
    • Re: [PATCH 0/11] Avoiding fragmentation with subzone groupings v26
      ... a standard kernel was getting < 1% of memory as large ... I don't have a list of real workloads that break anti-frag yet so so I want to get anti-frag out there and see does it help people who really care about hugepages or not. ... Small differences in aim9 seem to make very little difference to other benchmarks like kbuild. ... echo Attempting load of hugetlbfs module ...
      (Linux-Kernel)
    • Re: 2 servers, identical hardware, different speed
      ... I have 2 _nearly_ identical servers that, ... kernel compiles) turned out to perform VERY differently (one is 4 or ... Check that all usable memory listed in the BIOS-e820 map ... The benchmarks I run didn't really test memory and were not memory ...
      (comp.os.linux.hardware)
    • Re: MM managers and win98 compatibility
      ... The following are my test results under Windows 98 SE. ... should give a good starting point for the authors of the different memory ... The following memory managers didn't pass all validations: ... Now, regarding benchmarks run under Windows 98 SE, the following memory ...
      (borland.public.delphi.language.basm)