Re: Fastcode Memory Manager 1.1.1 Summary posted

> > The reason I consider the D2006 benchmarks to be optional is that
(as
> > far as I know) the full set of benchmarks has only been run on the
base
> > compiler since validations on multiple compilers began. Plus,
there's
>
> Which has been wrong all the time.

When I voted to validate against D6, D7 and D2005 and only benchmark
against D2005, it was a conscious decision to only benchmark on D2005.


> > > Delphi 2005 is the base compiler for the benchmarks. D6, D7, D2005
> > > and D2006 are base compilers for validation.
>
> > the matter of Delphi 2006 only being released recently, and I wonder
> > about adding it as a requirement to an ongoing challenge. So, if
all
>
> That is a fair question. I hope that the community will settle this
> imidiately. It is high time to finish the 2005 competition and move
on.

Agreed.


> > validations and benchmarks need to be run for D2006 as well as
D2005,
> > do they also need to be run for D6 and D7? What was the purpose of
>
> > > Delphi 2005 is the base compiler for the benchmarks. D6, D7, D2005
> > > and D2006 are base compilers for validation.
>
> > having a base compiler in the first place? If this has already been
>
> > > Delphi 2005 is the base compiler for the benchmarks. D6, D7, D2005
> > > and D2006 are base compilers for validation.
>
> > discussed at length, can someone please point me to it?
>
> I think that it has been debated and settled in this NG a long time
ago.
>
> Is it published at our rules page?

If it was, then it's contrary to what I voted for regarding validations
and benchmarks and I don't see it on the rules page. Please correct me
if I'm wrong.

The reason I keep asking you to point me to the discussion where things
like this have been decided is that I have either completely missed the
discussion or the rules are changing without enough discussion.


> > The validations and benchmarks take a significant amount of time.
> > Days, in fact, and this will increase with additional benchmarks. I
> > can't lose the use of any of my machines for days on end, and am m a
>
> Which only means that you cannot complete all the validation and
benchmarks.
>
> > little concerned about having them pegged at 100% for a lot of that
> > time.
>
> How concerned ??

For one thing, at 100% they're generating a lot of heat. After
literally days of testing, I'm concerned about damaging the systems.


> >If other people are concerned about the same thing, then fewer
> > people will run the tests, reducing the number of different machines
> > and operating systems that are tested.
>
> Nope. People are free to run any subset of the validations. Nobody are
> forced to run the entire suite.

If you combine individual benchmarks that are run on different systems
that run at different speeds, this will skew the benchmark results.

> > I argue that this will impact quality and verifiability.
>
> I do not understand. If we run only a subset of the validations then
quality
> is not secured. Building a validation system that is unable to run all
> validations is no good.

I'm not saying

>
> >For example, the person who submitted the
> > Athlon X2 benchmarks didn't run all of the validations. I think the
> > results will have to be removed from the final score.
>
> No. If all MM's were benchmarked on that system, then results are
valid.

Even if the validations haven't been run?


> > I certainly don't intend to ignore community decisions. I
apologize if
> > I've missed where some of these were made and would appreciate
someone
>
> My problem is that I have told you all this so many times. The
community is
> so silent. Could we please settle it now.

If it was discussed and voted on, then that's fine. I still need to
see the discussion. If not, then some of these need to be voted on. I
have the feeling that the rules change every time I turn around.


> > pointing me to the discussions. Think of these as suggestions where
> > I'm asking for further comments and discussion. And I like to
think of
> > myself as part of the community, at least a little. I hope that
means
> > I get to contribute to some of these decisions.
>
> You are a fully member of the community. Just as I am. Eventhough my
ego
> tells me that I am a little more member than you ;-)

I don't care about egos except that they get in the way.


> > I've written this before, but I'll sum it up here again.
> >
> > I'm concerned that making any tests difficult to run will make it
> > harder to get volunteers to run them properly or completely. It
also
> > makes it harder to test the B&V itself, which might mean that
> > volunteers who run in to problems have to run them more than once.
>
> We should take no shortcuts that decrease quality. The project is
what it is
> because we have always been willing to do whatever work needed.
>
> You miss the fact that testing has granularity. The entire test suite
is
> huge, but anyone can contribute with as little as one basic
validation of
> one MM on one OS, on one.....

This is one of the points we don't agree on and I think needs to be
discussed.

> > I think that at least some of the testing can be streamlined without
> > sacrificing quality. I'll give the example of the DLL validation
that
> > simply loads a single DLL, performs an operation on it and unloads
it
> > 100 times.
>
> An MM has state. Therefore test number 2 is different from number 1.
The dll
> test is very short compared to all the other tests and I would spend
my time
> running tests and making spreadsheets instead of trying to change it.
>
> > If you completely disagree and insist that the testing has to be
done a
> > certain way, that's fine, too. I'll fix any problems found with the
> > B&V and report on any results that are submitted.
>
> Fine. We should hear the community opinion too. This looks like a
discussion
> between you and I and in fact it should be a community discussion.

Agreed.

> > Ultimately, I just want to help make the testing easier so that more
> > testing gets done so Borland includes more Fastcode in Delphi so I
see
> > the benefit in the next and all subsequent releases.
>
> Fully agree. Just do not make it easier by leaving out tests.

Or by being smart about what we run and arriving at the same conclusion.

This is getting long. Maybe we should start separate threads for the
different subjects. Don't think of this as changing the rules. Think
of it as nailing them down and getting them documented in one place.

--
Regards,
Bruce McGee
Glooscap Software
.