Re: Static vs. Dynamic typing (big advantage or not)---WAS: c.programming: OOP and memory management

From: Jörn W. Janneck (jwjanneck)
Date: 07/16/04

Date: Thu, 15 Jul 2004 20:50:12 -0700

robert: terribly long post ahead... you made a few very interesting point,
so that's my lame excuse. :-)

Robert Strandh wrote:
> "Jörn W. Janneck" <jwjanneck at yahoo dot com> writes:
>> even *if* you were right, this would only explain the lack of success of
>> allegedly superior technology if you would also postulate an inordinate
>> amount of uniformity. because, you see, irrespective of whatever reason
>> people use to choose their tools, say even if people made completely
>> tool choices, the *market* favors them (the choosers) for their market
>> success.
> I am not sure I follow exactly what you mean, but you seem to be
> saying that even if a good tool is chosen from time to time, the
> market will realize this and prefer it to the others.

on average, and given some time, yes.

> Not so. First, the uniformity exists, partly because software is a
> "winner takes all" market where a slight initial advantage rapidly
> translates to market domination.

hm, it seems you *do* assume that there in fact *is* an initial advantage?

and how do you explain the veritable zoo of programming languages out there
by the "winner takes all" theory.

now, i won't deny that there are strong network effects in computing in
general, and definitely also for programming language choice/programming
tools. but given the observable fact that programing languages come and go,
sometimes within a decade or so, it wold seem as if the market has both
sufficient dynamics and sufficient variation to perform selection on.

> And, contrary to what people think,
> the market is not rational. In software, it is dominated by rumors,
> personal preferences, inertia, marketing hypes, etc. Combine that
> will the extraordinary psychological barriers against learning
> something new, and you have a pretty good picture of what is going on.

saying that the market is not rational is almost like saying that people are
not interested in money. remember, the products of the market we discuss
(programming languages and corresponding tools) are themselves tools, used
to produce other products (i.e. applications, operating systems, controller
software, ...) whose customers most likely could not care less about the
tool used to create them. and it is *those customers* that effectively speak
the ultimate verdict on what tool *users* are successful, and hence whether
tool *suppliers* produced sufficiently good products.

> Another contributing factor is that it is hard to measure
> productivity. Today, nobody can produce hard statistics to prove that
> a tool is superior to another. And unfortunately, the PDGs of today's
> software companies seem to want hard statistics. I claim that it is
> the job of a PDG to make decision based on intuition; otherwise he or
> she could be replaced by a spreadsheet program. But the PDGs in the
> software industry today usually do not have any intuition about
> productivity, simply because they have usually never worked in
> software themselves. Therefore, they must use other ways of making
> decisions, and you get the above list: rumors, personal preference,
> etc.

but you do not have to measure productivity differences in order for them to
exist and make a difference. what you say above may be a problem for
statisticians (and i agree, it is a very big one---software productivity
data is for the most part pretty useless), and it may also be a problem for
companies and the developers, but it is not a problem for customers---if you
can build a comparable car at half the price, you'll get my business. i
don't care whether you could *predict* that your tools would make you more
productive, or whether it was pure fool's luck that made you make the right
choice: your car costs less than the car of your competition, so i'll buy

and granted: for the consumer products as well, there are of course also
network effects that may superimpose onto the productivity differences, and
even overwhelm them. but that does not change that more productive is always
better than less productive. even a monopoly provider is interested in
producing the same features faster. of course, if you actually *do* have a
monopoly the pressure to worry about your productivity is lower, but it's
still a matter of having more or having less, and given that choice, most
people opt for more.

(but clearly, in the case of monopoly, they are no longer compelled to
choose, and so my market reasoning fails there. the only force that makes
them choose is their own greed, and it is quite possible that their
ignorance is greater than their greed.)

>> iow, the correlation between success and productivity is independent of
>> competence of the choosers.
> No, if most choosers are ignorant of better technology, then most will
> choose inferior technology, and if the market is incapable of fixing
> that, then it will remain that way.

true, but that is a lot of ifs. i do not believe that most choosers are
ignorant of better technology (at least not when you consider whole
companies, or perhaps divisions in them, as choosers, rather than individual
programmers), and "most" would actually not suffice---it would require
almost all choosers to be ignorant. i also don't think that the market is
incapable of picking more productive tools, because it is picking (new)
tools all the time, and there just does not seem to be a good reason why it
should make the choices it clearly does make in favor of less productive

but admittedly these are judgments, so we might have to agree to disagree

>> otoh, if you look at a single individual who is
>> successfully participating in the market for some time, it seems
>> to assume that the choices he or she made are not completely random.
> The choices are not random, but not the best either. Again, you trust
> the market to find the best tools, but unfortunately the market is not
> rational.

actually, i would not say that the market finds "the best" tools, in fact i
had pointed out already (somewhere else in this thread) that i think that in
a dynamic market you cannot expect the market leader to be the
technologically most advanced product *ever* (and if it happens to be so,
then only by pure chance).

but i do trust the market to make "pretty good" choices, and to tend towards
"better" ones over time. moreover (and this is how this subthread got
started), i suggested that if the market neglects to adopt a product for a
very long time, it would be a good idea to sit back and wonder whether there
is any technical property of the product that is responsible for this.

>> at least the reasoning i proposed was this: if something is superior,
>> this would imply that it makes its users (assuming sufficient variation
>> the market for it to have *any* users at all) more successful market
>> participants,
> This is true to some extent, but not completely, since the market
> chooses from other criteria.

the point however is this: either product A makes you more successful (for
whatever reason) or not. if it does, let's not call it "better", let's call
it "ghanfier," to remove the moral judgment from the term.

okay. what i am saying is this: being ghanfy is related to being popular.
over time, the market will select ghanfier products, because *by definition*
its users are more successful. someone producing something is well-advised
to choose tools according to their ghanfiness. in fact, ghanfiness is the
key criterion for picking a tool.

now it could be, of course, that ghanfiness and productivity are unrelated,
in which case its a stupid idea to use productivity to advertise a tool,
because it's like saying that it comes in a pretty box. however, i think
they are related, to the point of being almost the same thing, and i believe
that is also the premise under which people operate who use productivity to
advocate, for instance, the use of a specific language.

as far as i can see, there are only two objections to this:
(a) a property like ghanfiness does not exist, i.e. no objective property of
a tool is related to the success of its users, i.e. tool choice is
completely arbitrary.
(b) ghanfiness and productivity are unrelated.

i may have missed another objection, but that is how i see the landscape of
this argument.

at the core it is really about (a) advocating a language based on some
alleged productivity gains and *at the same time* proclaiming that (b)
productivity is independent of a product's success. the conjunction of those
two positions is what seems inconsistent to me.

>> thus presumably increasing the demand for it etc.
> This does not follow. Even in the case of individuals and companies
> that are successful because of superior technology, there is a
> tendency to reject that technology if it requires learning something
> unfamiliar. Take the case of Viaweb (Paul Graham) who managed to sell
> his system to Yahoo because of superior technology. The developers at
> Yahoo then (if I understood the situation right) spent many
> programmer-years to rewrite the system in an inferior programming
> language that they happened to know about. Pretty typical market
> behavior, actually.

i cannot say anything about that case, because i only know about from
graham's web site, and have no idea whether or how inferior their solution
was, or what was inferior about it. and of course, there will be lots of
specific examples where people postulate inferiority of a particular choice.
that does not mean they are right.

>> history is full of examples where a technology that multiplied
>> for some industry by a factor of 2 or more was developed and prouctized
>> not bought and used for a quarter of a century? after the industrial
>> revolution? or even in the it industry?
> Superiority is not just a matter of productivity. But yes, history is
> full of examples of better technology that lost because of marketing
> hype, inertia, rumors, etc. Betamax is the favorite example.

unavoidable in this kind of discussion. my understanding is that vhs was
ahead of betamax wrt recording time, and that this was what made people
choose it.

>> but how does that account for the fact that people learn new and (hence)
>> unfamiliar things in our industry all the time?
> Even performance-oriented people obviously have to learn something new
> from time to time, or else they will not be employed. And even though
> the barriers are strong, they can be overcome when the
> performance-oriented person realizes that learning is the only way
> out. But it takes extraordinary psychological effort to question
> something that already seem to work pretty well, in favor of something
> that might work better.

sure, but you are reasoning on the individual level. the fact that learning
is "the only way out" is a consequence of the market, learning is just the
process of individual adaption, and psychological barriers are just a
delaying factor for the individual, and perhaps also for the aggregate.

>> but if a company using a language that makes them twice or three times as
>> productive, improves their product quality, whatever, zooms past them in
>> market, and their choice would be to queue up for unemployment benefits
>> click on the smalltalk book at amazon and zoom along, i think they might
>> consider whether their personal preferences are the right guide for
>> that choice.
> This reasoning assumes that outsiders realize (or are willing to
> consider) the reason for the better productivity. That is not the
> case in general.

yes, but that's just an artefact of posing this issue in individual terms.
if of 100 unemployed programmers 20 click on the smalltalk book, 20 on the
java book, 20 on c++, 20 of haskell, and 20 on cobol, the market has
sufficient variation to pick from, even if all 100 choices were actually
made by their well-meaning grandmothers (i assume that these grandmothers
make their choices in ignorance of technical merits and do not intend to
offend any hacker-grandma---apologies if i did).

>> i think the basic issue i have with all these lines of argument is that
>> treat programming language choice as an exclusively personal matter
>> by nothing but individual taste. i don't think this is correct.
> That was not my intent. Such choices can be made on the company level
> by a well-informed PDG.

that sounds about right.

> I do not know Haskell well enough to agree that it would be
> unsuitable. But what you brought up is VERY INTERESTING because, as
> you wrote, decisions are made "based on whatever you know about it".
> The trouble here, is that people who are in a situation to make such
> choices are not necessarily well informed. Your Haskell example was
> interesting because I often hear people rejecting Lisp (I use Lisp as
> an example, because that is the example I am most familiar with)
> because of things they think they know, and that are completely false.

i agree with you on this assessment of lisp, but keep in mind: the reason
that people give for rejecting lisp, indeed, the whether they have any
rationalization at all or just dislike the fact that it starts with "L",
does not matter in the long run.

the paragraph you replied to was an attempt of mine at explaining why
choices that are made *appear* to be arbitrary (by suggesting that in the
scope in which they are perceived as choices they often in fact *are* pretty
much irrelevant). it's a side issue (albeit, i think an important and
interesting one), but does not affect the main point.

although perhaps the point you are making is that the same preselection-idea
can be used to explain a uniformity that is big enough to reduce the
available variation to a point where market forces have almost no leverage
to act on. is that it? yes, i suppose that is possible, it certainly occurs
for relatively short durations in localized markets, and it's conceivable
that something like this might stifle market forces even on larger scales in
market size and time. i really doubt it happens, though.

> Their psychological barriers make it so much more comfortable to
> reject something without being well informed. If you have to learn
> enough to make an informed decision, you might discover something
> better, that you would then have to learn. It suffices to look at
> some posts in this group about dynamic languages or about garbage
> collection to see this phenomenon. How much simpler to use
> disparaging arguments, or to ridicule some totally irrelevant aspect
> of some technology ("too many parentheses") than to learn about it.

true. this is very annoying if you are trying to introduce a language to an
audience, and very common.

> First, many instances of better technology never make it into the
> prefiltered selection, because they are rejected for silly or
> downright incorrect reasons. But it is true that the final choice is
> often made by some of the best programmers in a company. The trouble
> is that even the best programmers in a company do not often know to
> choose better technology that they have to learn first. In my
> experience, better languages never make it past the filter, usually
> because nobody in the company knows it.

i would even add another force to this: the "best programmers" in a company
are by definition the best in using the old technology. it wouldn't be
surprising that this in itself makes them opposed to new things that are not
seen as supporting or extending the old stuff, either because of some
"personal attachment" or for the very concrete reason that adopting a new
technology would put them on a more equal footing with the rest of the
crowd---will they also be the best afterwards?

> But rejecting some better
> choice for that reason is making implicit assumptions such as:
> * training the programmers is more expensive than the gain in
> productivity.
> * hiring new programmers for the new task is more expensive than the
> gain in productivity.
> And that is exactly what the problem is. Better technology is
> systematically rejected for non-rational reasons.

i would call this kind of reasoning "conservative," rather than
"non-rational." essentially, people need to weigh a number of unknowns with
very high variability. they are taking a bet either way, and conservative
risk taking usually errs on the side of the status quo.

that is btw why i think that in an evolving market (i.e. where technology
changes fast enough to essentially induce constant change) the market leader
will never be the technologically "best" product, simply because of the lag
induced by conservative adoption.

>> but the expected value of such a bet diminishes with the increasing
>> differences between contending tools. i find it unlikely that an it shop
>> would leave >2x productivity gain on the table for several decades,
>> for some competitor to pick it up and run with it.
> Such reasoning is what creates the current state of the market. After
> all, it suffices to look at some large software company and check out
> what THEY did. Since THEY obviously made a rational choice, then WE
> should make the same.

that's not quite the same. the large company is large because it made the
right choice(s) a long time, sometimes decades ago. that does not make it a
good choice right now, indeed it's likely that it is a choice many
competitors have already made.

a better gauge would be to look at relatively new companies that grow fast.
but here, too, you basically get a look at the effect of "old" choices, just
not quite as old.

> Your position is psychologically very comfortable. As long as you
> believe that it is unlikely that an IT shop would let that happen, you
> can pretty safely assume that your current choice is not too bad.

actually, what i said above is not quite right---*an* it shop can easily
miss an opportunity for a *very* long time, in fact that does happen quite a
bit. what i meant was that the totality of shops, i.e. the market, is very
unlikely to miss a good opportunity for very long. in fact, that is almost a
tautology, because one could reasonably *define* a good opportunity to be
something that a market is unlikely to miss for very long.

> This saves you a lot of time investigating something that might be
> better. The trouble is, most others do the same so nothing ever
> changes.

first of all, i think it is an observable fact that things in our industry
change all the time. secondly, i think the reverse reasoning is closer to
the truth: as a participant in a market in which you suspect new
opportunities to turn up very often, you need to look out for them all the
time, because you must assume that this is what your competitiors are doing.
and there goes the psychological comfort... in fact, i believe that this is
in part what is driving the *spread* of innovation, viz. the almost paranoid
observation of competing companies, their processes, marketing decisions,
technology adoption, everything.

long post, sorry. ;-)

-- j