Re: Linux, X, ld, gcc, linking, shared libraries and stuff

Wannabee wrote:
> Johannes Kroll wrote:
> > The examples you posted are called "dynamic loading". "Lazy binding" is
> > something completely different...
>
> Yes, oh well, thanks for the information Kroll. Maybe you could tell me
> what it is too ?

Well, "lazy binding" is also a type of "dynamic loading"...an "alternative
way" to do it...

[ In the following, wherever I say "DLL", this can also mean "shared
library" too...but, you know, rather than write "DLL / shared library"
throughout, I'll use "DLL" in a "generic" way :) ]

When the OS loads a program, with "non-lazy binding", then it also loads in
all the dependent DLLs (and all their dependent DLLs and all their
dependent DLLs and so on ;) and proceeds to attempt to "resolve" all the
"references" between them all...it does the whole lot in one go, there and
then, while loading the program...

With "lazy binding", the idea is to do this more "casually" or
"lazily"...when the OS loads the program, it does not bother to load the
DLLs or anything (though it still has all the "header information" about
what DLLs the program uses and stuff :)...instead, it just runs the
program...when the program first attempts to access any routine from that
DLL, though, this triggers the OS to actual load and link that DLL...hence,
there is a "load-and-link on demand" thing going on...it does it when it
needs to do it - when "references" are actually made - rather than doing it
"all at once" whilst loading the program...

What's the advantage in this? Often, not much..."lazy binding" will cause a
"delay" on that first reference to a DLL because, obviously, that's where
all the "loading and linking" happens...and if the DLL is used straight
away and throughout the program, then you're not really gaining any
advantage...many programs won't actually gain anything of significance,
often, it should be noted...

The possible advantage comes simply from the "only loaded and linked when
actually used" thing...you know, if a library is "optionally" or rarely
used then the fact it isn't loaded until it _IS_ referenced could prove
marginally more optimal...but, as noted, for most programs, there is
probably no actual useful difference...

You can imagine a program that, say, links to D3D9.DLL...but it only
actually uses that library _IF_ the user selects the "3D preview" option
from the menu...if the user never, in fact, selects that option while using
the program, then the library is never referenced and no time or RAM is
wasted on loading and linking it (this would be the "ideal" case of it
actually having a significant impact in actually being able to effectively
"drop" bothering with an entire DLL altogether when not actually needed
;)...even if the user _DOES_ select that option, then, anyway, the program
was at least "leaner" (not using so much RAM because the library was not
loaded) up until that point...so, the user does choose the "3D preview"
option, which causes D3D9.DLL to be loaded, after an hour of use of the
program...at least, for that first hour, it was running slightly "leaner"
in RAM because the library wasn't actually there...

And, you know, that kind of thing...but, yes, it is NOT really much of a
"big deal"...and, if your libraries are accessed almost immediately and
throughout the program, then, yeah, there probably is no advantage to a
program like that at all, in practice...and it's at the programmer's
discretion, anyway...you know, if you don't want "lazy binding" then you
just specify that and the linker sets things up for the "normal" non-lazy
way...

It's not really some "fantastic feature" but just an "alternative" way that
can be used...might sometimes "benefit" certain programs that use only a
library "optionally" or "sparsely"...

To be honest, it seems generally to not be of great use...if you really
wanted a more "selective" load-and-link thing, then you'd be better off
using _manual_ dynamic linking (the whole "LoadLibrary", "GetProcAddress",
"FreeLibrary" thing :)...because this would effectively get you the same
thing BUT it's better in that, unlike "lazy binding", it can actually know
to _UNLOAD_ the library when you call "FreeLibrary"...so, you know, this is
better still because then the library really is _ONLY_ in RAM for the
period of its use...with "lazy binding", it doesn't get loaded-and-linked
until that first access but I don't think any implementation of this out
there attempts to "unload" transparently (and it would NOT be a good idea
to try...you know, if it were to "guess" wrongly when a library is no
longer needed, then it would "unload"...then, oops, it is needed, so it's
re-loaded...then it "guesses" it doesn't need it anymore, so, oops, time to
"re-load" once more...hence, without a reliable way to be able to tell when
a library really isn't needed anymore, it can't "transparently" attempt to
do the reverse operation of "lazy unloading" on "last access" because
there's no way an OS can reliably "guess" when an access is a "last access"
or not...therefore, "lazy binding" only works for the "first
access"...after that, it's effectively no different to "non-lazy" with
regard to that library)...

Note that the only real reason we're talking about "lazy binding" in this
thread is NOT that we really need it for LuxAsm - in fact, LuxAsm will NOT
be using it, as we actually need it to "manually" deal with its own
linking, for the purposes of having "plug-ins" to the IDE - but it's just
because the "default" method LD uses is to "lazy bind", unless explicitly
stated otherwise...

So, indeed, we don't actually "want" this "lazy binding" at the moment in
any way for anything...but, as we've not been telling LD NOT to use "lazy
binding", then, by default, that's what LD will be doing...

So, you know, it's just a case of "remembering" that this is the case while
we are "testing"...because what kind of "linking" is used effects _WHEN_
you'll see any "file not found" messages:

Static linking: "File not found" appears at _LINK-TIME_...
Non-lazy dynamic linking: "File not found" appears at _LOAD-TIME_...
Lazy dynamic linking: "File not found" appears at _INVOKE-TIME_...

[ And, for completeness, if "manually" dynamic loading with "LoadLibrary" /
"FreeLibrary" (which is "dlopen" / "dlclose" in Linux terms ;)...then "file
not found" is an error return value from calling the "load" function, which
the program itself can explicitly deal with...this is why this method is
almost certainly needed for "plug-in" functionality and the "best"
way...its problem, of course, being that the program has to have code to
explicitly load and link itself included... ]

----

On a different subject, not related to this thread but something for NoDot
to consider about his "DotOS" stuff in another thread, is that, implicitly,
the stuff we were talking about before with the drivers was "manual dynamic
linking" because you explicitly _ask_ for "drivers" in the examples I
gave...and - at least in my design - this is an inherent thing to
delibrately _only_ support this kind of linking (explicitly, you call
"LoadLibrary" or whatever and are returned a "jump table" to its "exports"
and call "FreeLibrary" after using it :)...at first, this may seem a bit
"contravertial" to only support this type of linking...

BUT, remember, my design is based on "the toolmaker's view"...the OS
provides only "base functionality" and everything "higher-level" is
provided by "code libraries" (this is the underlying philosophy of
"toolmaker's view" to keep to that idea as close as possible, producing, in
fact, a very minimal OS indeed :)...hence, the other types of linking _are_
possible but the "responsibility" for them has been shifted _outside_ the
OS loader itself...in practice, the idea would be that linkers can attach
"startup code" (you know, just like C compilers do :) which "loads" the
libraries, calls "main" and then unloads the libraries afterward before
exiting the program...

In short, programs are actually expected (because of "toolmaker's view"
philosophy) to "do their own loading and linking"...the OS _ONLY_ provides
"base functionality" (as this is, in a nutshell, what "toolmaker's view" is
all about: The OS provides _ONLY_ the absolute very "basics"...everything
else is "deferred" to "code libraries" or the program itself to deal with
:)...this, though, does not bother those using C or Pascal compilers
because those will just include a "loader" routine in their "startup code"
that's linked to the program automatically by the HLL compiler / linker
(indeed, the linker will provide "generic code" which reads a "table" to do
the "loading / unloading" in the startup code, and the compiler will
automatically generate the required table as program data...a "standard"
for this table - a kind of ABI for it - could also be specified, just to
make it easier to have tools to read the table information and so forth
:)...

As noted, though, most OSes out there are anything but "toolmaker's view"
(Windows _certainly_ NOT so)...and, in a sense, the reason for me coming up
with this "OS design" is linked very closely to the fact that most OSes are
_crap_ exactly because they don't adhere to such a simple but powerful
"philosophy" like this (some OSes are better than others...there are things
in the Linux design that are very close to "toolmaker's view" but it's not
any kind of "pure" example of it at all...not that any of these OSes are
trying to be so, mind you :)...hence, my OS design is completely "inspired"
from what's so crap about other OSes, to try to come up with something that
wouldn't be, as far as I'm able...and, if you like, all the problems and
crapness of most OSes relate to the underlying "bad philosophy" of building
them too big, too complex, controlling everything, "usurping"
responsibilities that would be better handled by compilers / linkers /
programmers and such...

That, of course, is a "personal opinion" but I don't think it's objectively
far wrong...but convincing some people that "less is more" with OS design?
Well, I can't convince Randy that this is true for applications and he
_supports_ and _advocates_ assembly language because knowledge of it
promotes better and more efficient programs...I suppose the problem is that
this is easy to see in that case because it's a "technical" thing...while
"minimalism" as an "attitude" is a more "philosophical" thing...more of a
"religious" matter...

Beth :)


.

Relevant Pages

  • Re: C++ dlls not loading with WM 6, worked with WM5.0
    ... I need to build C and C++ libs and dlls that are linked in to by other ... The C++ libraries link in the C libraries. ... I have those built for Windows Mobile 5.0 and working fine. ... LoadLibrary(), and while it is able to load the C dll, it fails to load ...
    (microsoft.public.windowsce.app.development)
  • Re: static library and dynamic library
    ... Also the code of the library will load into the program. ... yeah, at link time, all the code and data from the library goes into the binary, which is part of why using lots of static libs tends to produce large binaries... ... so, the loader will also grab the libraries, and resolve the addresses to the addresses in the library. ... in windows, however, DLLs are typically fixed-address to allow sharing. ...
    (comp.lang.c)
  • Re: Mex vs. Generic DLLs
    ... and only load the appropriate libraries ... I'd have to load the ... to dynamic libraries. ... The overhead of actually finding, opening, parsing, etc. multiple DLLs ...
    (comp.soft-sys.matlab)
  • Re: EncryptMessage Memory leak
    ... You are correct about the dlls, ... I should say in any case that these two libraries are the main difference ... Yet memory continues to be leaked in the EncryptMessage call (I've proved ... these two dlls then I could see how you'd leak memory. ...
    (microsoft.public.platformsdk.security)
  • Re: How does Require deal with multiple files of the same name in different directories
    ... IMHO, once it is decided to load the file, this is required to happen, so it is not ... A program written for older W32F versions is available somewhere. ... As the old libraries won't work with the new W32F anyway, ... downloaded apps, which means checking any downloadable apps I (and ...
    (comp.lang.forth)