Re: newbe about API

From: Beth (BethStone21_at_hotmail.NOSPICEDHAM.com)
Date: 04/02/04 

Date: Thu, 1 Apr 2004 23:03:19 +0100

Herbert Kleebauer wrote:
> Beth wrote:
> > What's also interesting to note here is that Herbert's syntax with
the
> > sizes put into the operands means he doesn't need to "BITS 16" for
the
> > DOS part and "BITS 32" for the PE part...
>
> I also need it (seg16, seg32) to decide if a move.w or a move.l
> needs a data size prefix. seg16 is the default, so it can be
> omitted before the DOS stub.

Ah, I didn't spot that...

But, on further reflection, you'd kind of _have to_ have such a thing,
yeah? D'oh! My "thinking cap" has fallen off these past few days, for
sure...because the Intel encodings and prefixes for 16-bit and 32-bit
aren't "absolute" but are "relative" to the current "bitage"...that
is, they work like a "toggle switch": if in 16-bits, the prefixes turn
things 32-bit...if in 32-bits, then prefixes turn things 16-bit...so,
for a particular sequence of instructions, there's no real way to
know - from only looking at those instruction encodings - which way
around it is...

Hence, even with your great "size suffix" syntax, Intel's encodings
mean that it's kind mandatory to have something like this
somewhere...totally dumb that I didn't see this in your code here
because - duh - I've actually _made this point_ myself numerous times
before in other threads...

Oh, my brain is just malfunctioning at the moment...sorry about
that...

> ] ; ..............................................
> ] ; : Start of Code :
> ] ; ..............................................
> ] label_block
> ] seg32
>
> > The way I see it is that there should just be a distinction
between
> > the kind of "file pointer" we have and the "address in memory" of
the
> > code itself...
>
> I use @ for the current location pointer (the virtual address at
> which the next instruction is executed) and @@ for the current
> file pointer (the address in the exe file where the opcode for
> the next instruction is written).

Cool! So, your assembler _does_ do it the way I'd proposed made some
kind of sense...always nice to know that the "theory" works well in
practice...and, there you have it, Herbert...I independently thought
up a similar idea...let's indulge ourselves and call this "great minds
think alike" ;)...

Anyway, yeah...seems you've seen it too...there's actually two
_separate_ things involved here - where to put the code in the file
and where that code actually "thinks" it is when loaded into memory -
which often run _in parallel_ in most cases (as you say, you increment
both at the same time as you assemble instructions ;)...BUT they
aren't necessarily "one and the same" thing...

The problem I see in the more traditional "ORG" directives found on
some assemblers is that they can be guilty of _confusing_ the
two...NASM's basic premise of "only one ORG" is correct in that it
stops the two separate things being confused...as an example, here's a
typical example of the "ORG" creating confusion under something
MASM-like:

---------------- 8< -------------

ORG 7C00h

; Bootsector code goes here

ORG 7DFEh

dw AA55h

---------------- >8 -------------

Now, this is a very confused situation...the second "ORG" is being
used to jump to the two bytes just before the end of the 512 byte
bootsector (just in case someone reading doesn't know - because I just
know you already do, Herbert - a typical bootsector like this is
exactly 512 bytes in size (as the name suggests, it's the size of one
"sector", which actually can be other things besides 512 bytes...but,
in practice, for "compatibility", practically almost everything uses
512 byte sectors - once set, the value kind of "stuck" with people, I
guess - and this is the "compatible" size you'd have to make it to
ensure that all devices are covered...the last bytes of this 512 byte
"sector" are set to "AA55h" - which is an alternating "on off" bit
pattern in one byte and its reverse in the other byte - to let a BIOS
(though not all actually check this as they should) tell if it's
actually got a valid "bootsector" or just some random "garbage" that
happens to be in that sector on the disk)...

Our problem here is that we'd expect the second "ORG" to work like a
"padding" directive...it jumps the file pointer to the end of the
sector and any bytes not covered between the start and end are just
"padded" with zeroes or something...

BUT, with the first "ORG", we're actually setting up the "memory
address" (bootsectors are loaded into 0000:7C00h in memory :) so that
our address offsets are calculated correctly...but, _unlike the first
"ORG"_, we _DON'T_ actually want move the "file pointer" at all...it
should output as "byte #0" of the _output file_...all the "ORG" is
meant to be saying is "byte #0 in the file is 0000:7C00h _in memory_
once loaded"...we _DON'T_ want the assembler to be inserting 7C00h
(31,744 bytes, according to "Calculator" ;) of "padding" before the
code at the start of the file...

And the NASM guys rightly saw this as being quite ambiguous and
confusing...the very same directive - "ORG" - is being expected in the
above code to behave in different ways each time...the first one we
only want to "set the memory address" but, with the second one, we
only want to use it for "padding" (and, in fact, the "memory address"
isn't relevent because "dw AA55h" doesn't use it at all...in another
situation, we _may or may not_ want to actually deal with memory
addresses...and here's the bizarre thing: despite being called "ORG"
for "origin", when you change "ORG" like this later on, it stops doing
that...a "dw $" after "ORG 7DFEh" above would put "7DFEh" into the
word variable, NOT zero :)...

So, the NASM approach is "only one ORG" and then, if you want
"padding", there's other directives like "TIMES" to do that...this
also clears up the "ambiguity" of what _exactly_ should the assembler
_fill_ that "padding" with? In NASM, you've got to put the padding in
manually in this case, so you specify it yourself..."ORG" is for the
"memory address" only and NASM says "you can only declare this
once"...

BUT, there's where the problem comes into things...seems a bit like
NASM is _still_ getting confused between two completely different
things here, just because they usually "run in parallel"...I actually
_did_, ages back, try to _explain_ that there's a subtle difference
but failed miserably to describe it properly...I picked an awful
metaphor of "time" and probably confused the matter even further
still...

But, yeah, I looked at your code, Herbert, and you employed something
like the "$ = 100h" suggestion I was making at that time in your
assembler...better yet, you _have_ gone and separated the two quite
different (if usually "running in parallel" :) things...I only noticed
at first that you've just got an "application" here which highlights
what I wanted to talk about...but now I see, from what you've said,
that you've gone further than this and your assembler basically _does_
what I was going to suggest should be done, anyway...you've now also
given me a syntax as well as example code to try again at explaining
things without, Hopefully, confusing anyone this time like I did last
time, going on about comparing "file position" to "time" and other
strange nonsense...which actually made it _more_ confusing rather than
less so...

> If an instruction is assembled,
> both @ and @@ are incremented by the instruction size.

And this is exactly the thing I was wanting to address...but, better
yet, you've got a working assembler that _does_ separate the two
things and shows how it could actually work...

Indeed, the earlier "bootsector" problem becomes:

---------------- 8< -------------

@ = 7C00h
@@ = 0

; Bootsector code goes here

@@ = 510

dw AA55h

---------------- >8 -------------

...or something similar (well, the "dw AA55h" would be "dc.w $AA55" in
your syntax? ;)...

Although, this is merely a "translation" of how the MASM example
worked...as you say below - and NASM would resist because it
introduces "ambiguity" about what the assembler would "fill in" the
gaps in the file with (use "TIMES" to make your own padding instead,
right? ;) - you don't change @@ arbitrarily how you please...rather
you use the "alignment" and "padding" directives to do that job
instead...I was just "translating the syntax" here from the MASM-like
stuff above, so that people can see how what we're really talking
about is two _independent_ things (which the usual MASM-like "ORG"
simply confuses together, treating the first "ORG" differently to how
it treats the second "ORG"...which is all a bit confusing and
ambiguous)..."translating the idea" to your "@" and "@@" system would
result in different code, I do realise...

> You can
> modify @ at any time by an assignment like:
>
> @=$40000 or
> blk.b 100 (which is the same as @=@+100)
>
> but @@ only is modified by writing bytes to the output file
> (this means @@ can only be increased but never decreased; you
> can't go back and overwrite an already assembled code).

So, yeah, you, in fact, also have the NASM-like approach of insisting
that "padding" and "alignment" is done using the padding and alignment
directives (in NASM, this would be "TIMES"...or, in MASM style, using
"DUP" or "ALIGN" or something...don't know what it is in your syntax
exactly - still trying to "decode" how it works - but I'm sure you've
got your equivalents...they _must_be in that example code you posted
because you have to "align" parts of the PE file to certain
places...let me have another look at the code to find
it...*pause*...ah, "even" and "evencom" or something :)...

Yup, this is in-keeping with the NASM perspective on the matter...the
thing with NASM's approach is the "only one ORG"...this might be
taking a step too far...this is what your code highlighted when I
considered using NASM to do what you did there with your own assembler
(as NASM has the necessary "flat binary" and "BITS 16" / "BITS 32"
mode...and basically, should have everything else you could possibly
need to do this job :)..."only one ORG" is like saying "only one
assignment to @"...and, if you imagine doing what you've done in this
example with such a restriction, it's a troublesome limitation to have
this here...

At the very least, to make both the MZ and PE executable code read
easily, you'd want to put - as you did - a "@ = 0" and "@ = imagebase"
before the code respectively...then all the labels and address
calculations are correctly worked out in respect of where in memory
the MZ and PE file will be loaded...but being able to "@ = SomeNumber"
in different places could be used to even more useful and convenient
effect...

And, to be honest, I even mostly like the syntax you've adopted...it's
just an "assembler variable", so to speak...you just use it in
expressions or re-assign its value as you feel like...the only
possible change is to make it "$" rather than "@" to give it an
"Intel" flavour instead...otherwise, yeah, "$ = $ + 100h" and such
also makes a good syntax for it too...depending on how you've coded
the assembler, you can just make use of the "constant expression
evaluator" or whatever to handle this...just a case of also including
the ability to "assign" to "$" arbitrarily, not only take its
value...possible conflict with NASM in that "$$" (your "@@") is
already taken...would require thinking about what the syntax should
exactly be for an "Intel style" equivalent (like you've done - but in
Intel terms - it would seem "natural" that "$" is virtual address,
"$$" is the file pointer and perhaps something like "$SectionName" for
the address of the start of a section...doing it this way would kind
of also allow you to calculate offsets from the start of _other_
sections, not just the current one...say, "$ = $$ - $text" in the
middle of a ".data" section or something equally strange ;)...

Well, I don't know if I'm actually making any "proposals" to the NASM
guys, anyway...I just noticed this and thought I'd point it out and
discuss it...NASM can do the stuff needed already (if all else fails,
then you can use expressions everywhere, like the old "+ 7C00h"
adjustments onto every address in a bootsector to manually change your
"origin" with a constant addition or subtraction ;)...it's just
whether it would be more convenient and useful to simply do as Herbert
has done here and allow arbitrary changes to "@" or "$" - which is the
_virtual address_, unrelated to the "file pointer" - but still keep
it, as Herbert himself has exactly done, that you can't "step
backwards" in the file...that "@@" (or "$$"...but, as that's already
used in NASM, there is a "syntax issue" there ;) is the "file pointer"
and that variable is "read-only", unlike "@"...

But, yeah, Herbert, you've gone and done it again...I like your style
of thinking on these kinds of things...not too dissimilar from my
own...heck, I don't even think your "Motorola syntax" stuff is a bad
thing, even...it's "potentially confusing" for those who have no idea
what it is you're doing there (very possible for people to look at it
and be "confused", thinking it isn't x86 code when it is ;)...so,
yeah, I'll probably myself stick with "Intel syntax"...BUT, I don't at
all disagree with you that this style of syntax is generally a whole
lot less messy and much more clear...

It _was_ an "injustice" to the world that, through _accident_ alone,
Microsoft and Intel and IBM went and set up something that's often a
complete nightmare in places as the unfortunate "norm" and "dominant"
stuff...I'd bet that - if spoken privately and "in confidence" - even
Microsoft and Intel would admit to how even they aren't really pleased
with the way things have turned out on these issues...it's just, like,
IBM went and made a "glorious mistake" in the PC, propelling the both
of them into the forefront...the PC was actually a _step backwards_
originally that IBM didn't take at all seriously...it's come a long
way since then, for sure...but some of that "accidental" origin still
haunts us...like the bloody "real mode addressing" that the machine
_still_ has to boot up into for "compatibility" (which means the BIOS
is "real mode" and that's "address incompatible" with protected mode,
which means you lose the BIOS in protected mode, which means everyone
goes with "device drivers" instead, which allows Microsoft to get in
there first and make it all "proprietary", which means they take over
everything, which means hardware people have no choice but to support
Windows, which means they have "full coverage" for doing absolutely
nothing - it's just handed to them on a silver platter by hardware
manufacturers - while everyone else desparately crawls against the
grain, which means Microsoft have an impenetrable "unfair advantage",
which means they own everything and all software is crap because of
it, as what's the point writing great applications when the OS it sits
upon is such a pile of complete dung? Yes, quite seriously, this one
"hack" _is_ mostly responsible for the "chain reaction" which leads to
there being no useful alternatives to Microsoft ;)...it was a terrible
hack! Like I say, if you could talk privately and in confidence that
it wouldn't be spread around the whole world as "gossip", then I bet
you could even get the Intel engineers who came up with it to flatly
admit: "Yeah, total nightmare!! But we honestly didn't believe that
this chip would go anywhere that special...certainly no idea at all
that it would still be around _30 years_ later and dominating a
massive, massive desktop market around the world!! Yes, _IF_ we'd only
known that's where things were going to be headed, then we'd _NEVER_
have put such an ugly, ugly 'hack' into the chips and would have
designed them a touch more 'properly'...I mean, when you expect only a
few years of 'shelf life', you just naturally don't put so much effort
in and don't mind 'cutting corners' here and there"...

This is also a lesson in "don't underestimate important it is to get
things right and not think 'oh, doesn't matter if we just hack this
here and hack that there'"...why I stress the importance of trying to
foster a "do things properly / don't rush for the sake of rushing"
attitude all the time...that's exactly what Intel _did do_ and we're
living in a Microsoft dominated nightmare because of it! As the
quotation goes: "When I consider the great consequences that can come
from little things...I am tempted to think that there are _NO_ 'little
things' at all"...this is totally a "little thing" that Intel did with
real-mode addressing...but it has been the source of a string of
_nightmares_ over and over for going on _30 years_ now...it's not just
Microsoft dominance either...they took so bloody long getting it from
16-bit code to 32-bit code, from DOS to Win32, because of not simply
Microsoft's incompetence but because Intel's "little thing" puts up a
tall "barrier" between "real mode" and "protected mode" that's quite
"non-trivial" to cross while still holding on to all your
hardware...without this, Microsoft would probably have been just one
decade too slow from their typical "who cares?" attitudes...rather
than two decades too late because though Intel put in the necessary
stuff for full 32-bit multitasking OSes as far back as the mid '80s
(about '85 / '86 time, if I recall correctly :)...this "little thing"
meant Microsoft only came up with their "first attempt" ten years
later...and now almost _20 years_ later, with XP, they can actually
begin to honestly be proud of their "32-bit" boasts (not much else but
they've finally began to get this one thing right after all this
time)...almost two decades too late that the "boasts" in their adverts
are more than a little dry and distasteful because the whole thing has
gone "stale" sitting there for nearly two decades...so, Intel's
"little thing" doesn't just mean Microsoft dominance...it's also a
factor in why Microsoft's stuff is generally so crap...they'd have
been late and crap, anyway - because they are Microsoft, after all -
even without Intel's complexities confusing and slowing them down even
further...

Hence, Herbert, I might not use your syntax but I utterly sympathise
with your reasons for adopting it...if nothing else, it's a reminder
that _thing went wrong_ and are slowly being restored in the PC
world...that, elsewhere, they've been using _clear syntax_,
_understandable architectures_ and so forth for a long, long
time...the PC itself was a kind of "hack" and its OS was "hacked" out
(well, "legally stolen" and then "hacked" to have "Microsoft" written
on it ;)...then a "hack" called Windows was put on top of these other
"hacks"...but, oh dear, the public actually were buying this crap!!
Ummm, time to perhaps try to "de-hackify" all this crap a little
bit...for example, why is Windows such a security nightmare? Well,
basically, it's because most of it was written without any
consideration whatsoever to the consequences of "internet
connection"...even for Windows 95, the internet "revolution" wasn't
really planned into things...they kind of thought "oh, let's throw in
some 'internet'" at the last minute before release...to their surprise
and wonder, people actually kind of did buy their crap that they
broadcast in the adverts...oh dear, time to "play catch up" once more
in trying to get the OS to actually meet the demands that will be
placed upon it...it wasn't designed this way at all! Quick, "patch"
that! Ummm, send out an "update" for this! Aarrgh!! ;)

But, again, look over at Linux and FreeBSD...they don't have this
problem in quite the same way...why is that? Ah, this is because they
are based on modern UNIXen...which are designed _ground-up_ as
"multi-user" and "internet-friendly" for the beginning...there is a
_reason_ why the slashes are _forward slashes_ in internet
addresses...they were all saying "oh, at last...finally caught up with
what we've been doing for years, eh?" when the "revolution" fell upon
everyone else using crap Windows "client software" to connect...the
stuff in *NIX is honest ***-ups and doesn't show up all that
often...the stuff in Windows is not only mistakes which you'd expect
from any software, it's that the entire thing was not really designed
the right way at all...

It's like that post someone made about finding tons of "JMP [ ESP ]"
instructions inside their code...the fact that pratically every single
"buffer" in the entire Windows code has been "patched" to avoid
"buffer overrun" virus problems...these things basically confirm what
is already suspect from the way Microsoft approached the stuff...they
_didn't_ design Windows to work this way at all in the beginning...and
now they are desparately "patching up" the software to turn it into
what it should have been from the off...

In short, the reason that practically every single "buffer" in all of
Windows has a "buffer overrun vurnerability" that a virus could
"exploit", which they send out a "patch" to fix all the time is
because Microsoft clearly had absolutely NO "policy" to tell their
programmers to account for "buffer overruns" in their code...and this,
I'm sure, is because Microsoft never drafted up such a "policy", as
they weren't even thinking "internet security" as they set their
programmers to work...though Windows 95 was launched into the
"internet revolution" - carried that "revolution" along, to be fair -
it's easy to forget, though, that the actual Windows 95 code was
_written before_ this happened...what we see released was actually
coded years beforehand...indeed, _did_ the guys working on the
interface ever get told that "internet connection" was going to be
strong and important? Probably not, they presumed - as had been the
way up until then - that most machines would be "stand alone" and
"off-line"...that was the "norm" for Windows and PC desktops up until
then, after all...also, we must factor into this that, despite
Microsoft "hype" trying to suggest otherwise, some of the code clearly
originates back from Win3.x days (I cite the "mouse pointer bug" as
visual proof that this is the case...you don't repeat such an
incompetent visual bug twice, if you were always re-coding from
scratch every time...you might, indeed, accidentally replace one type
of bug for another but make the same exact bugs with completely new
code? Unlikely...on the other hand, if you take the Win3.x _C source
code_ and then re-compile it to 32-bits, then an algorithmical
"cosmetic bug" like we see on the mouse pointer jittering around as it
changes shape would be _exactly_ retained by a simple "re-compile" of
the old - correction: _ancient_ - original mouse code that has had
this problem from at least Win3.0 (if not before...I simply never
encountered Windows before this point to know whether the code is
_even older still_, which is very possible ;) right up until XP...and
it wasn't really "fixed" for XP...they were forced to re-write the
code to add in the new "transparency" and "mouse shadows"
features...it was never actually "fixed"...it just eventually got
"replaced" when they added new features, which mandated that the code
be revised...although, as I write this, I've just noticed that XP has
done what it tends to do from time to time...I've lost the mouse
"shadows"...time to go to the settings and switch it off then switch
it on to get it to appear again...XP's better than previous Microsoft
garbage but it's _still_ far, far from acceptable...how on Earth does
it just "forget" about the mouse shadows every once in a while, for
seemingly no particularly good reason? Oh, I'd give up on it all, if I
only could ;)...

> @ needs not only be assigned at the beginning of the DOS
> stub:
>
> ] @=$0
> ] dosmain:move.w s6,-(sp)
> ]
>
> and the start of the Win32 program:
>
> ] ; +--------------------------------------------+
> ] ; | Start of Windows Header |
> ] ; +--------------------------------------------+
> ]
> ] ImageBase== $00400000
> ] SectionAlignment== 4096
> ] FileAlignment== 512
> ]
> ] WinHeader=@@
> ] @=ImageBase
> ]
> ] ; see WINNT.H for
information
> ] dc.b 'PE',0,0 ; magic word
>
>
> but also at the end of each section:
>
> ] VSizeOf_text==@-Imagebase-VBaseOf_text
> ] @a=@
> ] evencom FileAlignment
> ] @=@a
> ]
> ] FSizeOf_text==@@-FBaseOf_text
> ] SizeOfCode==FSizeOf_text
> ]
> ] ; +--------------------------------------------+
> ] ; | End of .text Section |
> ] ; +--------------------------------------------+
>
>
> evencom writes NULL bytes to the output file till the file
> pointer @@ is a multiple of FileAlignment (mostly 512). This also
> increments @ which then must be reset to it's original
> value before the evencom instruction.

Ah, that's cool...just wondering, why "evencom"? That is, what does
the "com" mean and I notice you use things like "even 16" elsewhere,
which I presume is like "align 16" in MASM-like syntax (that is, "pad
until you reach a multiple of the number specified")? Why not just
"even FileAlignment" or something? Is there some subtle difference
between "even" and "evencom" that I'm not seeing here?

Hmmm, perhaps the first place, though, where I'd dispute your syntax
choices...it probably would make more sense - in "plain English"
terms - to call it "align xx" rather than "even xx"...also, because
it's called "even" does this actually mean that it doesn't allow odd
numbers or non-powers-of-two? Would "even 3" or "even 42875" be
considered "invalid" (yeah, there's no good reason I can think of for
allowing such things...other than to say that there's no good reason
NOT to allow such things and, who knows? Someone might find a use for
delibrately aligning their code and data unaligned - perhaps
_delibrately_ to get an "alignment exception" or to delibrately
straddle over two pages with a 16-bit access (going to be an
"unaligned access" doing that) in order to force both pages into
memory at the same time...or something like that? This is why I stress
looking for reasons _NOT_ to do something rather than looking for
reason _TO_ do something...even if you can't think of anything
yourself, then, in restricting something, you can be sure that along
will stroll someone who says "Wait a minute! Delibrately unaligned
alignment has a valid use here and here and here!"...so, I'd generally
caution to go for an "_allow_ it unless you can think of a good reason
why it really shouldn't be allowed" policy on these kinds of
things...the chief mistake that I'd challenge at Rene is that he's too
ready to "restrict by default" rather than _provide Liberty_ to the
programmer to choose "by default"...after all, life is full of
limitations enough without adding more, which aren't particularly
necessary and don't do anything but stop certain _perfectly valid_
things from being done)...

> For the next section
> @ then is incremented to a multiple of SectionAlignment
> (mostly 4096):
>
> ] ; +--------------------------------------------+
> ] ; | Start of .idat Section |
> ] ; +--------------------------------------------+
> ]
> ] FBaseOf_idat==@@
> ]
VBaseOf_idat==(@-ImageBase+SectionAlignment-1)/SectionAlignment*Sectio
nAlignment
> ] BaseOfData==VBaseOf_idat
> ] @=ImageBase+VBaseOf_idat
>
> The funny thing is, if you don't restore @ after an evencom
> and get a wrong size of image, the code will still run in Win9x
> but not in NT/W2k.

Yeah; Despite both having the Win32 API, their _internal structures_
are different (NT kernels are actual OSes, 9x kernels are hacked DOS
applications trying to "pretend" to be real OSes ;)...and these
differences actually show through to the surface with 9x stuff running
what NT kernels reject...to be fair, Microsoft - though goodness knows
why - seem to have intended that these differences should show through
because their documentation openly points out that 9x and NT based
kernels have different "tolerances" to unusual PE file structures...I
can never remember which way around it is - whether it's NT or 9x that
the more "tolerant" - but you can actually set the "file alignment"
and "section aligment" stuff smaller on one than the other, getting
yourself an even smaller executable file...just the problem is that
only one "stream" is able to actually run it and the other rejects
it...as I say, though, I totally forget which way around it is,
whether it's 9x or NT that can "tolerate" small alignments the
furthest...that's why any kind of "smallest PE" contest should really
stress whether that's for all versions of Windows and that you should
probably double-check that it works on both...

I suppose because of the different internal structures of the two
"streams", the actual "PE loader" code is simply completely different
in the two - even though processing the same format of file - and what
we're seeing is the little differences between the two implementations
there poking through...

Beth :)

Loading