Re: High memory

From: Beth (BethStone21_at_hotmail.NOSPICEDHAM.com)
Date: 01/03/05

  • Next message: Ro : "Re: a 'turbo' assembly language" 
    Date: Mon, 03 Jan 2005 07:49:18 GMT

    rjb wrote:
    > I'm just learning protected-mode programming, and I'm trying to figure
    some
    > things out. I believe DOS programs, which are what I want to write,
    always
    > start up in conventional memory in rm, and would then switch to pm, yes?
    But
    > I know you can run code beyond 1 Mb. The question is, how do you get it
    > there? It can't load up there at program start, because you're still in
    rm.
    > It can only be that your program loads it in from disk after switching to
    > pm. Is this correct?

    Herman covered this...effectively, yes...you _could_ load it into real mode
    memory (in "chunks") and then copy it up above 1MB...but if you want to put
    the whole thing there in one go, then, yes, switch to PM and "figure it
    out", as Herman nicely put it...

    One thing is that, usually, in DOS programming, you'd be using some
    "extender"...you can write your own PM switcher, of course, but the "DOS
    extenders" do that job for you and with the "DPMI" standard thingy, it'll
    work okay with other "extended" programs (note that, for example, SMARTDRV,
    I think it is, actually switches to PM and then DOS is actually strictly
    operating in V86 mode with this loaded...a manual PM switcher would, in
    fact, soon discover that PM _is_ already in operation...DOS programs
    running via Windows also have this potential issue because Windows runs in
    PM and the "DOS box" is only a V86 thing...only one program can actually do
    the PM switching...the DPMI stuff and DOS extenders tend to automatically
    account for this kind of thing...by going through the DPMI API stuff, you
    see, it can either literally switch to PM or if something else is in PM
    then they can "co-operate" via the "standard", if you know what I mean
    :)...

    And, if using some "extender", then these typically would probably offer a
    "loader" or "extended disk functions" or something like that, through which
    you can get it to directly load stuff above 1MB and that kind of thing...

    It depends, really; If you want the "education", then you might want to do
    without delibrately to gain an appreciation of what's going on...but if you
    just want your programs to work with more than that 640KB - which is, of
    course, enough for anyone - and don't really want the bother and hassle,
    find some "extender" which does all the "nitty-gritty" for you...

    > Also, is there any reason to have code (as opposed to data) in big
    memory?
    > Does it run faster there, or is there some other advantage?

    No, it doesn't run any faster...indeed, the point to be realised is that
    outside the CPU, memory is seen in a completely different light...this
    stuff is only how the CPU sees it in "real address mode" or "protected
    virtual address mode" (using the _full_ titles of these modes here - "real
    mode" and "protected mode" are actually just "shorthands" - just to
    emphasise that the names themselves actually do relate to this by having
    "address mode" (and even "virtual" in protected mode's case, emphasising
    that all addresses are actually "virtual" in protected mode) in the titles
    :)...

    Hence, from the "memory controller" perspective, there is absolutely no
    difference between memory below 1MB and above...indeed, if you actually
    look at a stick of memory, then you can actually work that out...for
    example, I've got a stick of 512MB here to look at and it has basically 8
    identical chips along one side...meaning each chip must be 64MB each (8 *
    64MB = 512MB :)...there is no physically discernable "smaller chip" of only
    1MB with "real mode memory" labelled on it or anything...the actual memory
    itself is absolutely no different...the "mode" thing only relates to the
    addresses the CPU "sees" and can form to pass onto the address bus...

    Indeed, another point here is "memory-mapped devices" too...the memory at
    0A000:0000h, with a PC configuration (because once outside the CPU itself,
    things turn to the _PC_ specifications, not the x86 ones...you _could_ plug
    up an x86 chip (or two), of course, to something that isn't actually a "PC"
    :)...again, the "modes" only relate to how the CPU "sees" things and what
    addresses it can pump out onto the address lines...once those addresses are
    outside the CPU, they can actually mean anything (just bits on a bus, after
    all :)...on a typical PC configuration, there's - and I can never remember
    which is which - a "North Bridge" and "South Bridge" that these things feed
    into and its job is to direct these to the right places...to the memory
    controller or, for "memory mapped" devices, it can redirect the data via
    the system bus to actual hardware devices...hence, when you write stuff to
    0A000:0000h, the "bridge" (whichever one it is..."north" or "south"...both
    do a similar-ish job...I can just never remember which one it is sent to
    first :) redirects the data across the system bus to the video card, which
    directly pops it into video memory...this is why reads and writes to this
    part of memory (or any other similarly "memory mapped" hardware area :) is
    slower because it is, in fact, going over the system bus (and, simply, an
    ISA or PCI bus isn't as fast as the special high-speed bus between CPU and
    memory...this high-speed bus is short so it's affordable to make it of the
    more expensive and faster stuff...slightly more complicated - and I would
    defer to a hardware "expert" because, really, I'm the first to admit I'm
    crap with the more "hardware" side of things...I'm "software" and only pay
    attention to these things as far as it benefits software...not really a
    "gadgets" kind of person, despite the irony of that being a computer
    programmer...knowing a little about it helps write better software but I
    really don't give a crap whether it's 5 volts or 45 volts or whatever
    involved - when AGP comes into things because this "extends" that bus to
    include the video card to speed things up...PCI express is faster still but
    I have absolutely no idea, to be honest, about this newer stuff...is it on
    a "privileged" bus like AGP or not? Not the slightest clue, I'm afraid
    :)...

    Anyway, the point being, as I say, that this only relates to how the CPU
    "sees" things inside its own little world...outside the CPU, it's all just
    bits on address lines and the memory chips make no difference between
    anything...so, in those terms, there's absolutely no difference between
    "real mode" memory and anything else, as far as the rest of the system is
    concern...

    Indeed, the CPU itself isn't really making a difference either, in a manner
    of speaking...the "real mode" on these CPUs is just really a "compatibility
    mode" where it delibrately "cripples" itself to behave like an original
    8086...one little "trick" reveals this (you otherwise wouldn't notice it
    because the "emulation", so to speak, is very good that it completely
    appears to be operating in a different "mode" :) with the "unreal mode" (or
    "big real" or one of a number of other names for this "non-standard" mode
    :)...basically, the "trick" involves switching to PM, setting the segment
    registers to "flat 4GB" address space and then switching back to real
    mode...being careful not to ever reload any of the segment registers
    (you've no real choice with CS, of course, because the CPU itself accesses
    this all the time fetching instructions...so code has to stay below 1MB
    with this "trick"), this "trick" exploits the fact that the CPU doesn't
    check the segment registers except when they are being loaded (and that the
    "limit" of each segment is stored in a "hidden" part of each segment
    register)...and, thus, if you are careful to leave them (DS, ES, etc. :)
    with their PM "flat 4GB" settings, the CPU checks the "limit" against the
    value stored in the "hidden" part of these registers...this you've
    carefully maintained is a 4GB limit, not a 1MB limit (we need to switch to
    PM to set this up, though, because you can only get access to these
    "hidden" parts via that mode :) and the CPU happily accesses up to
    4GB...bringing about this "unreal mode" because it otherwise operates like
    "real mode" except the "limit" has disappeared on your data segements...so,
    just use 32-bit memory addressing and you can access all that memory
    directly...

    This little "exploit" reveals the true nature of "real mode"...the CPU, in
    a manner of speaking, is always in "protected mode"...and "real mode" is
    just an "emulation" setting it has to delibrately "cripple" its operations
    to "pretend" to be an original 8086...the "illusion" is effectively
    "watertight" - except for a few "loopholes" and "exploits" exposed by
    things like "unreal mode" - that you get the impression that there's
    something actually "physical" about this 1MB barrier...since the 80286, not
    actually true...the CPU is 32-bit, the memory is just one big "array"...the
    "limitations" are all "false"...an "emulation" to "cripple" itself
    delibrately to give the "appearance" of being an original 8086...

    Hence, not "faster" or anything...there is, in fact, no difference
    whatsoever..."real mode" is only an "illusion", really...an
    "emulation"...solely for "backwards compatibility" purposes...this makes
    sense, though...you wouldn't actually create two nearly identical but
    separate chips and stick them on the same piece of silicon...not a clever
    way to do it...instead, they just create a 100% PM chip and then add in
    these "modes" like "real" and "v86" for the "compatibility", where it
    "pretends" not to be operating that way...if you know what I mean,
    anyway...indeed, one can even see that the "V86 mode" is merely where it
    kind of "half switches on" the "real mode" emulation but not completely,
    that some PM stuff still applies in places...

    Anyway, yes...this all means that there is no "advantage" to that memory
    above 1MB other than the slightly obvious one that there's lots of it up
    there (especially on modern configurations, there's an awful lot more of it
    "up there" than there is down here in real mode...the vast majority is "up
    there", in fact :)...and other things related to that, such as the point
    Herman made that older hardware and such might be "stuck" below 1MB in what
    it can use, so, by extension, the memory below 1MB kind of becomes
    "precious" from that perspective (not "special", though...just "not an
    option" for hardware that's old and has no idea about post-real-mode stuff
    above 1MB :)...

    So, yeah, if your program is not going to "break the bank" when it comes to
    the size of the code itself, there seems no particular "advantage" to
    putting it above 1MB...indeed, in practical terms, the easiest way of doing
    things is probably to have all the code operate below 1MB and if you need
    lots of data, then it's just the data which is >1MB...you'll find that
    "unreal mode" works on this premise (because CS gets "reset" and goes back
    to the "real mode" limitations...so you can do "big data" but not "big
    code" with this "trick")...the "expanded memory" and "extended memory"
    specifications (old standards for when memory first starting breaking the
    1MB "limit"...not particularly used anymore because they had absolutely NO
    "rules" regarding how that > 1MB memory was organised) just use "big data"
    but not "big code"...some "DOS extenders" don't really usefully "extend"
    the BIOS stuff into these areas...hence, in practical terms, you'll
    probably find that "big data" is easier and makes the most sense, anyway
    (or, another way to put it, once you want "big code", hassles and problems
    start to appear with getting DOS and the BIOS and everything to function
    correctly :)...and simply putting the code in "real mode" memory, as
    normal, only using > 1MB for your "data" is the easiest route (should, of
    course, your actual code _fit_ into this space...but if, as you say, you
    hardly see yourself breaking one 64KB segment, let alone two, let alone the
    640KB which is enough for anyone - hmmm, appears to be "enough" for you,
    anyway - then this isn't a problem for you to do :)...

    The "barrier" between "real mode" and "protected mode" is more logical than
    physical...DOS and the BIOS only works with "real mode" (excluding a few
    rare things like the VESA "protected mode" stuff :)...that's the true
    problem, really...switch to PM and these things are totally screwed up and
    can't be used anymore BUT you get access to all that memory...stay in real
    mode, DOS and the BIOS are happy BUT you get this annoying "limit" on the
    memory...and "never the twain shall meet", so to speak...

    This little "incompatibility" - though Intel really did need to just dump
    "real mode addressing" ungraciously for the horrible "hack" it was - has
    haunted the x86 and PC...in a sense, it's because of this - it's a large
    instrumental part, anyway - of how Microsoft keep that "monopoly" of
    theirs...basically, if the BIOS worked just as happily in protected mode
    (and was revised to be "multi-tasking aware") then it could serve as
    "universal device drivers" for _ALL_ OSes equally...then it would come down
    to a "software versus software" conflict between Microsoft and others...and
    Microsoft simply aren't good enough coders to win that battle (or,
    alternatively, Microsoft "improve their act" and get that good...in which
    case, we're _still_ happy because good old "commercial competition" has now
    forced them to code properly and "compete" with delivering the best
    solutions :)...

    This never happened, of course...when protected mode appeared, it rendered
    the BIOS unusable in PM...but, instead, OSes used "device drivers" so that
    dealt with the problem instead...the slight "flaw in the plan" was, of
    course, that Microsoft got there first...and they made their "device
    drivers" only applicable to Windows (its API and its "memory model" and so
    forth :)...hardware manufacturers making hardware think "right, we need to
    make drivers for our hardware"...they look around...there's Windows, used
    by 90+% of all users...well, simply, if, as hardware manufacturers, we want
    to shift any of these "units" of hardware then it's a "given", we've got to
    make sure we have Windows device drivers...hence, Microsoft gain their
    "100% hardware coverage" (or "as good as") without lifting a finger...the
    hardware manufacturers themselves are going to strive to make sure that
    they have "Windows support" because they know that if they don't, they
    ain't selling any reasonable amount of "units" of their hardware at all
    (and, of course, they ain't charities...they want to sell as many as
    possible and make themselves a nice little profit out of it all :)...

    What about other OSes? Well, it's a possibility...but, on the other hand,
    it does cost money and effort to develop these device drivers...is it
    really worth the effort for a small amount of Linux users? I mean, there's
    not that many Linux users and how small a percentage of them are actually
    using our hardware? "1% of 1%", so to speak? It's not worth putting
    together a "team" to develop them...developing "drivers" for Linux _AND_
    BeOS _AND_ Menuet _AND_ "John's Hobby OS" _AND_ all the rest of them? Oh,
    you've got to be kidding...we're not a bloody charity, you know! ;)...

    Hence, what emerges quite naturally from all of this? Microsoft sit around
    in the Bahamas (avoiding taxes there, you see...not even "income tax"
    exists on those islands..._THE_ proverbial tax haven ;), sipping cocktails
    by the pool while the hardware manufacturers themselves slave away creating
    all the "hardware coverage" for them...check out the copyright on most of
    the device drivers you have...Microsoft do a few "core" ones, just to make
    sure it'll work at least in "safe mode" to allow other drivers to be
    installed...they also "charitably" volunteer to host others drivers on
    their "Windows update" (ooh, big deal...they just stick 'em onto their
    servers and check a few "version numbers" here and there to distribute the
    right files :)...but they don't actually write most of them...they do
    NOTHING and, quite naturally, their "monopoly" is supported by hardware
    manufacturers just trying to make a living...worse, it's
    "self-perpetuating" that even if the hardware manufacturers can't stand
    programming for Windows, they hang themselves by their own rope...in making
    drivers for Windows, they make Windows even more "THE" platform that must
    be supported...Windows further secures itself as the only OS to offer 100%
    hardware coverage and more or less 100% problem-free
    installation...understandably, most users prefer the OS that can give them
    that guarantee (though, again, the irony is that it's really the _hardware_
    people who are doing this)...

    What must something like Linux do instead? No hardware manufacturer support
    at all (the exceptions of, for example, nVidia providing Linux drivers is
    actually to do with the fact that nVidia refuse to give out
    "specifications" of how their hardware works...hence, if nVidia don't make
    the drivers, no-one does...hence, because of this "no specifications"
    policy, nVidia kind of "oblige" themselves to create drivers for other OSes
    like Linux...not "compulsory", mind you...there's the nasty prospect that
    they may one day simply think "screw that" and, indeed, Linux is screwed
    for nVidia support...quite nasty when a lot of brand new machines have
    nVidia or ATI and it's now the trend that other manufacturers just make
    cards with either of these two "chipsets" inside them)...the only option
    typically being "grab the specifications off the 'Net and develop them
    manually"...

    Indeed, the fact that Linux actually does really rather well from this
    seriously "disadvantaged" position is a testiment to how well the "thousand
    eyes" open-source principle really can work...it sometimes comes under
    criticism for not supporting this or supporting that but considering the
    Microsoft "unfair advantage", it does amazingly well (and it gets
    better...Linux support for hardware has drastically improved as time has
    gone on...indeed, because it's all "thousand eyes", it naturally goes from
    "strength to strength" as more developers join on board...because it's
    those developers who end up creating this support)...and, on the contrary,
    I find that this is more a fantastic "proof of concept" that the open
    source "thousand eyes" really _CAN_ work because considering the
    significance of Microsoft's "unfair advantage" in all of this, Linux
    hardware support seriously bucks expectations and it's all down 100% to
    those "thousand eyes"...remember, Microsoft do NOTHING and get handed "100%
    hardware coverage" on a silver platter...Linux has to slog its way through
    "specifications" (_IF_ hardware manufacturers have bothered to provide
    any..._IF_ those specifications are in any way comprehensible to get good
    drivers working from them) and gets no "outside help" whatsoever (excluding
    the nVidia-like exceptions...but that's down to them refusing to give
    specifications to their hardware)...these are polar opposites in terms of
    getting the job done, so just how well Linux actually manages to not really
    be too far behind Windows is, I'd say, a remarkable bit of "proof" that
    when the open source thing works (doesn't always work but when it does), it
    can, indeed, work a "miracle" or two...

    Yeah, all down to crap "real mode addressing" and the wrong person ("Sir"
    Gates ;) being in the right place at the right time...

    "Sometimes when I consider what tremendous
    Consequences come from little things...
    I am tempted to think...
    There are no 'little things'"
    [ Bruce Barton ]

    Beth ;)

  • Next message: Ro : "Re: a 'turbo' assembly language"