Re: From the LuxAsm list.

From: Beth (BethStone21_at_hotmail.NOSPICEDHAM.com)
Date: 08/04/04 

Date: Wed, 04 Aug 2004 02:38:22 GMT

Randy wrote:
> Percival wrote:
> > Randall Hyde wrote:
> > >>Beth, said :
> >
> > I don't think that it would be possible to add an IDE to an assembler
> > like this...
> > The IDE would have to be built in or something...
>
> While technically this isn't true, for all intents and purposes it would
be.
> Every interactive system like this I've seen (APL, BASIC, Lisa, Incra,
etc.)
> have all incorporated the editor and assembler in the same program.

Well, meet LuxAsm...a system where it isn't so...

Another idea I've thrown into the mix is that the system is strictly
"client / server" with the IDE as the "GUI server"...everything else is
merely a "module" that can be plugged into the IDE (including "filters",
which are just "modules" that wrap around tools to make them fit into the
IDE scheme...could be used to potentially "integrate" some other tools -
like NASM or MAKE - into the LuxAsm IDE and to take advantage of what can
be taken advantage of, from the LuxAsm IDE, in doing so :)...

There is, of course, a _LOGICAL_ dependency between the editor and the
assembler...they will be composed in a "tightly integrated" manner...

But this is NOT a weakness...this is the _advantage_...it gives direct,
"specific" support...LuxAsm is a _specific assembler_ and this is one of
the places where this most obviously shines through...the tools _ARE_
"tightly integrated" to work together..._THAT_ is the advantage...

But, regardless, I know what you're talking about...which is why I put
forward the "modular" client / server architecture to base LuxAsm on top
of...so that the connections aren't direct...rather, the assembler "module"
can query the IDE for the presence of the linker "module" and then, if they
find each other, then they establish a "connection" to send "messages"
between the two, then they "tightly integrate"...we _WANT_ them to do
this...

Similarly, the "editor" module will have the facilities in place to, yes,
very specifically deal with all the added features the "assembler" module
can provide it...the "editor" module will send a "query" to the IDE asking
"Is the LuxAsm assembler loaded?"...and if the IDE replies "yes"...then it
asks "Cool...could you establish a 'channel' to the assembler, please? I'd
like to talk to it directly"...IDE: "Sure thing...here's a 'handle' to
it"...editor: "Hello, LuxAsm assembler? Are you there?"...assembler: "Yes,
receiving loud and clear!"...editor: "Cool! Here's my special 'tightly
integrated' functions for you here, send me over pointers to your special
'tightly integrated' functions in return!"...assembler: "Sure, here they
are!"...

And, hey presto, the two are directly connected and "tightly
integrated"...note, also, the way that the IDE can establish direct
"connections" between the modules rather than all "messages" having to
always pass through the IDE itself...hence, the "overhead" for the initial
"linking" up of the tools is minimal...once the "negotiation" stuff happens
at the start - when the LuxAsm IDE loads and then loads up the "modules" -
the rest of the communications won't really be suffering any real
"overhead", as they get a direct connection to each other - a pointer to
the message processing "callback" (what Microsoft might call a "window
procedure"...except, of course, in this case, it's not necessarily attached
to a "window"...better thought of as a "message processing procedure"
instead...or maybe a "process procedure" or "event procedure"...the name's
not so important as the idea, of course :) - and then send directly to each
other...literally calling each other's procedures...

[ Or maybe, in fact, this could be made "multi-threading" and then we could
have multiple processes for each - rather than just "shared libraries" - in
which case, we've be using something like the "message queue" IPC in Linux
instead...this is a "detail" to be discussed...do the "modules" use a
co-operative or concurrent structure? Could work either way...a
"co-operative" style is easier...concurrent opens up extra possibilities
and is more flexible...this kind of thing, in fact, is where we're still in
discussion and working the specifics out :) ]

> > Unless you get some really cool thing done with linux/unix pipes that i
> > doubt will work on Windows.
>
> Not that LuxAsm is destined for Windows, anyway...

Indeed; Although, it can be done on Windows just as well...after all,
what's special about "UNIX pipes"? I mean, just because Microsoft haven't
put support in directly for such things doesn't mean it isn't
possible...you just create your own methods of "linking" together things
and how the "modules" intercommunicate...I had these ideas originally with
Windows in mind...it's being "translated" for LuxAsm to Linux...although,
indeed, UNIX / X has the right attitude in the first place for the kind of
thing we need that it's infinitely better on Linux...should have noticed
that sooner, in fact...but wasn't thinking on that wavelength until Frank
said "yes, let's actually create LuxAsm!"...then, yup, all my ideas fit
into place, almost as if "pre-destined" to have done so...or some
"subconscious" thing that I always knew it would be there in the end...

Nevertheless, it's all perfectly possible on Windows...Windows doesn't have
the "pipes" functionality built-in to its support? Then code your own...

> > An IDE frontend would have to be built into the assembler, and if
> > someone wanted another IDE they would have to patch the code.

Not necessarily; Mirroring the X Windows system that LuxAsm runs upon and
is specific to, I've pre-planned for this eventuality (you see, Frank, I
don't just make these things up for fun...well, okay, I do...but some small
amount of thought is expended in advance ;)...

And the "trick" I've settled for and proposed to the rest of the team is
simply to make the whole thing "client / server" and "event-driven" as a
"foundation" upon which to build it all...exactly the same kind of
"architecture" as X or Windows follows, really...except that this is solely
about the much more limited area of just the IDE and the toolset that goes
with that, not a full blown GUI (no, indeed, X, of course, provides the GUI
:)...if you like, it's somewhat like a "second layer" of event-driven
client / server architecture...X provides this for the GUI...the LuxAsm IDE
provides it for a kind of "higher-level" logical thing of just IDE-like
stuff (a "mini-desktop" inside the LuxAsm IDE window, so to speak :)...but
using the same style of "architectural layout"...

This allows "tight integration" and "modularity / separation" at the same
time...I know, a touch of "oxymoron" about that...but, well, think about
those DLLs loaded into your address space under Windows..."integrated"
tightly into the very workings of the program but, at the same time, it
separated and Microsoft can play around with the code inside it for each
new version...sending "messages" between windows and so forth...

Well, with a bit of "dynamically linking", the main LuxAsm IDE "server" can
pick up the plug-in "modules" off the disk...they are formatted in a
special way (which also doubles as a "check" that the "module" really is a
proper LuxAsm IDE module and not just some random shared library dropped
into the wrong part of the disk ;)...so, each module has an "Init" function
and a "clean-up" function or whatever, through which they have a chance to
initialise and uninitialise themselves...the IDE provides an "API" to the
"modules" so they can call things like "AddMenu" or "AddButton" or
"ShowDialogue" or something along those lines (API to be finalised later in
discussion, of course ;)...and the IDE calls a "callback" or something
similar (if we go for a "multi-threaded" style then the *NIX IPC like
"message queues" or "shared memory" could be employed...again, a matter of
discussion...is the flexibility of "multi-threading" that necessary for the
extra complication? ;)...then they can "send messages" to each other...

One additional thing I'd think to add for "performance" reasons and for
better "tight integration" is a simple IDE API of "GetModule"...with this
query via the IDE, one "module" can get a direct "channel" to another
one...just using the IDE as a "middle man" for the modules to locate and
link to each other but, thereafter, that "middle man" is cut out and they
can send messages _directly_ to each other without the IDE being involved
(an idea that GUIs might one day consider, should Microsoft learn how NOT
to be a control freak at some point? After all, rather than "DirectX",
standardise how the display drivers and such all work - device drivers need
to present a "device-independent" layer to the OS, anyway, so "portability"
is already covered at this point - and then provide a "GetDriver" API (with
suitable "DRIVER_DEFAULT_DISPLAY_DEVICE" flags to select the default
display symbolically without needing to know which particular card in which
particular slot it is ;) to establish a "connection" directly to the driver
then the applications can _truly_ "deal direct" with the device drivers...a
not particularly complicated idea so, of course, this kind of thing is way
beyond Microsoft...they need someone else to do it first, because they
_steal_ all their "innovations" from everyone else rather than thinking up
any for themselves ;)...

Note that the IDE "main module" actually has absolutely nothing built-in,
other than opening a window and providing the API and the "module"
stuff...that is, even the "text editor" is just another "module"...

Yes, as I warned the LuxAsm team, this carries more "upfront" work at the
start...but if you want a flexible and solid "foundation" to build things
on...well, few things come more solid but yet totally flexible than this
style of design...it's trademark "me", basically: A "best of both worlds"
solution because when I want my cake, I want to eat it too (a daft saying
to my mind...why would someone want a cake but then doesn't eat it? What a
waste of a good cake! ;)...

You can basically rig up a set of "foundations" like this in all manner of
ways...and all of it is inherently done in a flexible kind of way...each
"module" can be developed in isolation...they can also directly "attach" to
each other and then supply functions to each other for direct "tight
integration" (for instance, the "editor" and "assembler" modules will
locate each other and then the "editor" will send the information to the
"assembler" and it will send messages back :)...if the "editor" fails to
locate the LuxAsm "assembler" (which is equipped to be able to comprehend
these special "messages" to do with "feedback" and "incremental assembly")
then, oh well, it falls back to just being an ordinary "Notepad" text
editor...then there's the idea of "filter" modules which act as "wrappers"
for external tools not designed specifically for the LuxAsm IDE...so, a
"NASM filter" can be written which integrates with the IDE but "converts"
things into command-lines for input and parses the output to send back the
error messages or whatever...as things progress, additional "messages" or
API can just be added on to expand the "features" available from the IDE
(just like Microsoft do to Windows all the time adding new messages and API
to expand what Windows can do :)...

It's ambitious because this carries a lot more "upfront work" than
applications usually get...you know, drafting out how "modules" work, an
"API" and set of "messages" for the IDE and so forth...but, for that "down
payment" at the start, you get a solid and flexible set of "foundations"
that, indeed, should be tough enough to stand skyscrapers (yes, with that
ornate balcony for you, Frank ;) on top of...how can I be sure? Well, ever
taken a look at the size of Windows lately? Well, if it can handle the
"Mount Everest" of Windows sitting on top of it, then it can handle the
more modest "small tower" that LuxAsm will work up its way to being, as a
"tightly integrated GUI-based assembly language toolset"...

Oh...you've probably worked it out already but, of course, we're "open
source" and, therefore, the details of this "API" and "module" interface
and the "messages" will be documented...any third-party tool or IDE is, of
course, free to use that information to add on a new "module" to
LuxAsm...which will be somewhat similar to just looking up API and messages
for, say, X or Windows...except that this is only an IDE rather than a GUI
or even an entire OS, so it'll all be much, much simpler and smaller and
modest, of course...

> Worse than that, going with an intermediate file format as I did with
> Lisa has other drawbacks - specifically, you can use generalized
> text-based tools on the "source" files. You have to write a utility to
> convert the specialized files to standard text files (and vice versa).
> Almost elminating the usefulness of the system.
>
> The approach Beth is proposing probably keeps the external
> files in source format and creates an intermediate form (probably
> OBJ with some metadata) as a "background" task.

Yes; The source files are fully intended to remain as normal plain text
files...UTF-8 support is about as "non-standard" as that's likely to
get...any editor will be able to create and edit those files...

What happens, though, is that the source files are, as Randy notes,
assembled into an "intermediate form" as a binary file too as a parallel
process...the "editor" module deals with the editing but sends on the
changes to the "assembler" module, which deals with the assembly and then
sends back any "warnings and errors" that may have happened while doing so
back to the "editor" (this provides the "feedback" stuff :)...you're
actually editing a plain text file in the editor but, in parallel, the
changes being made are also passed onto the "assembler" so that the "object
code" version is also being kept up to date at the same time...

As for that "intermediate form" then I was actually thinking that the way
to go about it - as suggested to the LuxAsm team (waiting for feedback on
the idea :) - is to note that a ".lib" file is merely a "collection of .o
files"...an alternative way to look at this, is that - at least
"functionally" so - what's an ".o" file? Well, it could be said that it's
effectively a ".lib" file with only one "module" in the "collection"...

Hence, with this perspective on the matter, why bother with ".o" files at
all? Rather, we simply always deal with ".lib" files throughout (ooh,
Rene's foaming at the mouth...not only are libraries being used but they
are even more _central_ to the operation than most assemblers!! It's
"library crazy"!! ;)...if the programmer is doing some "monofile" thing
with only one source file then, sure, the ".lib" actually only houses one
".o" file...a touch of "overhead" in the file...but, well, "big
deal"...though you can use it in such a "monofile" manner, of course...

Again, as I was suggesting to the LuxAsm team, the ".o" file was thought up
first...then someone realised "ah, stick a bunch of them together and get a
'.lib' file"...but then once you've got a ".lib" format, then there's
actually no real reason to keep the ".o" files...a ".lib" with just one
".o" file in it is still a valid ".lib" file...

Therefore, I see it being the case that what LuxAsm does is simply maintain
a big ".lib" file for the project...if it turns out that's there only one
souce file - "monofile" - then, okay, all that extra stuff isn't really
necessary but it doesn't get in the way or stop anything...it's only an
"intermediate file" and but the "overhead" of some "header" at the start,
anyway...who cares? But, otherwise, it can work directly with the ".lib"...

There may be a necessity - or even just a case of "desirable" - that maybe
LuxAsm will invent its own "embellished" library format...for instance, if
"source code in the executable" is to be supported then we'll have to add
on some extra "non-executable" sections to house the source code...this
stuff can be added on, though, as an "extension"...you know, just like how
"debugging information" is dealt with...LuxAsm recognises these sections
and can interpret them to also link in the "source code into the
executable" (unlike Rene, I think the "default" will be reversed...you
_ask_ for this to happen rather than it happens automatically and then you
have to "strip out" the source code later...but, well, that's more a
"cosmetic" / "personal preference" issue...I just don't like the idea of
"ramming things down people's throats" too much...leave the option
there...if people like it, they'll use it...if they don't, then, oh well,
better luck next time ;)...some "metadata" can be thrown into a section
like this too for any "additional information" that might be needed to
smooth the process...

The other point of doing it via a ".lib" is then that the whole project can
be stored in one binary file...moving a project around involves just moving
the ".lib" file around...with the "source code in the library" idea, then
you could really actually have "monofile" programming of a different
kind...that literally the entire project is in one big ".lib" file...that's
a nice idea...but then, yes, that makes it difficult to get at the source
code for use in other editors...

I confess that I'm still deciding whether to simply stick with plain text
for the source or maybe to do something like Rene does ("monofile") in a
slightly different way...that is, the source code is in the library
itself...but the "editor" can read plain text files or the "librarian"
module can "export" the source files from the library...further, you could
even consider such "exporting" in a CVS light as "checking out"
instead...twist around the ".lib" format and put CVS facilities into
it...all the LuxAsm tools have command-line interfaces too so you can
"export" your source files from the library using that into plain text...

Although, if this idea is to be considered, it's definitely in the "future
development" category...not to be attempted straight away but an idea that
could be worked towards as everything else is done...

Also, the "project tree" would effectively - a bit like with C++ Builder -
be just a "makefile" and MAKE underneath it all (compliant to other
MAKEs...in fact, at the beginning, it probably will be GNU MAKE as a "stop
gap" until we write our own...ummm, _IF_ we write our own...don't know if
we really need to...depends if someone has a "yearning" to do it...well, C
(the person) is developing with LuxAsm as a kind of "practice" for some
other compiler stuff he wants to work on later...and, boy, do my "mad
ideas" give one heck of an "obstacle course"...but if you can deal with
these ideas, you can deal with anything ;)...

Then, basically, LuxAsm would reach where _I'm_ really planning on it
heading...the problem with assembly language is a lack of support tools and
"large-scale" thinking...as a "C side" tool (no, not a "seaside
tool"...that's a bucket and spade for making sandcastles ;)...and that kind
of thing...ah, yes, I confess an inclination towards the "assembly rebirth"
concept myself...it's an ambitious goal, for sure...but I think the
weakness lies in the tool support...it's all about ".obj" files rather than
".lib" (inherently presuming one or two source files)...who - except for
SourceForge people just because it's "built-in" to how the site operates -
uses CVS as a matter of course for assembly language projects? It's all
"small thinking" in my view...so, I think I said this a long way back but
it's still true, I'm thinking that a toolset with a "large-scale" view is
part of the "missing link"...these are features to make HLLs quake in their
boots...

I don't think in half measures...this is a strength and a weakness I
have...alone, this means a tendency to plot "world domination" and then,
ummm, not get any of it done...but in a team with LuxAsm...well, you never
know, "it just might work"...anyway, I'm a believer in the old notion of
"aim high" because, well, even if you miss the target, then because you
were aiming so high then falling short of the target can still mean doing
something rather spectacular regardless...you know: Let's go for "World
domination"...drat! Failed...only managed to dominate Asia instead...wait a
minute! That's hardly "small fry"...it's still pretty miraculous...cool!
I'll just stick with Asia alone then...one can hardly complain at that,
surely? ;)...

> Then the issue becomes one of how to deal with incomplete
> or syntactically incorrect programs.

My thinking here is not too get too clever about it...the errors are
reported back into the editor with that whole "underline" thing, as
mentioned before...but what goes into the binary file when the source code
has errors? I was thinking along the simplistic lines of just "leave it be
in error"...you know, leave those errors hanging in the air...

Because the perspective here is slightly different...you're working on the
project in "sessions"...you load up the IDE, open up the "project" and then
edit it, which makes changes to the "project" and then when you complete
the "session", it just closes it whatever it may happen to be...when you
load it up next time, it opens up at that previous state (whether
incomplete or not) and then you carry on...

Being "incremental" and "continous" as you're typing away then there's no
real dividing line, if you see what I mean...it's all just one long
continuous "editing" split into various "sessions" when you load up the IDE
and get to work...unlike with the "non-incremental" method, for instance,
the notion of "build number" is confused...it doesn't make sense here
because you don't compile the whole thing, you work on it line by line
continuously..."seamlessly", so to speak...hence, it's not a series of
"compile -> edit -> compile -> edit"..."compile" and "edit" happen in
parallel...

As the saying goes, "Where do you start drawing a circle?"...anywhere,
basically...so, if "seamless" and "continous" then you could, for example,
end a session mid-way through with a bunch of "unresolved externals"...it
just "shuts down" at that point...when you open up a new "session", it
loads back in the "previous state", those "unresolved externals" are still
there and you can carry on...

If you like, it's logically just one really long "session" from start to
finish...but practically, you're not going to work on it in one sitting
only...so, basically, it kind of "hibernates" between sessions...what's
currently going on is saved to the disk (that "metadata" in the ".lib" file
probably being the place that goes :) and is picked up where it last was
for the next "session"...

So, just leave errors as errors...you know, a similar attitude to that Word
spell checking thing again as an example...rather than popping up a
dialogue box with "YOU IDIOT! THAT'S SPELT WRONGLY" every time you make a
spelling mistake that needs to be dismissed with the "OKAY, I'M AN IDIOT!"
button (I assure you, nothing more annoying than that...oops, a simple typo
and it's throwing a tantrum at you! ;), you stick a red underline below it
to show there's a problem there and then it's up to the user when and how
they want to tackle those errors...

At the lowest level, there's the question of what exactly goes into the
binary file for a syntactical incorrect instruction...you know, they type
"MVO" rather than "MOV"...what does the "assembler" stick into the file
regarding this? A good question...but it's a "minor detail", really...it
could simply put nothing...or one idea is it puts a "UD2" opcode or a "INT
3" opcode there...if it refuses to actually finally compile the executable,
then it could put anything into that slot for the time being...I don't
really see it as a big issue...generally, it simply leaves errors be...the
user will get around to fixing those when and how they like (the
"underline" thing and "message window" keeps track of these to be
re-visited at any time you like...all very much like the Word spelling
checking, basically :)...if the error is fatal then it doesn't produce an
executable...what's in the "intermediate form" at that point at the error
position? Does it actually matter? It's not going to go any further and
produce the executable until the "fatal" error is dealt with, so what's
stored in the "intermediate form" seems immaterial...but, for completeness,
it can be some kind of delibrately incorrect "placeholder" like a "UD2" or
"INT 3" or whatever...the "metadata" - for anything that can read that -
could hold information pointing out the location of any such
thing...anything that isn't "fatal" then just leave it be...

It's a good question but a minor detail...hardly reason to throw out the
entire thing...but, generally, the simple answer is probably "do
nothing"...leave errors as they are...don't assemble anything until it
makes sense that it can be assembled...just leave the errors be...once the
programmer gets around to correcting them, the problem vanishes...yeah,
actually, that's the real answer: Do absolutely nothing about it at
all...thinking it over, this is one of those "don't usurp authority" /
"don't be a control freak" situations...indeed, don't flash up "Spelling
error!" dialogue boxes all over the screen...discreetly put a red underline
under the words and then carry on...the user will get around to solving
these things in their own good time at some point...

> Incra and LISA always caught
> syntax errors while you were entering the source file, so syntactically
> incorrect source files meant the file was corrupted. However, if someone
> edits a text file and you attempt to reload that source, this can create
> problems
> if you expect a syntactically correct intermediate format (later versions
of
> Lisa actually allowed syntactically incorrect source lines and stored
those
> as pure text; any attempt to assemble such a line produced a "syntax
> error").

Yeah, something to think about...there's no problem if LuxAsm is used
exclusively, as it can work out some way and then consistently stick by
it...but the idea of a source file being edited by some other editor and
then stuck back into LuxAsm...there are "issues" that need be considered
there...don't worry...I'll think of something...much can be dealt with by a
"leave it be in error" and "won't assemble" / "won't link" policy...would
need to think through all ramifications though, as I've not really done
that totally...

Ah, why are they using some other editor, anyway? They're just being
annoying for the sake of it, aren't they? Boo! Hiss! ;)

Beth :)

Quantcast