Re: Discovering variable types...

not_at_any.adr
Date: 08/07/04 

Date: Sat, 07 Aug 2004 12:46:50 -0400

On Sat, 7 Aug 2004 06:44:17 +0000 (UTC), erewhon@nowhere.com (J French) wrote:

>No - I am a great fan of OverLoad
>- it is /OverLapped/ in the ReadFile API that shot me in the foot

I know... It messed with my head at first, too.

>>It appears, given the inability to determine variable size from addresses, I
>>have no choice but to do it this way.
>
>IMO an advantage, as you will not have all sorts of fiddly stuff
>determining the data length
>ie: a ruddy great Case Statement dealing with (spits) Variants

I would not have gone that way either.

If you are using my mini memory manager, take a look at the SDK for the heap
calls. You will find a function "HeapSize" which will tell you the size of
any heap block owned by a pointer.

        var
                x : pointer;
        begin
                gemem(x,4096);
                writeln(sizeof(x)); // prints "4" on screen.
                Writeln(heapsize(hheap,x)); // prints "4096" on screen.
        end;

This is because X is actually stored in the executable's memory space but the
memory it points to is on the heap.

Now a function similar to HeapSize that would tell me the memory space used by
any given variable, from it's address, would be invaluable in several
circumstances, not the least of which is sizing read and write operations.
You wouldn't need case statements, just the size of the memory block occupied
by a variable, as a number.

>The 'hit' is not just in the disk head movement, actually the disk
>head seldom really moves
>- it is in the convoluted pile of tests that takes place as the OS
>decides whether the Handle is valid, whether the file is open for
>read, whether it is locked

Yep.. but it's not so pricey as you might think. Or rather it's a price worth
paying at some level, since the last thing you need is carelessness with
irreplaceable data.

Double buffering (i.e. creating your own internal buffers) makes lots of sense
in a lot of situations and I use it myself. However, I tend to leave that to
the application level, not the "primitives" level I'm working on at the
moment. Thus I can write buffering schemes that are tailored to the actual
needs of each program rather than trying to cook up some universal scheme that
isn't actually right for anything... windows has already done that.

>I have a hunch that it is something like 6000 times faster pulling
>data from memory rather than getting it from a ReadFile

Given the difference between disk speed and processor speed that's been in
evidence since the XT, I'd agree... and the gap is widening all the time.
Memory and processor speeds are increasing at a rate something like
exponential in relation to storage speed.

The real deal here is custom buffering. For example: my half-completed midi
project is what's prompting me to improve disk access. When finished it will
pre-load whole songs with a single ReadFile call, parse them from memory and
then dispose of the buffers. Were I to do it byte by byte from disk, parsing
a 120k midi file (which is rather large for that file type) could take a
couple of minutes, from a memory buffer I can do it in about a second. The
advantage of doing this at the app level instead of in the wrappers is that
another program may need multiple or different sized buffers in which case I'm
not restricted by the limitations of the disk access unit.

Delphi's BlockRead, while ostensibly a good choice for this kind of activity,
does something rather strange that I've not figured out yet. Their "block IO"
subroutines are bizarre beyond belief. On occasion getting the file from disk
into a memory buffer can be *extremely* slow... I've seen delays of up to 3
seconds before disk access occurs, hence: the new file access routines.

As usual, if something is worth doing it's worth doing right, so I may as well
spend a couple of days getting the whole thing together and have it for life,
rather than putting a work-around in the app and then struggling with this
again in 3 weeks.

>>I will overload for the base variable types only. Anything else (records,
>>arrays, memory chunks, etc.) will have to go through the bulk io routine...
>
>I would also make all of them call the same underlying reader
>- there is something nice about just having one 'real' routine

Already done.

>I think you should look at Streams
>- to me the Native Pascal file access is ... well bizzarre
>- I think that Borland recognized that
>as did Microsoft when they moved from FCBs to Handles in about MSDOS 2

I experimented with TFileStream etc. but in reality these are just wrappers
for Delphi's blockio routines, leaving me with the original problem plus a
potential for new ones. It made more sense for me to just get down and dirty
and do it my way (with apologies to the chairman of the board).

>Yes that is exactly what I anticipated - also the Boolean, although
>mostly one would not bother to test the results

I must be strange then... I always test IO results...
I simply don't trust hardware to do what it's supposed to.

>>Easy... just set all the PAS and DCU files to read only.
>>The compiler will fail and your code itegrity is preserved.
>
>I would simply change them back from R/O, tinker and continue
>- it is the 'psychological' division that I miss

Hmmm ... sounds about typical.

I'm more of the "if it ain't broke don't fix it" philosophy.

>>For files Nparam holds the record size and Tparam holds the file path.
>>For directory ops Nparam is a filecount and Tparam holds the search wildcards.
>>etc.
>
>That is pretty much what I would go for
>- but I would have different Record types for the File ops and Dir ops

Naaa... I'm not trying to store complex information... just the file/dir
handle, record size and filename/wildcards. Anything else would be overkill
in my situation.

>( I often get slagged off here for recommending using Strings as
>buffers - yet a string is just a chunk of memory that knows about
>itself. )

Yep. The only disadvantage of that is that many times the buffers must be of
fixed size and ansi strings aren't.

Again, the "do it my way" thing comes to the fore and some time ago I created
a "wrapped pointer" setup that would let me allocate a chunk of memory with a
descriptor block beginning 8 bytes south of the pointer itself. This allows
me to keep buffer size and current postion right in the memory block. It's
really just an expansion on the old pascal strings but it works nicely. (Of
course with the new memory manager, I only need to keep 4 bytes for the buffer
position. The heap keeps the size parameter for me, so I guess I really
should update that as well.)

>>I've even got it opening and closing CD-Rom doors!
>
>Useful ...

But a major pain to get working...

// open the drive door
function FEjectMedia(DName : ansistring) : boolean;
  var
    Dstr : ansistring;
    Fh : longint;
    br : longword;
  begin
    result := false;
    if fisdrivereal(dname) then
      if FGetDriveType(dname) in [d_removable,d_cdrom] then
        begin
          dstr := '\\.\' + Dname[1] + ':';
          fh := createfile(pchar(dstr),generic_read,0,
                            nil,open_existing,a_normal,0);
          result := deviceiocontrol(fh,$2D4808,nil,0,nil,0,@br,nil);
          closehandle(fh);
        end;
    if not result then
      exception(FExc_door_open_cmd);
  end;

// close the drive door
function FInjectMedia(DName : ansistring) : boolean;
  var
    Dstr : ansistring;
    Fh : longint;
    br : longword;
  begin
    result := false;
    if fisdrivereal(dname) then
      if FGetDriveType(dname) in [d_removable,d_cdrom] then
        begin
          dstr := '\\.\' + Dname[1] + ':';
          fh := createfile(pchar(dstr),generic_read,0,
                            nil,open_existing,a_normal,0);
          result := deviceiocontrol(fh,$2D480C,nil,0,nil,0,@br,nil);
          closehandle(fh);
        end;
    if not result then
      exception(FExc_door_close_cmd);
  end;

>Have a look at the TFileStream, not with a view to using it, but with
>a view to nicking ideas from it.

Already done.

>
>Ah, and there is one more 'generic' that needs to go with your FGet
>stuff - that is: ReadDelineatedString

Got you beat! [smile]

function FgetTxt( FH : Fhandle; RData, Dls : ansistring) : longint;

Where DLS = an end marker that can be anything you supply...

        BytesRead := FgetTxt(FH,MyStuff,','); // comma delimited text.
        BytesRead := FGetTxt(FH,MyStuff,#12+#10); // Dos text
        BytesRead := FGetTxt(FH,MyStuff,' '); // words

[grin] Your turn...

-----
Laura

Relevant Pages

  • Re: [PATCH] Physical Memory Management [0/1]
    ... It aggregates physical memory allocating and management API in one ... PMM has it's own allocator which runs in Obound time where n ... Via a misc char device, the module allows allocation of continuous ... devices would have to develop a method of sharing buffers. ...
    (Linux-Kernel)
  • Re: section objects and scatter/gather DMA
    ... There are tricks for allocating memory ... can create a small service that allocates the buffers for you and passes ... maintain a certain history depth. ... To get to a dynamic pool management (free up pool ...
    (microsoft.public.development.device.drivers)
  • Re: Handling high UDP throughput
    ... The product that uses this sustains 540MbS with a 38kHz interrupt running using more than half the processor's power, so a lot goes on in the system but a lot of time is available for TCP/IP. ... The Ethernet driver was optimized, the memory movement was optimized (just using an inline memcpy that does a DMA transfer adds 30% to the effective speed), the IP checksum was in assembly, and a zero-copy TCP/IP stack was required. ... How much TX buffers did you have? ...
    (comp.arch.embedded)
  • Re: CE6.0 Driver Pointer Marshalling - passing pointers out only?
    ... This is an array of pointers to buffers that get set up ... the driver allocates the buffers (via an internal allocation ... routine from its own block of reserved memory). ... So I can map ...
    (microsoft.public.windowsce.platbuilder)
  • Re: Techniques to reduce RAM/ROM usage in an embedded system
    ... >Please share some of your experiences/techniques in reducing RAM/ROM ... >usage in an embedded system. ... I know some C compilers provide memory ... recognizing code that can be put into routines (maybe routines can be ...
    (comp.arch.embedded)