Re: Library design for downloading an unknown amount of data?

Nobody wrote:
On Wed, 26 Aug 2009 12:48:27 +0200, Jef Driesen wrote:

One of the goals of my project is to hide the protocol details from the user and provide a common, easy to use api for all supported devices. Thus the idea is that the user should be able to download data without having to know about packet sizes, etc. That should be handled internally by the device backend code.

But what do you want to do with the data? That should be the main
influence on the API you provide, not the mechanisms for getting at the
data. Work from the top down rather than bottom up.

In general, the more you specify, the less flexibility you have for
implementation. Don't specify anything the application doesn't need to
have specified.

E.g. if a function returns a pointer to an array of structures, that means
that the back-end has to provide that data as a contiguous array in a
specific format. OTOH, if you provide an abstract handle along with
iteration and/or indexing operations, you have more flexibility in
implementing each backend.

The library contains two major layers.

The first one is the device layer, which itself consist of a protocol layer and a memory layout layer. The protocol layer is all about the low-level data transfer (packets, timings, etc). The memory layout is the information that describes where the dives are stored in the memory. This leads two two different api's in the library:

1. An api for downloading the entire memory (using only the protocol layer).

2. An api for downloading individual dives (using both the protocol and memory layout).

Both api's return binary data, and its structure is entirely defined by the manufacturer of the device. For the memory dump it's a single blob of binary data. The typical usage for this data is for diagnostic purpose. When the other api fails for some reason, the end user can send us a memory dump so we can try to reproduce the problem, without the need for the device itself. So an application only needs to save this data. It could also try to parse the memory dump, but the other api is intended for that purpose.

An single dive is also a blob of binary data, but with some additional metadata that is required for parsing that data, such as current time at the time of the download, model/serial number, firmware revision, etc.

The second layer is the parser layer that contains the information to extract the interesting information (depths, pressures, etc) from the binary dive data.

The typical usage is that an application opens a device handle and downloads some dives. Next, it creates a parser for that type of device and processes each dive with it. The data that comes out of the parser is than displayed in the application, stored in its database, etc.
.

Relevant Pages

  • Re: Recover/defrag memory API
    ... I can see that somehow it does recover a lot of memory. ... The answers you have been getting is that there IS NO SUCH API IT CAN USE. ... THAT CAN DEFRAG AND/OR RECOVER APPLICATION MEMORY. ... unless you, too, want to create fraud which destroys overall system performance. ...
    (microsoft.public.vc.mfc)
  • Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation
    ... > The problem is that our driver and library implement an API that we don't ... > tells the library to register it. ... The app then goes on its merry way until ... at which point it tells the library to deregister the memory. ...
    (Linux-Kernel)
  • Re: CE6.0 Driver Pointer Marshalling - passing pointers out only?
    ... VirtualAllocEx can be used to allocate memory in a specified process. ... Read/WriteProcessMemory can be used to access this memory. ... structures that relate to API parameters. ... Use VirtualCopyEx to map a region of kernel memory to the client's ...
    (microsoft.public.windowsce.platbuilder)
  • Re: Library design for downloading an unknown amount of data?
    ... The point is how to pass the data back to the application, with an api that is easy to use and at the same time also efficient (e.g. no unnecessary memory copies etc). ... Like I mentioned in my original post, this is an iterator style api, that downloads dives until all dives are downloaded. ...
    (comp.lang.c)
  • Re: WinINET API... Yay or Nay?
    ... indicate progress, and want to completely control the download, then for the ... DoFileDownload API of IE ... ... determined that the WinInet API was too unstable and unreliable to use. ...
    (microsoft.public.vb.winapi.networks)