Re: Library design for downloading an unknown amount of data?

On 26 Aug, 11:48, Jef Driesen <jefdrie...@xxxxxxxxxxxxxxxxxxx> wrote:
Nick Keighley wrote:
On 26 Aug, 09:36, Jef Driesen <jefdrie...@xxxxxxxxxxxxxxxxxxx> wrote:
Loïc Domaigné wrote:

Let me give some background info first. I'm writing a library to
download data from a number of external devices (dive computers).

as in a device for recording the time and depth of a dive?

Indeed.

Some additional information about the library can be found on our
website:http://www.divesoftware.org/libdc/

sounds interesting. Multiple manufacturers just adds to the fun!


If you mean returning chunks of data until the application has retrieved
all the data, that is a different story. I suppose you have something in
mind that resembles the typical file I/O pattern where you keep reading
data until some EOF condition is reached:

unsigned int nbytes;
unsigned char buffer[1024];
while (device_dump (device, buffer, sizeof (buffer), &nbytes) != EOF) {
     /* Process the chunk */

}

A potential problem with this approach is that the buffer size will be
very different from the packet size that is used in the underlying
transfer protocol. Some devices send everything in a single packet,
others use multiple packets, where the packet size can be fixed of
variable. Thus this would require some internal caching, making the
implementation more complex.

any way of finding out the packet size? Could you make the buffer and
packet size identical or could you make the buffer size bigger than
any packet size?

For protocols where the data is send in multiple packets, the packet
size is usually known in advance. In some protocols it is simply fixed,
and for others you can choose it yourself (within some limits), but you
should use the maximum allowable size for maximum speed.

For protocols where everything is send in a single packet, the first few
bytes usually contain the total length of the packet. So you can only
get the length during the transfer itself.

but at least you know at the beginning of transmission. At that point
you could malloc the correct sized buffer.

One of the goals of my project is to hide the protocol details from the
user and provide a common, easy to use api for all supported devices.

good idea

Thus the idea is that the user should be able to download data without
having to know about packet sizes, etc. That should be handled
internally by the device backend code.

ok

Right now the device with the highest memory capacity has 2MB of memory.
With the current api, you could allocate a 2MB buffer and that would
work for all supported devices.

couldn't you alloacte an amount specific to the device? ie. have the
library
make the decision? So you get an API like so:-

Result device_download (Device_context*, Byte** data, size_t
*data_size);

But that is a lot of overkill for those
devices that only have 32KB of memory. And what if someday there appears
a new device with even more memory?

hide that detail in the device driver

Even if you can't do that the book-keeping isn't *that* hard.

I didn't say it was too complex, only more complex. Thus I was only
saying that if there is an easier solution that is equally good, I would
prefer that.

(Note that this resembles somewhat one of my candidate solutions where
the application is handed a pointer to the internal cache, but now we
don't have to expose the internal cache and we can also destroy it earlier.)

Downside is that if the application wants the data in a single buffer
(which is very likely), it will have to dynamically increase the buffer,
or use a very large "worst case" buffer.

or build a linked list of buffers and then glue them all together when
the EOM occurs.

But in that last case, we are
back to the situation that I wanted to fix. That is the area where I
like the solutions in my original post, because there you get the exact
size together with the data, so you can allocate a buffer of the right
size in one shot.

the realloc() case isn't too bad. A common strategy is to double (or
some other
constant between 1 and 2) the buffer size each time it runs out of
memory.
If you are really memory impoverished at the receive end (unlikely I
think)
then you could use another realloc() to trim the buffer when you get
to the end.

I'm not really concerned that an application will run out of memory.

until now I wasn't certain you were running on modern "desk top"
hardware.
Since you are memory isn't *that* important.

The
receiver end is typically a desktop PC, where these amounts of data are
rather small compared to the available memory. But I don't think that is
an excuse for consuming more memory than necessary,

oh I agree, but it becomes *less* critical


especially because
for most devices I do know the total size in advance. But if I want to
support all devices with a single api, I also need to take into account
those few devices where it is not know in advance. And who knows
somebody will want to run this on a less powerful mobile device.

ah

I'm actually more concerned about the added complexity on the
application side. This linked list or growing buffer is something that
will have to be implemented in every application

I'd kind of assumed you wrote that once and hid it in a library.

application
device specific code
memory manager

Depands how clever you want to be. You could have one memory
management
strategy or you could select one. Your device context could specify
which.

thinking out loud...

struct
{
size_t initial_buffer_size;
double buffer_growth_multiplier;
Byte* (*grow_memory) (double);
Byte* (*eom_action) (Byte*);
} Device_context;

gets tricky but not undoable to handle realloc() and linked list
memory management. Depends how clever you want to be.






.

Relevant Pages

  • Re: Library design for downloading an unknown amount of data?
    ... A potential problem with this approach is that the buffer size will be ... Some devices send everything in a single packet, ... In some protocols it is simply fixed, and for others you can choose it yourself, but you should use the maximum allowable size for maximum speed. ... Right now the device with the highest memory capacity has 2MB of memory. ...
    (comp.lang.c)
  • CONFIG_PACKET_MMAP revisited
    ... I've been looking into faster ways to do packet captures and I stumbled on ... In that discussion Jamie Lokier suggested having a memory buffer that's ... shared between user and kernel space and having the NIC do DMA transfers ...
    (Linux-Kernel)
  • Re: How much RAM needed for low end Ethernet application?
    ... >> the fact that the Ethernet controller had buffer memory. ... it seems a convenient resource for packet ... How many Ethernet controllers are able to address this memory directly? ...
    (comp.arch.embedded)
  • Re: Question about NDIS memory management
    ... Thanks for the reply Stephan - in my original message when I said "NDIS ... nor the memory that make up the ... actual buffer area. ... When the memory is full a new packet and buffer are ...
    (microsoft.public.development.device.drivers)
  • Re: Discovering variable types...
    ... >- but I suppose MS expect us to use wrappers ... memory allocations for your variables from disk as well. ... >They most certainly are of fixed size, changing the size of a String ... >>me to keep buffer size and current postion right in the memory block. ...
    (comp.lang.pascal.delphi.misc)
Loading