Re: Library design for downloading an unknown amount of data?

On 26 Aug, 09:36, Jef Driesen <jefdrie...@xxxxxxxxxxxxxxxxxxx> wrote:
Loïc Domaigné wrote:

Let me give some background info first. I'm writing a library to
download data from a number of external devices (dive computers).

as in a device for recording the time and depth of a dive?

<snip>

[try a download and retry on failure now knowing the data size]

Do you mean returning the exact size to the application, so it can retry
the download with a buffer of the correct size? The problem with that
approach is that retrying the download is not really an option in my
situation. Downloading is often very slow (e.g. order of minutes) and in
some cases even requires some action by the end user (e.g. pressing a
button on the device). So this is not something you want to do.

sounds icky. Growing up on fairly slow comms links (or even mind
boggling
slow comms links) repeating transfers unnecessarily Just Seems Wrong.

If you mean returning chunks of data until the application has retrieved
all the data, that is a different story. I suppose you have something in
mind that resembles the typical file I/O pattern where you keep reading
data until some EOF condition is reached:

unsigned int nbytes;
unsigned char buffer[1024];
while (device_dump (device, buffer, sizeof (buffer), &nbytes) != EOF) {
     /* Process the chunk */

}

A potential problem with this approach is that the buffer size will be
very different from the packet size that is used in the underlying
transfer protocol. Some devices send everything in a single packet,
others use multiple packets, where the packet size can be fixed of
variable. Thus this would require some internal caching, making the
implementation more complex.

any way of finding out the packet size? Could you make the buffer and
packet size identical or could you make the buffer size bigger than
any
packet size?

Even if you can't do that the book-keeping isn't *that* hard.

The simplest implementation is probably to
download everything, cache the data, return it chunk by chunk to the
application and free the cached data once the final chunk is delivered.

yup

(Note that this resembles somewhat one of my candidate solutions where
the application is handed a pointer to the internal cache, but now we
don't have to expose the internal cache and we can also destroy it earlier.)

Downside is that if the application wants the data in a single buffer
(which is very likely), it will have to dynamically increase the buffer,
or use a very large "worst case" buffer.

or build a linked list of buffers and then glue them all together when
the
EOM occurs.

But in that last case, we are
back to the situation that I wanted to fix. That is the area where I
like the solutions in my original post, because there you get the exact
size together with the data, so you can allocate a buffer of the right
size in one shot.

the realloc() case isn't too bad. A common strategy is to double (or
some other
constant between 1 and 2) the buffer size each time it runs out of
memory.
If you are really memory impoverished at the receive end (unlikely I
think)
then you could use another realloc() to trim the buffer when you get
to the end.


--
What goes "Pieces of Seven! Pieces of Seven!"?
A Parroty Error.

.

Quantcast