Re: Library design for downloading an unknown amount of data?

Loïc Domaigné wrote:
Hi Jef,

Let me give some background info first. I'm writing a library to
download data from a number of external devices (dive computers).

[snip]

I don't know if that makes sense, but you could use a in-out parameter
for the size argument:

device_status_t
device_dump (device_t *device,
unsigned char data[],
unsigned int* size,
);

As input parameter, size contains the size of my data[] array, and as
output value the size of the data downloaded. If the array data[] is
too small, you should guarantee that you won't overflow the data
array. The application could then handle such situation by reissuing a
device_dump() as long as there are data to download.

You could alternatively use two parameters instead of a in-out
parameter for the size:

device_status_t
device_dump (device_t *device,
unsigned char data[],
unsigned int size,
unsigned int *download_size
);

This last version is exactly what I have at the moment. But with the
exception that I'm returning an error code if the buffer is too small.
Of course I'm not overflowing the buffer :-)

What are you meaning with "... reissuing a device_dump() ..."?

Do you mean returning the exact size to the application, so it can retry
the download with a buffer of the correct size? The problem with that
approach is that retrying the download is not really an option in my
situation. Downloading is often very slow (e.g. order of minutes) and in
some cases even requires some action by the end user (e.g. pressing a
button on the device). So this is not something you want to do.

If you mean returning chunks of data until the application has retrieved
all the data, that is a different story. I suppose you have something in
mind that resembles the typical file I/O pattern where you keep reading
data until some EOF condition is reached:

unsigned int nbytes;
unsigned char buffer[1024];
while (device_dump (device, buffer, sizeof (buffer), &nbytes) != EOF) {
/* Process the chunk */
}

A potential problem with this approach is that the buffer size will be
very different from the packet size that is used in the underlying
transfer protocol. Some devices send everything in a single packet,
others use multiple packets, where the packet size can be fixed of
variable. Thus this would require some internal caching, making the
implementation more complex. The simplest implementation is probably to
download everything, cache the data, return it chunk by chunk to the
application and free the cached data once the final chunk is delivered.
(Note that this resembles somewhat one of my candidate solutions where
the application is handed a pointer to the internal cache, but now we
don't have to expose the internal cache and we can also destroy it earlier.)

Downside is that if the application wants the data in a single buffer
(which is very likely), it will have to dynamically increase the buffer,
or use a very large "worst case" buffer. But in that last case, we are
back to the situation that I wanted to fix. That is the area where I like the solutions in my original post, because there you get the exact size together with the data, so you can allocate a buffer of the right size in one shot.
.

Quantcast