Re: Library design for downloading an unknown amount of data?

Ben Bacarisse wrote:
Jef Driesen <jefdriesen@xxxxxxxxxxxxxxxxxxx> writes:
<snip>
The point is how to pass the data back to the application, with an api
that is easy to use (e.g. without having to go through many pages of
documentation) and at the same time also efficient (e.g. no
unnecessary memory copies etc). That's what I would like to solve.

[Apologies if I have missed some context that renders this suggestion
pointless -- I have not been following this thread properly.]

That's fine. All feedback is appreciated here.

Have you considered a call-back API? The "user" passes in a function
that is called for every chunk of data. The user then has more
control: they can allocate and copy, copy to a pre-existing buffer,
or simply do something with the data with no copying at all.

It is more complex but it can sometimes solve more problems.

Two things to bear in mind. This works best if the chunks can be
meaningful to the application -- so I would try to avoid presenting
the call-back with incomplete messages for example. Always provide
some way to get user supplied data included in the call-back. A void
* is often all you need. The user includes it when registering a
call-back and it is passed the called function along with the "real"
data.

I know this does not answer any of your questions about data
allocation and ownership, but you seem to have a handle on what the
problems are with that.

I do use callbacks in my api. For instance for reporting progress and other notifications. In my opinion, callbacks are almost perfect for this kind of task. But using them as a mechanism to return data to the application feels wrong.

Let's have a look at my device_dump() function. Its purpose is to download a blob of binary data and return that back to the application. There is no need to return partial chunks. Progress notifications are handled separately. Thus the callback function would only be called once, right before the function returns control back to the caller.

typedef void (*callback_t) (const unsigned char *data, unsigned int size, void *userdata);

device_status_t
device_dump (device_t *device,
callback_t callback,
void *userdata)
{
unsigned char buffer[SOMESIZE]; /* Or malloc'ed of course. */

/* Do some work here. */

if (callback)
callback (buffer, sizeof (buffer), userdata);

return SUCCESS;
}

That does work of course, but it also scatters the code over at least two places.

typedef struct {
unsigned char *data;
unsigned int size;
} mydata_t;

static void
mycallback (const unsigned char *data,
unsigned int size,
void *userdata)
{
mydata_t *mydata = (mydata_t *) userdata;

mydata->data = malloc (size);
mydata->size = size;

memcpy (mydata->data, data, size);
}

int
main (int argc, char *argv[])
{
mydata_t mydata;
device_t *device = ...;

device_dump (device, mycallback, &mydata);

free (mydata.data);

return 0;
}
.

Relevant Pages