Re: Auto-update protocol

D Yuniskis <>
wibbled on Wednesday 10 March 2010 19:48

Hi Tim,

If you were to open() with O_WRONLY or O_RDWR then you stand to modify
the file in place with unpredictable results unless coordination
mechanisms are in place.


If it didn't, how could you write any system with shared access to a set of
files, eg an RDBMS?

Yes. I was hoping I could rely on particular (existing) services
to do that "consequentially" for me. E.g., using connection
oriented protocols in the hope that they would "lock" files
they were actively transfering (vs. connectionless protocols
which can opt to forget they were even involved in a transfer
"moments ago")

Can I ask for some parameters? How big are these images likely to be in kB
with any header overhead?

Just because we've considered the obvious, doesn't mean that there might not
be something already existing - such as one particular instance of an FTP
server that does this.

If you are at the mercy of customer server hardware, I suspect the only
sane way is to provide a daemon of your own for them to install. Note
however, this could be Apache with your custom config and service tree of
Perl/PHP/whatever. Whether you write a server from scratch or wrap the
protocol in HTTP and bundle an Apache blob, at least you give the user
something to install which you do control.

I don't think there is any other way around it.


Do you have any expectations of your customer's systems - are you contrained
to one OS, do you supply the update server - or do they, will they insist on
using any random OS they happen to like? Would it be less of a headache to
sell them a little embedded update server with an FTP interface that they
dump images onto (which of course are auto managed by the box in a correct
fashion)? They might like that - little black box to plug n go and you get
to own it. Would one of your existing devices be powerful enough to just re-
badge for the job, with perhaps additional flash bolted on. Just thinking
out loud here...

Personally I'd probably write a perl server based off FTP or TFTP using
the many available CPAN modules to do a forking server (or a multiplexed
one) and also to implement the protocol. There's be very little actual
code to write so it shouldn't take long. But of course that's just me.

I suspect I am just going to leave others deal with *their*
problems as best *they* see fit! :> I can leave myself a note
as to how *I* would (in the future) update my devices and
"not worry" about whether this works for other people.
(I am amazed at how often people fail to read/understand
"simple instructions". Particularly when it comes to updating
firmware -- e.g., why are there so many "bricked" devices
out there? couldn't people *read* how NOT to brick their
device??? :< )

Always a difficult one. Are your users technical? Most sysadmins would be
expected to flash all manner of devices using anything from TFTP to
XModem/RS232 and not break the box (otherwise their boss chops their nuts
off). If you sold the devices to me, i would not consider running a little
TFTP server a problem, provided clear instructions were given on how to load
the new images.

But if your users are AV dudes, that might be too much to expect - their
expertise lies in different places.

Correct. The TEXT is considerably larger than the RAM available
in the devices (which is often the case!). It is going to be
a significant challenge to arrange the image so that I can
do the update while the device is still "operating" (something
I've opted not to mention in this thread as its just extra
complexity that doesn't, IMO, impact the update protocol

I don't suppose you have enough flash for 2 images with a pivot in the
bootloader? I expect that is obvious to you, so I apologise for wibbling
eggs to granny ;->

Reliability is of paramount importance. Can never have corrupted
images. Must (also) always be able to *complete* an update
(subject to the service being available, of course).


Efficiency -- in terms of how it impacts other things on the
network and the server(s) -- is next most important. Since
updates are (should be!) infrequent, don't want them to bring the
rest of the "system" to its knees when they occur.

I think randomised polling will deal with that adequately.

Simple is not important. However, I am trying to push the
complexity into the *device* instead of the server. :<

Having written a fair few mini servers in perl, it is a small cost to
make other stuff work.

As you say, you basically want a naming scheme and a server that caches
the file to RAM when asked for a download and verifies the file before

You also want to make that server multithreaded *and* smart -- so it
doesn't try to cache 5 copies of the same image for five different
clients (and not service them "sequentially"). :-/

Again - step back. How many devices and how big is the image. If the
probability leans towards 5 concurrent connections for a few minutes for a
single 64kB image, although you might like to be clever, it's not really
worth it if the server has 2GB RAM!

Sometimes things just "don't seem to work". And, if the client
has the final say (it does!) on whether or not to flash itself,
if you can't somehow *force* it to do so, then you have to
resort to some other trickery (e.g., edit the image to convince
the device that it is a "new" image -- even though it isn't).

<shrug> I've just found that giving devices too much of a say
in how they behave (i.e., allowing them to ignore you) often
results in situations where you find yourself trying to
outsmart the device.

Does the client have a list of all the other clients - otherwise such an
announcement is unlikely to escape the VLAN segment.

Part of ongoing network discovery. E.g., the devices all need
to synchronize their clocks, converse with each other when
something goes wrong (i.e., to determine if the reason "I"
am not getting data from the audio server is because of something
specific to *me* or common to "all of us", etc.), etc.

The only reliable way unless you know UDP broadcasts are guaranteed to get
between devices would be to coordinate through the server - at least as far
as each device is given the IPs of one device on each other VLAN or
broadcast domain.

If the expectation is that many devices will likely be within the same
broadcast domain, and it is not important that they can talk to *all* their
friends, broadcasting should be good enough. I don't have enough

I.e., it's a distributed system and the devices are intended to
be semi-autonomous. But, that doesn't mean they can't elect to
take advantage of information that they might have that could
be beneficial! Surely they won't get anything of interest
from some *printer* sitting on the network :> But, if they
find similar peers, they *could* (if designed intelligently
to avail themselves of this capability).

Of course, if they are the lone wolf on the wire, they're SOL. :>
(so, maybe they opt to poll for updates at a different frequency
than when they *know* there are cooperating peers available?)

Yes. Any protocol has to take into account the fact that there
might be switches, routers, etc. "in the way" and how to bridge
those -- or, how *not* to!

Your unlikely to have issues with Layer 2 switches IME even over trunked
connections to the same VLAN on other switches, but once routing is
involved, it's mostly game over for broadcasts.

it *now*. IME the edge case use will be rare but when you need it, you
*really* need it...
I could do that indirectly by just cycling power to *one* client.
At IPL, he probes for an update. Finds one. Informs the others
on the network!

Wow, that sounds cool! ;-)

Yes it would.

Progress! :>

Call me devil's advocate(!) I'm a newbie with tiny systems but I've run some
fairly decent linux installations.

Tim Watts

Managers, politicians and environmentalists: Nature's carbon buffer.